【数据同步】SeaTunnel 初体验，5000 字深入浅出带你用上 Oracle-CDC

Apache 是啥？

下一代高性能、分布式、海量数据集成框架。

支持上百个数据源、传输速度快、准确率高，丰富易扩展的连接器和插件化的连接器设计，能够更轻松的运行复杂的集成。

是一个分布式、高性能的数据集成平台，用于数据迁移和实时流处理任务。

JDK 下载安装

需安装 Java (Java 8 或 11， 其他高于 Java 8 的版本理论上也可以工作) 以及设置 JAVA_HOME。

本次选择了安装 8u381 版本，大部分的 bug 有得到修复，JDK 下载页地址： https://www.oracle.com/java/technologies/javase/javase8u211-later-archive-downloads.html

# 解压tar xvf jdk-8u381-linux-x64.tar.gz# 配置环境变量,添加2行cat ~/.bash_profileexport JAVA_HOME="/data/jdk-8u381-linux-x64"export PATH=$PATH:$JAVA_HOME/bin# 环境变量生效source ~/.bash_profile# 验证java -versionjava version "1.8.0_381"Java(TM) SE Runtime Environment (build 1.8.0_381-b09)Java HotSpot(TM) 64-Bit Server VM (build 25.381-b09, mixed mode)

SeaTunnel 下载安装

其实安装很简单，下载解压，开箱即用，截止作者撰写本文的时候，最新版本 2.3.5，所以本文的下载链接均为 2.3.5 版本，目前社区最新版本是 2.3.7。

官网下载安装包：[bin] apache-seatunnel-2.3.5-bin.tar.gz

参考文档：

• https://seatunnel.apache.org/zh-CN/docs/2.3.5/start-v2/locally/deployment

• https://seatunnel.apache.org/zh-CN/docs/2.3.5/seatunnel-engine/deployment

下载及解压

cd /data/export version="2.3.5"wget "https://archive.apache.org/dist/seatunnel/${version}/apache-seatunnel-${version}-bin.tar.gz"tar -xzvf "apache-seatunnel-${version}-bin.tar.gz"

配置环境变量

cat  /etc/profile.d/seatunnel.shexport SEATUNNEL_HOME=/data/apache-seatunnel-2.3.5export PATH=$PATH:$SEATUNNEL_HOME/bin# 环境变量生效source /etc/profile.d/seatunnel.sh

安装连接器

配置插件文件（按需）

连接器数据量很多，全部下载需要时间及耗费存储空间，本次仅添加了 Oracle-cdc 和一些常用的连接器，配置文件如下

cat  config/plugin_config--connectors-v2--connector-cdc-mysqlconnector-cdc-mongodbconnector-cdc-oracleconnector-elasticsearchconnector-http-wechatconnector-jdbcconnector-kafkaconnector-mongodbconnector-redis--end--

自动安装连接器插件（指定版本）

这里跟大家讲一下，从 2.2.0-beta 版本开始，二进制包不再默认提供连接器依赖，第一次使用需执行如下命令安装连接器。

若手动安装，安装包地址：https://repo.maven.apache.org/maven2/org/apache/seatunnel/

# 2.3.5 为当前安装的版本sh bin/install-plugin.sh 2.3.5

安装比较耗时，配置的越多越耗时，如下展示部分日志

Install SeaTunnel connectors plugins, usage version is 2.3.5install connector :  connector-cdc-oracle[INFO] Scanning for projects...[INFO][INFO] ------------------< org.apache.maven:standalone-pom >-------------------[INFO] Building Maven Stub Project (No POM) 1[INFO] --------------------------------[ pom ]---------------------------------[INFO][INFO] --- maven-dependency-plugin:2.8:get (default-cli) @ standalone-pom ---[WARNING] The artifact xml-apis:xml-apis:jar:2.0.2 has been relocated to xml-apis:xml-apis:jar:1.0.b2[INFO] Resolving org.apache.seatunnel:connector-cdc-oracle:jar:2.3.5 with transitive dependenciesDownloading from central: https://repo.maven.apache.org/maven2/org/apache/seatunnel/connector-cdc-oracle/2.3.5/connector-cdc-oracle-2.3.5.pomDownloaded from central: https://repo.maven.apache.org/maven2/org/apache/seatunnel/connector-cdc-oracle/2.3.5/connector-cdc-oracle-2.3.5.pom (3.2 kB at 2.3 kB/s)Downloading from central: https://repo.maven.apache.org/maven2/org/apache/seatunnel/connector-cdc-oracle/2.3.5/connector-cdc-oracle-2.3.5.jarDownloaded from central: https://repo.maven.apache.org/maven2/org/apache/seatunnel/connector-cdc-oracle/2.3.5/connector-cdc-oracle-2.3.5.jar (31 MB at 411 kB/s)[WARNING] destination/dest parameter is deprecated: it will disappear in future version.[INFO] Copying /root/.m2/repository/org/apache/seatunnel/connector-cdc-oracle/2.3.5/connector-cdc-oracle-2.3.5.jar to /data/apache-seatunnel-2.3.5/connectors[WARNING] Notice transitive dependencies won't be copied.[INFO] ------------------------------------------------------------------------[INFO] BUILD SUCCESS[INFO] ------------------------------------------------------------------------[INFO] Total time:  01:17 min[INFO] Finished at: 2024-07-25T11:34:52+08:00[INFO] ------------------------------------------------------------------------## 过程省略一万字。。 ##

安装完成后，连接器存放目录为： $SEATUNNEL_HOME/connectors/ 【重要】

安装完成后，检查连接器安装情况

ORACLE CDC 连接测试

这里以 11G 为例：

安装 JDBC 驱动

下载地址： https://mvnrepository.com/artifact/com.oracle.database.jdbc/ojdbc8/12.2.0.1

JDBC 驱动下载后存放目录：$SEATUNNEL_HOME/lib

源库相关配置

这里需要重启，生产环境谨慎操作！！！

开启归档及补充日志（注意查看磁盘空间，避免爆盘事件）

ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;alter system set log_archive_format='ARC%S_%R.%T.arc';shutdown immediatestartup mount;alter database archivelog;alter database open;

源库创建表空间、用户，及授权

-- 创建用户及表空间CREATE TABLESPACE logminer_tbs DATAFILE '/mnt/oracle/oradata/logminer_tbs.dbf' SIZE 25M AUTOEXTEND ON NEXT 512M MAXSIZE 32767M;CREATE USER LOGMINER_USER IDENTIFIED BY Vir123 DEFAULT TABLESPACE logminer_tbs QUOTA UNLIMITED ON logminer_tbs;-- 给用户授权GRANT CREATE SESSION TO logminer_user;GRANT SELECT ON V_$DATABASE to logminer_user;GRANT SELECT ON V_$LOG TO logminer_user;GRANT SELECT ON V_$LOGFILE TO logminer_user;GRANT SELECT ON V_$LOGMNR_LOGS TO logminer_user;GRANT SELECT ON V_$LOGMNR_CONTENTS TO logminer_user;GRANT SELECT ON V_$ARCHIVED_LOG TO logminer_user;GRANT SELECT ON V_$ARCHIVE_DEST_STATUS TO logminer_user;GRANT EXECUTE ON DBMS_LOGMNR TO logminer_user;GRANT EXECUTE ON DBMS_LOGMNR_D TO logminer_user;-- 授权表权限GRANT SELECT ANY TABLE TO logminer_user;GRANT ANALYZE ANY TO logminer_user;-- 增加权限，解决权限不足问题GRANT SELECT ANY TRANSACTION TO logminer_user; GRANT SELECT ON V_$TRANSACTION TO logminer_user;

SeaTunnel 增加作业配置文件

配置文件目录: $SEATUNNEL_HOME/config

cd $SEATUNNEL_HOMEcat config/oracdc_consoleenv {  # You can set engine configuration here  parallelism = 1  job.mode = "STREAMING"  checkpoint.interval = 5000}source {  # This is a example source plugin **only for test and demonstrate the feature source plugin**  Oracle-CDC {    driver = "oracle.jdbc.driver.OracleDriver"    result_table_name = "customers"    username = "LOGMINER_USER"    password = "Vir123"    database-names = ["VIRDB"]    schema-names = ["VUSER"]    table-names = ["VIRDB.VUSER.CT_DB_LABORATORY_BAK"]    base-url = "jdbc:oracle:thin:@10.10.10.10:1521/VIRDB"    source.reader.close.timeout = 120000    connection.pool.size = 1  }}transform {}sink {    Console {        source_table_name = "customers"      }}

本地启动作业，查看日志情况

sh ./bin/seatunnel.sh -e local --config ./config/oracdc_console

源端插入一条数据后，日志更新

汇总信息可以看到，总数从 11 条变成 12 条

常见问题

作业启动报错

报错信息为：Unable to create a source for identifier 'Jdbc'

===============================================================================2024-07-24 14:38:08,672 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Fatal Error,2024-07-24 14:38:08,673 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Please submit bug report in https://github.com/apache/seatunnel/issues2024-07-24 14:38:08,673 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Reason:SeaTunnel job executed failed2024-07-24 14:38:08,674 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:202)        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)Caused by: org.apache.seatunnel.api.table.factory.FactoryException: ErrorCode:[API-06], ErrorDescription:[Factory initialize failed] - Unable to create a source for identifier 'Jdbc'.        at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:100)        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:332)        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:188)        at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:88)        at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:156)        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:149)        ... 2 moreCaused by: org.apache.seatunnel.api.table.catalog.exception.CatalogException: ErrorCode:[API-03], ErrorDescription:[Catalog initialize failed] - Failed connecting to jdbc:oracle:thin:@10.10.100.140:1521:jszx via JDBC.        at org.apache.seatunnel.connectors.seatunnel.jdbc.catalog.AbstractJdbcCatalog.getConnection(AbstractJdbcCatalog.java:123)        at org.apache.seatunnel.connectors.seatunnel.jdbc.catalog.AbstractJdbcCatalog.open(AbstractJdbcCatalog.java:129)        at org.apache.seatunnel.connectors.seatunnel.jdbc.utils.JdbcCatalogUtils.getTables(JdbcCatalogUtils.java:78)        at org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSource.<init>(JdbcSource.java:57)        at org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSourceFactory.lambda$createSource$0(JdbcSourceFactory.java:78)        at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:112)        at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:73)        ... 7 moreCaused by: java.sql.SQLException: No suitable driver found for jdbc:oracle:thin:@10.10.100.140:1521:jszx        at java.sql.DriverManager.getConnection(DriverManager.java:689)        at java.sql.DriverManager.getConnection(DriverManager.java:247)        at org.apache.seatunnel.connectors.seatunnel.jdbc.catalog.AbstractJdbcCatalog.getConnection(AbstractJdbcCatalog.java:119)        ... 13 more

解决方法：

(1) 驱动配置错误

正确的应该是 oracle.jdbc.driver.OracleDriver 而非 oracle.jdbc.OracleDriver

(2) 驱动错误问题

之前用了 ojdbc14-10.2.0.3.0.jar 支持不了，重新下载了 JDBC 的驱动，如上文提到的 "安装 Jdbc 驱动"

找不到 ORACLE-CDC 驱动

===============================================================================2024-07-24 17:15:37,657 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Fatal Error,2024-07-24 17:15:37,657 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Please submit bug report in https://github.com/apache/seatunnel/issues2024-07-24 17:15:37,657 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Reason:SeaTunnel job executed failed2024-07-24 17:15:37,661 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:202)        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)Caused by: java.lang.RuntimeException: Plugin PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='Oracle-CDC'} not found.        at org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery.createPluginInstance(AbstractPluginDiscovery.java:234)        at org.apache.seatunnel.engine.core.parse.ConnectorInstanceLoader.loadSourceInstance(ConnectorInstanceLoader.java:61)        at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSource(JobConfigParser.java:77)        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:327)        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:188)        at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:88)        at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:156)        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:149)        ... 2 more2024-07-24 17:15:37,661 ERROR [o.a.s.c.s.SeaTunnel           ] [main] -

解决方法：

重点关注

"Caused by: java.lang.RuntimeException: Plugin PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='Oracle-CDC'} not found."

报错找不到 ORACLE-CDC ，可以通过配置文件 $SENTUNNEL_HOME/connectors/plugin-mapping.properties，查看当前支持的连接器名称，再配置后进行安装！

cat $SENTUNNEL_HOME/config/plugin_config--connectors-v2--connector-cdc-oracle--end--再次运行连接器插件安装sh bin/install-plugin.sh 2.3.5

权限问题

ORA-01031: insufficient privileges

===============================================================================Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:202)        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: java.lang.RuntimeException: One or more fetch                       ers have encountered exception        at org.apache.seatunnel.connectors.seatunnel.common.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherM                       anager.java:147)        at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:1                       60)        at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:93)        at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceReader.pollNext(IncrementalSourceReader.java                       :111)        at org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.collect(SourceFlowLifeCycle.java:156)        at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.collect(SourceSeaTunnelTask.java:116)        at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:168)        at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:121)        at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:703)        at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1004)        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)        at java.util.concurrent.FutureTask.run(FutureTask.java:266)        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)        at java.lang.Thread.run(Thread.java:750)Caused by: java.lang.RuntimeException: SplitFetcher thread 0 received unexpected exception while polling the records        at org.apache.seatunnel.connectors.seatunnel.common.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:165)        at org.apache.seatunnel.connectors.seatunnel.common.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:81)        ... 5 moreCaused by: org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector                        will be stopped.        at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:42)        at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.                       java:341)        at org.apache.seatunnel.connectors.seatunnel.cdc.oracle.source.reader.fetch.logminer.OracleRedoLogFetchTask$RedoLogSpli                       tReadTask.execute(OracleRedoLogFetchTask.java:130)        at org.apache.seatunnel.connectors.seatunnel.cdc.oracle.source.reader.fetch.logminer.OracleRedoLogFetchTask.execute(Ora                       cleRedoLogFetchTask.java:73)        at org.apache.seatunnel.connectors.cdc.base.source.reader.external.IncrementalSourceStreamFetcher.lambda$submitTask$0(I                       ncrementalSourceStreamFetcher.java:106)        ... 5 moreCaused by: java.sql.SQLSyntaxErrorException: ORA-01031: insufficient privileges        at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:494)        at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:446)        at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1054)        at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:623)        at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:252)        at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:612)        at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:226)        at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:59)        at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:747)        at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:904)        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1082)        at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3780)        at oracle.jdbc.driver.T4CPreparedStatement.executeInternal(T4CPreparedStatement.java:1343)        at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3822)        at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1165)        at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.                       java:291)        ... 8 more

解决方法： 增加 TRANSACTION 的权限

GRANT SELECT ANY TRANSACTION TO logminer_user; GRANT SELECT ON V_$TRANSACTION TO logminer_user;

总的来说，SeaTunnel 的安装配置还是很简单的 ，如果你是初次接触，出现了各种文档不熟悉，各部件不熟悉漏了下载的情况，以及各个 JAR 包、Connector 的位置存放以及配置混乱的等问题，可以通过社区 Github、邮件列表及微信社群来进行解决，包括查看社区的文章及参加社区的活动！

最后也感谢 Apache SeaTunnel 社群支持的小伙伴们！

参考文档

• ** 下载地址：**https://seatunnel.apache.org/zh-CN/download

• 文档参考：https://seatunnel.apache.org/zh-CN/docs/2.3.5/connector-v2/source/Oracle-CDC

• 源码查询： https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-oracle-e2e

• issue 记录： https://github.com/apache/seatunnel/issues/6799

本文由 科技 提供发布支持！