Apache DolphinScheduler 3.2.1 版本发布:增强功能与安全性的全面升级

Apache DolphinScheduler 3.2.1 版本发布!此次更新修复了重要问题,增强了功能与安全性,包括解决升级失败、任务组件路径限制等问题,引入 SQL 分割、k8s 自定义标签等新特性,并修复多个 CVE 安全问题。感谢社区贡献者的共同努力!详细信息及下载:[官网链接](https://dolphinscheduler.apache.org/zh-cn/download/3.2.1)

近期,Apache DolphinScheduler 社区激动地宣布 3.2.1 版本的发布。此次更新不仅着力解决了前一版本(3.2.0)中遗留的问题,而且引入了一系列的功能增强和优化措施。

1708503625620eba7641ff3ebfb4f160e2bb8a7d7fdb3

原先的问题主要源于部分重要代码在发布过程中未能成功合并(cherry-pick),加之这部分代码的合并过程较为复杂,因此,3.2.1 版本基于 2024年2月的 dev 分支代码,剔除了一些不兼容的特性后发布。

全部 Changeloghttps://github.com/apache/dolphinscheduler/releases/tag/3.2.1

下载地址https://dolphinscheduler.apache.org/zh-cn/download/3.2.1

主要修复和功能增强

新特性和优化

  • SQL 任务现支持使用 druid 进行 SQL 分割,支持设置 maxRows

  • 支持自定义 HTTP body 渲染。

  • Kubernetes (k8s) 现支持自定义标签 (label)。

  • 新增支持阿里云语音告警源。

  • Helm chart 现支持 JDBC 注册中心;支持任务类型过滤。

关键问题修复

  • 修复从 3.1.x 升级到 3.2.x 的失败问题。

  • 解决任务组件在使用资源中心时只能使用绝对全路径的限制。

  • 修复启动参数优先级设置错误。

  • 解决数据质量任务无法执行的问题。

  • 修复任务组队列失效问题。

  • 解决任务定义列表修改时任务消失的问题。

  • 修复特殊情况下删除工作流实例导致的空指针异常(NPE)。

  • 解决 Master 和 Worker 之间的通信问题。

  • 修复 Kyuubi 数据源在 UI 中不显示的问题。

安全性改进

此版本也对几个关键的 CVE 问题进行了修复,包括:

  • CVE-2023-49250

  • CVE-2023-51770

  • CVE-2023-50270

  • CVE-2023-49068

  • CVE-2023-49109

BugFix

  • fix: Resource relate path invalid when tenant change (#15581)

  • [fix][worker]:Fix the issue of missing disk usage reporting information  in the worker (#15565)

  • [Fix] Fix WorkflowInstance batch start failed will throw incorrect exception. (#15577)

  • Fix create parent directory will cause FileAlreadyExistsException (#15576)

  • Fix Recover WorkflowInstance will casue workflow Instance state is success but task insatnce is killed/paused (#15574)

  • fix: data quality may fail in docker mode (#15563)

  • fix: start param for wf not work (#15544)

  • fix: ddl without drop exists (#14128)

  • fix switch js (#15487)

  • fix: data quality can not use (#15551)

  • Fix createFile with permission will not work (#15556)

  • [Bug][force-success] force success add end time (#15144)

  • [Bug][Task Api] fix 'MACPATTERN' in ProcessUtils and cover all cases on MacOS in ProcessUtilsTest (#15480)

  • Fix TaskGroupQueue will never be wakeup due to wakeup failed at one time (#15528) Exit JVM when OOM (#15538)

  • Fix exception occur in RpcServer side, it will not be sent to RpcClient (#15536)

  • front: When you edit a task in the task definition list, the front task list is displayed (#12819)

  • [Fix] [Bug] Change default version of Workflow/TaskDefinition to 1 (#15498)

  • [Bug] Fix a bug,  When the worker service offline, workerNodeInfo cache in master cannot delete the offline worker (#15459)

  • fix workflow will have same updatetime when import (#14810)

  • [BUG] #15013 Fix retryInterval in RetryPolicy will never be used in RetryUtils (#15014)

  • Throw IllegalArgumentException if parse time placeholder error (#15514)

  • Fix PostgresqlDatabaseContainerProvider get Image is incorrect (#15434)

  • [Bug][Api] Fix NPE when deleting a workflow instance (#15485)

  • Directly Throw exception when taskInstancy log path is empty which log need to be queried (#15511)

  • Fix notify failover WorkflowInstance will cause NPE (#15499)

  • [HotFix] Fix createTaskInstanceWorkingDirectory failed if the old path exist (#15377)

  • [bug] Exception when using host in ipv6 format (#14040)

  • [Bug][Master]serial_wait strategy workflow unable to wake up (#15270)

  • [BUG][Task] fix java task classpath (#15470)

  • [Bug] [Audit log] Fix Audit log UI query error (#15427)

  • [Bug][Registry] Optimizing waiting strategy (#15223)

  • Set TaskGroupQueue updateTime when force start (#15510)

  • TaskGroupPriority only compare When TaskGroup is same (#15486)

  • Remove taskQueue and looper in worker (#15292)

  • Display the resource file doesn't exist message in task create page (#15350)

  • Recreate new TaskInstance Working Directory when exist in worker (#15358)

  • [Bug] Close SSH session after remote shell finish (#15348)

  • Fix check value rather than key in AbstractDataSourceProcessor#checkOther (#15351)

  • Fix resource file usage(Delete Resource/ResourceUser which is deprecated)

  • [Bug][Master] send ACK event timeout (#15346)

  • Fix k8sTaskExecutionContext setting configYaml (#15116)

  • [Fix #15129] [Dependent] The date rules of the dependent node are ambiguous. (#15289)

  • Fix failover Master might not release taskGroup (#15287)

  • [HotFix] Fix TaskOutputParameterParser might OOM if meed a bad output param expression (#15264)

  • [Bug-15215][Api] non-admin should not modify tenantId and queue (#15254)

  • Set the tenant as the owner in final stage (#15256)

  • Use chown to set the file owner (#15240)

  • [Fix] Change HTTP plugin timeout param to number type (#15234)

  • fix switch condition (#15228)

  • Fix docs style is incorrect by CI pass (#15167)

  • Expire session when update user password (#15219)

  • Fix home page workflow instance miss status (#15193)

  • fix security issue (#15192)

  • fix can't stop bug (#15191)

  • Remove API Result in Service (#15181)

  • Exclude DataSourceAutoConfiguration in worker server (#15169)

  • [Bug] Fix TriggerRelationMapper cannot work due to miss DatabaseIdProvider (#15153)

  • Fix spotless (#15164)

  • Fix incorrect button display text (#15160)

  • [Fix][Data Quality] Change t_ds_dq_rule_input_entry field name fix PostgreSQL not support value issue (#14992)

  • fix missing 'KYUUBI' in droplist of datasource (#15140)

  • [Bug] Fix endless loop (#15092)

  • fix: execute sql error: datasource plugin 'doris' is not found。 (#15123)

  • Fix confusing constant string for unit convertor (#15126)

  • [fix-#11726] fix error when set connection proerty both in the URL and an argument (#15093)

  • [Fix-15072][Resource Center] Non-admin user can not query resource recursively (#15097)

  • [E2E][Bug] Fix k8s-e2e (#15098)

  • Fix SqlTask cannot split the given sql when browser in windows (#15062)

  • [Fix-15036] [API] Fix task definition edit doesn't work (#14801)

  • remove sub workflow finish notify (#15057)

  • Fix missing Kyuubi type in UI (#15051)

  • [Fix-14885][pom] fix spotless format file path (#14889)

  • Fix When the task instance status is 'STOP' (#14967)

  • Revert "[Bug] [Resource] fix resource delete bug (#15003)

  • [Bug] [Resource] fix resource delete bug (#15003) Delete File generated by UT (#15022)

Improvement

  • [Improvement][UT] Improve Worker registry coverage (#15380)

  • refactor comments & function name for confuse (#15546)

  • [Improvement][HTTP] support custom rendering of http body (#15531)

  • [improvement][api] Fix typo for controllers (#15438)

  • [Feature-15475][DinkyTask] DinkyTask supports Dinky-1.0.0 and common sql (#15479)

  • [Improvement][K8S]Optimize MDC for K8S tasks (#15390)

  • Enable set ServerLoadProtection fot Master/Worker (#15439)

  • [Feature] timed scheduler Improvement (#15449)

  • [Improvement][E2E] add e2e javatask case (#15469)

  • [Enhancement][API]Enhance mysql connection properties (#15433)

  • [Improvement][E2E]e2e improve  add workflow httpTask e2e case (#15420)

  • Add config for defaultTenantEnabled (#15391)

  • Use DefaultUncaughtExceptionHandler to log the uncached exception (#15496)

  • adjust the sequence of alarm group and add validate (#15382)

  • Use Druid to split sql (#15367)

  • optimize add select filter (#15378)

  • [Improvement][Helm] using helm-docs to generate docs automatically (#15299)

  • [Improvement][K8S] Custom label of a K8S task can be passed to the pod (#15369)

  • Optimize server startup log (#15362)

  • [Improvement][E2E] support e2e compose v2 fix code style (#15325)

  • [Improvement] Ensure that HttpUtils can only get result from certification URL (#15288)

  • delete debugger (#15316)

  • Set maxRows in SqlTask (#15342)

  • [Feature-15146][dolphinscheduler-task-sqoop] add sqoop source/target type (#15146)

  • [Feature-15248][dolphinscheduler-alert-plugins] add alert plugin aliyun-voice (#15248)

  • [Improvement-15260][dolphinscheduler-datasource-hana] add hana  related dependencies (#15260)

  • fail-fast for dependent check (#15197)

  • [Improvement] Move delay calculation to Master (#15278)

  • Add dolphinscheduler-extract-common module (#15266)

  • Support parse task output params under multiple log (#15244)

  • [Improvement-15009][Parameter] Change project parameter value to text (#15010)

  • Remove spring cache for dao (#15184)

  • [Improvement] Clean up Scheduler logic (#15198)

  • [Improvement][Alert] Add a test send feature when creating an alert instance (#15163)

  • [Improvement][Helm] support task type filter (#15179)

  • [Improvement][Resource Center] Display brief file name in file-details page (#15137)

  • [Improvement][Alert] Add timeout params for HTTP plugin (#15174)

  • [feature#14654] alert-spi support prometheus alertmanager (#15079)

  • [Improvement][K8S] Remove ResourceQuota (#14991)

  • [Improvement] Refactoring K8S task plugin with connections managed in connection center (#14977)

  • [DSIP-19] Support sagemaker connections in the connection center, as well as external connections to the connection center in sagemaker tasks (#14976)

  • [DSIP-19] Support zeppelin connections in the connection center, as well as external connections to the connection center in zeppelin tasks (#14434)

  • [Feature-14832][Listener]Implementation of Listener Mechanism (#14981)

  • Remove mapper usage in tools (#15073)

  • [Feature-14678][Master][UI]Dependent task parameter passing (#14702)

  • Add IT for mysql5/postgresql16 initialize/upgrade (#15063)

  • Add IT for dolphinscheduler-tools module (#15043)

  • Set kubectl version to v1.28.3 (#15053)

  • Add dolphinscheduler-dao-plugin module (#15019)

  • [improvement][Resources] Improve details page return to the previous list page (#14951)

  • [Improvement][Alert] Alert plugin enhance fail message (#15024)

  • [Improvement][Registry][Jdbc] Add jdbc registry config in helm charts (#14431)

  • [Improvement][Master] Calculate the remainTime then we set the delay execution. (#15012)

Document

  • [Doc][Docker] fix typo on start with docker (#15534)

  • [Doc] remove skywalking, update note (#15028)

  • Change download url in backend.yml (#15526)

  • [Doc][K8S] Add DS K8S Operator into k8s deployment character (#15516)

  • Add guideline link into DolphinScheduler mail list (#15447)

  • Remove unused cache-evict.png (#15220)

  • [Doc-15500][Task] Update cli opts of spark and flink (#15501)

  • doc write wrong,should be MinIO it's not MinION (#15395)

  • [Doc]remove temporary markdown comments (#15385)

  • doc: Classify docs to avoid misleading (#15282)

  • Add deploy on Terraform on README (#15189)

  • Modify the documentation that python task will not work properly when 'n' indicates the presence of a variable and needs to use 'repr(value)' (#15145)

  • [Docs] fix typo (#15032)

Chore

  • merge schema 330 into 321 and change docs (#15582)

  • Set the workflow instance ready state to running in failover (#15572)

  • cp: Reduce the size of tarball to continue ASF release (#15004)

  • chore: Docs change for 3.2.1 release (#15539)

  • [DS-15489][style]rename the vo object suffix (#15504)

致谢

感谢所有贡献者的辛勤付出,特别是以下成员(排名不分先后):

致谢名单

AliceXiaoLuGallardotclovelll
BaiJvJohn Huangdavidzollo
CommandCVJohnZpdbac
DyqerOrange-Summerdestroydestiny
EricGao888Radeitydevosend
SbloodySSusurHefuchanghai
aiwenmoarlendpimizao
c3Vucaishunfengimp2002
izualzhykevinkelinlenboo
lgcareerliunaijieliyou
lizhengleily109974pegasas
qingwlireelerickchengx
ruanwenjunsdhzwcsleo
tangjiaolongxdu-chenrjxinxingi
xjlgodxujiaqiangzhanqian
zhaohehuhuzhihuasuzhongjiajie
zhuangchongzhutong6688

本文由 白鲸开源科技 提供发布支持!