03e1e6fe45
[Bug] [Worker] Optimize the getAppId method to avoid worker OOM when kill task ( #11701 )
...
* Fix kill job may cause worker oom
2 years ago
3f2ca7bca3
[Fix-9980] [Server] fix heartBeatTaskCount bug ( #11232 )
...
* fix heartBeat bug
* modify class name
* fix conflict
Co-authored-by: JinyLeeChina <jiny.li@foxmail.com>
2 years ago
c110f4acae
Refresh server startup time when recovery from waiting ( #11595 )
2 years ago
7ff34c3947
[Feature-7024] Add waiting strategy to support master/worker can recover from registry lost ( #11368 )
...
* Add waiting strategy to support master/worker can recover from registry lost
* throw exception when zookeeper registry start failed due to interrupted
2 years ago
8774415197
Split ExecutionStatus to WorkflowExecutionStatus and TaskExecutionStatus ( #11340 )
2 years ago
5e9c7dad23
Add dolphinscheduler-bom to manage the dependency version ( #11025 )
2 years ago
2be1d4bf0a
Fix worker cannot shutdown due to resource close failed or heart beat check failed ( #10979 )
...
* Use try-with-resource to close resource, and add heart error threshold to avoid worker cannot close due to heart beat check failed
* Move heartbeat error threshold to applicaiton.yml
2 years ago
66624c5c86
[Bug] [Master] Worker failover will cause task cannot be failover ( #10631 )
...
* fix worker failover may lose event
2 years ago
3ecbee3885
Optimize MasterServer, add MasterRPCService ( #10371 )
...
* Optimize MasterServer, avoid NPE
2 years ago
d80cf21456
Clean up unused dependencies and packaging issues ( #9944 )
2 years ago
fd6b43bc81
[Dev] Switch version in pom.xml to dev-SNAPSHOT ( #9223 ) ( #9299 )
3 years ago
a1bef43981
[Feature-8579][Task] Task back-end plugin optimization, new plugins only need to modify the plugin's own module ( #8584 )
...
* Task plug-in optimization, task-related classes are migrated to the task plugin module
* remove tasktype enum class.
* fix code style.
* fix code style
* fix code style
* fix ProcessServiceTest.testUpdateResourceInfo test class
* fix code style
* fix test task plugin manager check parameters null
* fix test ut
* update aws license
* change TaskTimeoutStrategy class
3 years ago
84f2cfec39
[Feature][Server] Custom timezone - Add timezone handler unified ( #8477 )
...
* date convert of timezone
* remove @JsonFormat
* add unit test
* fix time preview in scheduler
* optimization & add env config
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
4ba6d667fc
[Feature][DataQuality] Add Data quality Module #4283 ( #6718 )
...
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
* fix Constants bug
* remove the unused class
* add unit test
* fix code style error
* add unit test
* refactor data quality common entity
* fix code style error
* add unit test
* close e2e test
* fix code smell bug
* modify dataquality enum value to 14 in TaskType
* add data qualtiy task
* update
* add getDatasourceOptions interface
* fix checkstyle
* close e2e test
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* add unit test and alter dataquality task result
* fix checkstyle
* fix process service test error
* add unit test and fix code smells
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* change execute sql type name
* revert ui pom.xml
* fix data quality task error
* fix checkstyle error
* fix dq task src_connector_type ui select bug
* fix spark rw postgresql bug
* change mysql driver scope
* fix form-create json bug
* fix code smell
* fix DolphinException Bug
* fix ui validate rule and Alert title
* fix target connection param bug
* fix threshold validate change
* add rule input entry index
* change statistic_comparison_check logic
* remove check type change
* add DateExpressionReplaceUtil
* fix null point expetion
* fix null point expetion
* fix test error
* add more sql driver
* fix test error and remove DateExprReplaceUtil
* add get datasource tables and columns
* add get datasource tables and columns
* remove hive-jdbc in pom.xml
* fix code smells
* update sql
* change the pom.xml
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
* update sql
* [Feature][DataQuality] Add data quality task ui (#5054 )
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* fix dq task src_connector_type ui select bug
* fix threshold validate change
* remove check type change
* add get datasource tables and columns
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality module (#4830 )
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* update
* close e2e test
* fix spark rw postgresql bug
* change mysql driver scope
* add more sql driver
* remove hive-jdbc in pom.xml
* change the pom.xml
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality task backend (#4883 )
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* refactor data_quality_module
* add header license
* data quality module refactor
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix code smell
* fix check style
* fix unit test error
* task statistics value add unique code
* fix unit test error
* fix checkstyle error
* fix checkstyle
* fix security hotspot
* fix unit test error
* fix security hotspot
* fix check
* add data quality task error handling
* fix unit test error
* add unit test
* add unit test
* optimize data quality result alert
* fix unit test
* fix sql script error
* fix bug
* update sql script
* fix checkstyle
* add license
* fix checkstyle
* fix checkstyle
* fix unit test
* add jacoco dependencies
* fix unit test
* fix unit test
* add jacoco dependencies
* add unit test
* add unit test
* add license
* fix checkstyle
* fix pom
* fix checkstyle
* fix checkstyle
* merge dev
* fix ui error
* fix pom error
* fix pom error
* fix test error
* fix test error
* mssql-jdbc exclude azure-keyvault
* fix test error
* merge dev and add unit test
* add notes
* rollback the CollectionUtils
* fix
* update sql
* fix
* fix
* fix query rule page error
* change dq.jar path
* fix sql error
* fix ui error
* fix(dq): jar path&task enum description
* add notes on DataQualityApplication
* fix dq result jump error
* fix(ui): page condition
* feat(ui): add show error output path
* change version
* remove all chinese word in sql
* merge
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
3 years ago
103ccb01da
change version to 2.0.4-SNAPSHOT ( #8188 )
3 years ago
e4cdd82b7c
[Improvement][TaskLog] Unified task log ( #7831 )
...
* task logger
* update
* log format
* remove master config task-logger
* udpate log level
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
707e3a6ec1
[Improvement][Logger] Logger server integrate into master and worker ( #6894 )
3 years ago
67cc260d52
Split the components into individual package, refactor configurations to use Spring properties ( #7160 )
...
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
3 years ago
3a351da752
fix failover logic ( #7275 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
f577b5bd30
to #7228 ( #7249 )
...
Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com>
3 years ago
a80a7b3fcf
[Feature-6963][MasterServer] unified cache manager ( #7187 )
...
* unified cache
* reduce db select
* checkstyle
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
ff9bc806ac
[Bug-7206] [MasterServer] fix process isntance always running when task timeout ( #7207 )
...
* fix timeout
* add task timeout map to avoid repeated timeout event
* split task check list for retry and timeout
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
36ebfe0984
[Implement][server]TaskProcessor code optimization ( #7146 )
...
* TaskProcessor code optimization
* fix code style
* update Collectors.toMap
3 years ago
d83735ab51
[Fix-7037][dolphincheduler-server] process instance can not be killed when complement data ( #7140 )
...
* fix bug 7037
* update processComplementData
Co-authored-by: SbloodyS <sbloodys@qq.com>
3 years ago
8d39bf14b1
[Fix-7093][dolphincheduler-server] fix dependent node stuck in running ( #7116 )
...
* fix bug 7093
* remove redundant else
Co-authored-by: SbloodyS <sbloodys@qq.com>
3 years ago
af9ed35295
[Feature][MasterServer] add cache for workGroup and schedule #6987 ( #7161 )
...
* to #6987
* to #6987
* to #6987
* to #6987
* to #6987
* to #6987 : @Param -> @Name
* to #6987 : fix Sonar
* to #6987 : fix Sonar
Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com>
3 years ago
ba2b2a67c2
[Feature-6988][MasterServer] add cache manager for workflow ( #7090 )
...
* add cache manager for workflow
* [DS-6988][MasterServer] add cache manager for workflow
* cache evict code optimization
* test
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
2b519f31e9
[Feature-#6422] task group queue ( #6722 )
...
* add task group
* modify task group
* pull dev
* add license header
* modify code style
* fix code style
* fix sql error
* fix error
* fix test
* fix test
* fix test
* fix test
* fix code style
* fix ut
* code style
* fix unit test
* test ut
* ut
* add unittest
* test ut
* modify back ut
* majorization code
* fix conflict
* fix ut
Co-authored-by: wangxj <wangxj31>
3 years ago
9fffbdb316
add TaskProcessorFactoryTest Ignore. ( #7130 )
3 years ago
c42531e39c
to #7108 ( #7109 )
3 years ago
2ed3c0c3a7
Recover unit tests that have not been run for some time ( #7054 )
3 years ago
0dcff1425a
[DS-5229][feat] Implement server port custom config ( #6947 )
...
* [DS-5229][fix] server port custom config
This closes #5229
* [DS-5229][feat] Implement server port custom config
This closes #5229
* [Bug] [readme] Error link to Docker and k8s in readme apache#6802 (#6862 )
* [Bug] [readme] Error link to Docker and k8s in readme #6802
1、modify the error link
* [Bug] [readme] Error link to Docker and k8s in readme #6802
1、modify the error link in readme_zh_cn.md
* [DS-6829][WorkerServer] skip create log dir and print log in dryRun model (#6852 )
Co-authored-by: caishunfeng <534328519@qq.com>
* [DS-5229][feat] Implement server port custom config
Modify review suggestion
This closes #5229
Co-authored-by: GaoTianDuo <gaotianduo_yewu@cmss.chinamobile.com>
Co-authored-by: wind <caishunfeng@users.noreply.github.com>
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
100a9ba3fc
to #7055 : tenant-auto-create: true ( #7056 )
...
Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com>
3 years ago
c2585a1170
[BUG] master task log can't output repair ( #7029 )
...
* update task_definition_log field typ
update task_definition_log field task_params type,modify text to longtext to be consistent with the main table
* update t_ds_task_definition_log field task_params
update t_ds_task_definition_log field task_params
* [FIX:211128] master task log Can't output repair
Co-authored-by: crane <zhangbq3306@163.com>
3 years ago
be3bd4c83d
[DS-7004][MasterServer]fix master still work when it lose zk connection ( #7045 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
a0955a078a
[Fix-7019][MasterServer] Fix the issue with failing to save and run the process that contains the switch task. ( #7046 )
3 years ago
8e93939121
[DS-6963][MasterServer]add cache manager for tenant and user ( #6974 )
...
* cache manager for tenant and user
* add test
* fix checkstyle
* add dependencies
* fix sonar check
* use spring boot cache configuration
* add license
* check style
* check style
* add license
* change proxy name to processor
* add license header
* check style
* add license file
* add license file
* add license
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
e6fe39ea47
[DS-6961][MasterServer] batch dispatch ( #6962 )
...
* [DS-6961][MasterServer] batch dispatch
* fix test
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
2de4a3cbb5
Add a demo docker-compose for testing the meter module ( #6938 )
3 years ago
e6239e808b
[DS-6829][WorkerServer] skip create log dir and print log in dryRun model ( #6852 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
28872d8706
[DS-6891][MasterServer] reduce db operation when process instance running ( #6904 )
...
* remove taskInstanceCacheManager
* remove taskInstanceCacheManager
* [DS-6891][MasterServer] reduce db operation when process instance running
* [DS-6891][MasterServer] reduce db operation when process instance running
* fix test
* fix Transactional method call
* checkstyle
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
595e4843d0
[DS-6849][MasterServer] fetch more commands and handle in parallel ( #6850 )
...
* [DS-6849][MasterServer] fetch more commands and handle in parallel
* add return
* handle command with check
* add test
* delete master prefix
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
653eae2419
[BUG][Master] process cannot finish and its status is always running ( #6886 )
...
* fix 6882: process cannot finish and its status is always running
* fix 6882: process cannot finish and its status is always running
3 years ago
f564687a89
[Improvement][Log] optimize task log path to reduce task running time ( #6731 )
...
* [Improvement][Log] optimize task log path to reduce task running time
* Reduce the creation of log directories
Co-authored-by: Kirs <acm_master@163.com>
3 years ago
94f6c5c27e
fix complement data retry bug; issues: [Bug] [Master] A bug on task retry mechanism #6613 ( #6843 )
...
Co-authored-by: yangqiyu <yangqiyu@deepexi.com>
3 years ago
4114cb07f6
Add actuator and enable metrics, add Grafana configs ( #6840 )
3 years ago
6045728428
[ci] Add checkstyle prohibit rule System.out.println ( #6778 )
...
* [ci] Add checkstyle prohibit rule System.out.println
* Correct RegexpSingleline place
3 years ago
5741c758b3
[DS-6771][MasterServer] fix failover worker interrupt ( #6801 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
Co-authored-by: Kirs <acm_master@163.com>
Co-authored-by: OS <29528966+lenboo@users.noreply.github.com>
3 years ago
e36d18b588
[Fix-6792] resolve API and alert port conflicts ( #6792 ) ( #6815 )
...
* [Fix-6792]Multiple ApplicationRunners on classpath make SpringApplication.run only run one (#6792 )
3 years ago
11844415b0
Refactor datasource and task plugins and simplify its usage ( #6767 )
3 years ago
Wenjun Ruan
JinYong Li
kezhenxu94
Eric Gao
Kerwin
caishunfeng
zixi0825
wind
aaronlinv
zwZjut
SbloodyS
wangxj3
Sunny Lei
zhang
Hua Jiang
OS
chouc
KyoYang
Jiajie Zhong
mask