66dcf2376a
Optimize some code ( #8324 )
3 years ago
4ba6d667fc
[Feature][DataQuality] Add Data quality Module #4283 ( #6718 )
...
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
* fix Constants bug
* remove the unused class
* add unit test
* fix code style error
* add unit test
* refactor data quality common entity
* fix code style error
* add unit test
* close e2e test
* fix code smell bug
* modify dataquality enum value to 14 in TaskType
* add data qualtiy task
* update
* add getDatasourceOptions interface
* fix checkstyle
* close e2e test
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* add unit test and alter dataquality task result
* fix checkstyle
* fix process service test error
* add unit test and fix code smells
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* change execute sql type name
* revert ui pom.xml
* fix data quality task error
* fix checkstyle error
* fix dq task src_connector_type ui select bug
* fix spark rw postgresql bug
* change mysql driver scope
* fix form-create json bug
* fix code smell
* fix DolphinException Bug
* fix ui validate rule and Alert title
* fix target connection param bug
* fix threshold validate change
* add rule input entry index
* change statistic_comparison_check logic
* remove check type change
* add DateExpressionReplaceUtil
* fix null point expetion
* fix null point expetion
* fix test error
* add more sql driver
* fix test error and remove DateExprReplaceUtil
* add get datasource tables and columns
* add get datasource tables and columns
* remove hive-jdbc in pom.xml
* fix code smells
* update sql
* change the pom.xml
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
* update sql
* [Feature][DataQuality] Add data quality task ui (#5054 )
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* fix dq task src_connector_type ui select bug
* fix threshold validate change
* remove check type change
* add get datasource tables and columns
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality module (#4830 )
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* update
* close e2e test
* fix spark rw postgresql bug
* change mysql driver scope
* add more sql driver
* remove hive-jdbc in pom.xml
* change the pom.xml
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality task backend (#4883 )
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* refactor data_quality_module
* add header license
* data quality module refactor
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix code smell
* fix check style
* fix unit test error
* task statistics value add unique code
* fix unit test error
* fix checkstyle error
* fix checkstyle
* fix security hotspot
* fix unit test error
* fix security hotspot
* fix check
* add data quality task error handling
* fix unit test error
* add unit test
* add unit test
* optimize data quality result alert
* fix unit test
* fix sql script error
* fix bug
* update sql script
* fix checkstyle
* add license
* fix checkstyle
* fix checkstyle
* fix unit test
* add jacoco dependencies
* fix unit test
* fix unit test
* add jacoco dependencies
* add unit test
* add unit test
* add license
* fix checkstyle
* fix pom
* fix checkstyle
* fix checkstyle
* merge dev
* fix ui error
* fix pom error
* fix pom error
* fix test error
* fix test error
* mssql-jdbc exclude azure-keyvault
* fix test error
* merge dev and add unit test
* add notes
* rollback the CollectionUtils
* fix
* update sql
* fix
* fix
* fix query rule page error
* change dq.jar path
* fix sql error
* fix ui error
* fix(dq): jar path&task enum description
* add notes on DataQualityApplication
* fix dq result jump error
* fix(ui): page condition
* feat(ui): add show error output path
* change version
* remove all chinese word in sql
* merge
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
3 years ago
f88def8ef7
[IMPROVEMENT-8178] Add Netty processor in Spring container ( #8179 )
...
* Add Netty processor in Spring container
3 years ago
103ccb01da
change version to 2.0.4-SNAPSHOT ( #8188 )
3 years ago
b5fa54b6be
[Bug-7865][MasterServer] retry logic optimization ( #8156 )
...
* submit task optimization
* cloneAndReset
* update
* tolerant task restart when init
* fix snoar check
* fix test
* delete unuse file
* taskInstance key
* code style
* skip complete when retry
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
90ea6ebb6f
[Improvement-7907][refactor] Optimization for query task instances list when build dag flow ( #7915 )
...
Optimization for query task instances list when build dag flow
This closes #7907
3 years ago
12d7e6261e
[Improvement][MasterServer] process complement data optimization #7925 ( #7976 )
...
* [Improvement][MasterServer] process complement data optimization #7925
* [Improvement][MasterServer] process complement data optimization #7925
Co-authored-by: shangeyao <sgy960921>
3 years ago
a8c592bd93
[Bug-7788][MasterServer] fix submit duplicate tasks sometimes when retry ( #7809 )
...
* [Bug-7788] fix submit duplicate tasks sometimes when retry
* [Bug-7788][MasterServer] fix submit duplicate tasks sometimes when retry
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
24a97fda75
[Bug] [dolphinscheduler-api]support workflow instance to definition ( #7930 )
...
* add task save and binds workflow
* add task update with upstream
* support workflow instance to definition
* fix ut
3 years ago
e4cdd82b7c
[Improvement][TaskLog] Unified task log ( #7831 )
...
* task logger
* update
* log format
* remove master config task-logger
* udpate log level
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
4b22ad6cf6
[Implement][MasterServer]TaskProcessor code optimization ( #7754 )
...
* task processor optimization
* fix test
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
707e3a6ec1
[Improvement][Logger] Logger server integrate into master and worker ( #6894 )
3 years ago
d3bd7309fb
[Bug-7686][Server]fix restart server after kill force ( #7688 )
...
* [DS-7686][Server]fix restart server after kill force
* update registry logic
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
9a1cea402e
[Fix-7062][MasterServer] NPE when run switch conditions sub-process ( #7687 )
3 years ago
8808c0a700
[Bug-7474][MasterServer] fix failover when node host is null ( #7475 )
...
* fix failover when node host is null
* add failover execute thread
* worker handle dead server
* fix task instance failover time check
* fix upgrade sql
* failover logic update
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
3af4d765c2
[Fix-7538] [server] Fix when there is a forbidden node in dag, the execution flow is abnormal ( #7613 )
...
* when there is a forbidden node in dag, the execution flow is abnormal
Co-authored-by: hongjie.li <hongjie.li@dmall.com>
3 years ago
e172015cd3
fix conflict pr 6997 ( #7571 )
...
Co-authored-by: shangeyao <sgy960921>
3 years ago
9e763ad0d1
[Feature-#6422] [api-server] task group queue ( #7491 )
...
* add task group
* modify task group
* pull dev
* add license header
* modify code style
* fix code style
* fix sql error
* fix error
* fix test
* fix test
* fix test
* fix test
* fix code style
* fix ut
* code style
* fix unit test
* test ut
* ut
* add unittest
* test ut
* modify back ut
* majorization code
* fix conflict
* fix ut
* add task group api
* reset file
* fix ut
* fix lost column
* fix ut
* fix ut
* fix ut
* fix ut
* delete duplicate code
* fix code style 、name
* fix ut
* fix mapper
Co-authored-by: wangxj <wangxj31>
3 years ago
1efa85ca27
[Fix-7534][Master] zookeeper failover error ( #7562 )
...
* fix bug_7534
* fix bug_7534
* fix bug_7534
* fix bug_7534
* fix bug_7534
* fix bug_7534
Co-authored-by: SbloodyS <sbloodys@qq.com>
3 years ago
9f56123a26
fix bug_7537 ( #7551 )
...
Co-authored-by: SbloodyS <sbloodys@qq.com>
3 years ago
b23f756c22
[Fix-7146][server]Fix the task processor thread is not safe. ( #7488 )
...
* Fix the task processor thread is not safe.
* fix code style
3 years ago
41bf1a955e
[Improvement-7213][MasterServer] execute thread pool code optimization ( #7258 )
...
* threadpool optimization
* threadpool params
* rebase dev
* ut check fix
* add return
* rebase dev
* event loop
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
1cbc86b9e6
add task-logger config ( #7423 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
99b8ec6492
[Bug-7319][MasterServer] fix taskNode NPE when switch else branch is empty ( #7320 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
939fa0c3ba
fix TaskResponseProcessor process NullPointerException ( #7318 )
3 years ago
67cc260d52
Split the components into individual package, refactor configurations to use Spring properties ( #7160 )
...
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
3 years ago
Wenjun Ruan
zixi0825
Kerwin
wind
天仇
Assert
JinYong Li
aaronlinv
lhjzmn
wangxj3
xiangzihao
kezhenxu94