f8d46a26c1
[Feature-11530] add state history for process instance ( #11757 )
...
* [Feature] add state history for process instance (#97 )
* add state history for process instance
* upsertProcessInstance
* remove unuse method
* fix UT
Co-authored-by: caishunfeng <534328519@qq.com>
2 years ago
67e7f88d8b
Refactor heart beat task, use json to serialize/deserialize ( #11702 )
...
* Refactor heart beat task, use json to serialize/deserialize
2 years ago
7ff34c3947
[Feature-7024] Add waiting strategy to support master/worker can recover from registry lost ( #11368 )
...
* Add waiting strategy to support master/worker can recover from registry lost
* throw exception when zookeeper registry start failed due to interrupted
2 years ago
0ca3086296
Fix workflow instance may failover many times due to doesn't check the restart time ( #11445 )
2 years ago
5fa3e7b1ed
Add pause to task instance status ( #11390 )
2 years ago
8774415197
Split ExecutionStatus to WorkflowExecutionStatus and TaskExecutionStatus ( #11340 )
2 years ago
3701a24d15
[Improvement][Task Log] Task status log print description instead of code ( #11009 )
...
* use execution status instead of status code
2 years ago
e1c78d1a49
Move utils, builder under master ( #11000 )
2 years ago
998e4d74dd
Kill task when we do master failover to avoid task lost ( #10997 )
2 years ago
df0416c193
Fix taskInstance's host is not worker nettyServer address ( #10926 )
...
* Fix taskInstance's host is not worker nettyServer address
* Remove unnecessary mock
2 years ago
cade66a9b6
[Fix-10827] Fix network error cause worker cannot send message to master ( #10886 )
...
* Fix network error cause worker cannot send message to master
2 years ago
b5653ea7f2
Complement numbers will run in a loop under the serial strategy fixed. ( #10862 )
...
* Complement numbers will run in a loop under the serial strategy fixed.
* e2e rerun
2 years ago
3f69ec8f28
[Fix-10842] Fix master/worker failover will cause status incorrect ( #10839 )
...
* Fix master failover will not update task instance status
* Add some failover log
* Fix worker failover will rerun task more than once
* Fix workflowInstance failover may rerun already success taskInstance
2 years ago
67d14fb7b3
[Fix-10785] Fix state event handle error will not retry ( #10786 )
...
* Fix state event handle error will not retry
* Use state event handler to deal with the event
2 years ago
35a10d092f
[Fix-10666] Workflow submit failed will still in memory and never retry ( #10667 )
...
* Workflow submit failed will still in memory and never retry
2 years ago
8f621ff98b
[Optimization] Calculate global parameter and local parameter at master. ( #10704 )
...
* Global parameter and local parameter calculation external expansion.
* k8s task ut fix.
* TimePlaceholderUtils import DateUtils fix
* follow the review comments to fix.
* follow the review comments to fix.
* e2e rerun
2 years ago
35b25da863
Validate master/worker config ( #10649 )
2 years ago
66624c5c86
[Bug] [Master] Worker failover will cause task cannot be failover ( #10631 )
...
* fix worker failover may lose event
2 years ago
b5184138fa
[Feature] Time function analysis extension. ( #10624 )
...
* Time function analysis extension.
* param add.
* clear useless logs and update method notes
* permission omission fix.
* extending time functions to optimize static methods
* e2e rerun.
2 years ago
b100f6c489
Remove the schedule thread in LowerWeightHostManager ( #10310 )
2 years ago
ad2646ff1f
Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe ( #10479 )
...
* Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe
2 years ago
e21d7b1551
[Feature][metrics] Add master, worker metrics ( #10326 )
...
* Add mater metrics
* fix UT
* Add url to mysql profile
* Add worker metrics
* Update grafana config
* Add system metrics doc
* Add process failover counter
* Add metrics image
* Change jpg to png
* Add command insert metrics
* Fix UT
* Revert UT
2 years ago
3ecbee3885
Optimize MasterServer, add MasterRPCService ( #10371 )
...
* Optimize MasterServer, avoid NPE
2 years ago
d80cf21456
Clean up unused dependencies and packaging issues ( #9944 )
2 years ago
49979c658e
[Fix-8828] [Master] Assign tasks to worker optimization ( #9919 )
...
* fix 9584
* master recall
* fix ut
* update logger
* update delay queue
* fix ut
* remove sleep
Co-authored-by: 进勇 <lijinyong@cai-inc.com>
Co-authored-by: JinyLeeChina <jiny.li@foxmail.com>
2 years ago
ee2b855ced
[BUG][TaskGroup] Task group does not take effect ( #10093 )
...
* fix 10092: Task group does not take effect
* fix 10092: Task group does not take effect
* fix 10092: Task group does not take effect
2 years ago
0cc0ee77fa
[Bug][Master] fix master task failover ( #10065 )
...
* fix master task failover
* ui
2 years ago
a1b6b033ad
[Fix-9906] After the serial wait execution strategy stops the running workflow instance, the instance will be woken up and executed if there is a wait instance. ( #9907 )
...
* After the serial wait execution strategy stops the running workflow instance, the instance will be woken up and executed if there is a wait instance.
* clear logic
* Resource overloading
2 years ago
ebc4253d50
[fix][Service] BusinessTime should format with schedule timezone ( #9714 )
...
* BusinessTime should format with schedule timezone
* fix test error
* fix test error
* fix test error
2 years ago
3815a86a3b
[Improvement][Master] Fix typo for MasterTaskExecThreadTest ( #9513 )
2 years ago
69923546a1
[DS-9263][Improvement][master]optimize failover ( #9281 )
...
- add FailoverService.java
- move failover method from MasterRegistryClient to FailoverService
- move failover code from FailoverExecuteThread to FailoverService
This closes #9263
2 years ago
52ba2c6475
[Improvement-9288][Master] add task event thread pool ( #9293 )
...
* add task event thread
* license heander
* ci test
* delete unuse file
* fix CI test
Co-authored-by: caishunfeng <534328519@qq.com>
2 years ago
ddf1ff98fa
[Improvement-7697][Master/Worker] Change the task ack to runnning callback ( #8719 )
...
* rebase dev
* change task state to dispatch if dispatch success
* update task host when dispatch
* add dispatch task event
* test
* check tenant after enable auto create
* handle dispatch state
Co-authored-by: caishunfeng <534328519@qq.com>
2 years ago
fd76436d87
[Feature-#8318] [master-server] Sub process transfer parameters to father process ( #8522 )
...
* load param from subprocess
* unit test
* modify code style
Co-authored-by: wangxj <wangxj31>
2 years ago
a1bef43981
[Feature-8579][Task] Task back-end plugin optimization, new plugins only need to modify the plugin's own module ( #8584 )
...
* Task plug-in optimization, task-related classes are migrated to the task plugin module
* remove tasktype enum class.
* fix code style.
* fix code style
* fix code style
* fix ProcessServiceTest.testUpdateResourceInfo test class
* fix code style
* fix test task plugin manager check parameters null
* fix test ut
* update aws license
* change TaskTimeoutStrategy class
2 years ago
b0fc6e7a69
[Feature-5752][server]Add blocking task ( #6272 )
...
* fix python task execution error
* modify code style
* delete AbstractTask#setCommand
* migrate blocking task alert feature and common attribute
* blocking task migrate(complete base features)
* migrate blocking task test
* resolve TaskNode.java conflict
* fix package problem in BlockingParameters
* resolve duplicate code
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* optimise code based on pr#6283 in ds master repo
* add log
* modify log
* resolve conflicts
* change hardcode to variable
* Add blocking task UI
* refactor old code to compatable with the newest branch
* add override annotation
* remove old code
* remove never used packages
* add missing packages declaration
* Refine the i18n description
* add UT
* resolve some code smell problems
* add fetch-depth
* enable unit test in sonar analysis
* disable unit test
* remove unused packages
* resolve duplicate codes
* add logs when task status changed and rename some variables
* add BLOCK & READY_BLOCK status and corresponding processing code
* revoke UI
* revert unit-test
Co-authored-by: Jave-Chen <baicai.chen@gmail.com>
2 years ago
4ba6d667fc
[Feature][DataQuality] Add Data quality Module #4283 ( #6718 )
...
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
* fix Constants bug
* remove the unused class
* add unit test
* fix code style error
* add unit test
* refactor data quality common entity
* fix code style error
* add unit test
* close e2e test
* fix code smell bug
* modify dataquality enum value to 14 in TaskType
* add data qualtiy task
* update
* add getDatasourceOptions interface
* fix checkstyle
* close e2e test
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* add unit test and alter dataquality task result
* fix checkstyle
* fix process service test error
* add unit test and fix code smells
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* change execute sql type name
* revert ui pom.xml
* fix data quality task error
* fix checkstyle error
* fix dq task src_connector_type ui select bug
* fix spark rw postgresql bug
* change mysql driver scope
* fix form-create json bug
* fix code smell
* fix DolphinException Bug
* fix ui validate rule and Alert title
* fix target connection param bug
* fix threshold validate change
* add rule input entry index
* change statistic_comparison_check logic
* remove check type change
* add DateExpressionReplaceUtil
* fix null point expetion
* fix null point expetion
* fix test error
* add more sql driver
* fix test error and remove DateExprReplaceUtil
* add get datasource tables and columns
* add get datasource tables and columns
* remove hive-jdbc in pom.xml
* fix code smells
* update sql
* change the pom.xml
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
* update sql
* [Feature][DataQuality] Add data quality task ui (#5054 )
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* fix dq task src_connector_type ui select bug
* fix threshold validate change
* remove check type change
* add get datasource tables and columns
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality module (#4830 )
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* update
* close e2e test
* fix spark rw postgresql bug
* change mysql driver scope
* add more sql driver
* remove hive-jdbc in pom.xml
* change the pom.xml
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality task backend (#4883 )
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* refactor data_quality_module
* add header license
* data quality module refactor
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix code smell
* fix check style
* fix unit test error
* task statistics value add unique code
* fix unit test error
* fix checkstyle error
* fix checkstyle
* fix security hotspot
* fix unit test error
* fix security hotspot
* fix check
* add data quality task error handling
* fix unit test error
* add unit test
* add unit test
* optimize data quality result alert
* fix unit test
* fix sql script error
* fix bug
* update sql script
* fix checkstyle
* add license
* fix checkstyle
* fix checkstyle
* fix unit test
* add jacoco dependencies
* fix unit test
* fix unit test
* add jacoco dependencies
* add unit test
* add unit test
* add license
* fix checkstyle
* fix pom
* fix checkstyle
* fix checkstyle
* merge dev
* fix ui error
* fix pom error
* fix pom error
* fix test error
* fix test error
* mssql-jdbc exclude azure-keyvault
* fix test error
* merge dev and add unit test
* add notes
* rollback the CollectionUtils
* fix
* update sql
* fix
* fix
* fix query rule page error
* change dq.jar path
* fix sql error
* fix ui error
* fix(dq): jar path&task enum description
* add notes on DataQualityApplication
* fix dq result jump error
* fix(ui): page condition
* feat(ui): add show error output path
* change version
* remove all chinese word in sql
* merge
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
2 years ago
b5fa54b6be
[Bug-7865][MasterServer] retry logic optimization ( #8156 )
...
* submit task optimization
* cloneAndReset
* update
* tolerant task restart when init
* fix snoar check
* fix test
* delete unuse file
* taskInstance key
* code style
* skip complete when retry
Co-authored-by: caishunfeng <534328519@qq.com>
2 years ago
90ea6ebb6f
[Improvement-7907][refactor] Optimization for query task instances list when build dag flow ( #7915 )
...
Optimization for query task instances list when build dag flow
This closes #7907
2 years ago
4b22ad6cf6
[Implement][MasterServer]TaskProcessor code optimization ( #7754 )
...
* task processor optimization
* fix test
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
8808c0a700
[Bug-7474][MasterServer] fix failover when node host is null ( #7475 )
...
* fix failover when node host is null
* add failover execute thread
* worker handle dead server
* fix task instance failover time check
* fix upgrade sql
* failover logic update
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
b23f756c22
[Fix-7146][server]Fix the task processor thread is not safe. ( #7488 )
...
* Fix the task processor thread is not safe.
* fix code style
3 years ago
41bf1a955e
[Improvement-7213][MasterServer] execute thread pool code optimization ( #7258 )
...
* threadpool optimization
* threadpool params
* rebase dev
* ut check fix
* add return
* rebase dev
* event loop
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
939fa0c3ba
fix TaskResponseProcessor process NullPointerException ( #7318 )
3 years ago
67cc260d52
Split the components into individual package, refactor configurations to use Spring properties ( #7160 )
...
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
3 years ago
caishunfeng
Wenjun Ruan
zhuxt2015
WangJPLeo
kezhenxu94
JinYong Li
BaoLiang
gaojun2048
Paul Zhang
worry
wangxj3
Kerwin
Martin Huang
zixi0825
wind
天仇