caishunfeng
ddd4474189
change quartz executor to be managed by spring ( #8774 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
Martin Huang
b0fc6e7a69
[Feature-5752][server]Add blocking task ( #6272 )
...
* fix python task execution error
* modify code style
* delete AbstractTask#setCommand
* migrate blocking task alert feature and common attribute
* blocking task migrate(complete base features)
* migrate blocking task test
* resolve TaskNode.java conflict
* fix package problem in BlockingParameters
* resolve duplicate code
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* resolve code style
* optimise code based on pr#6283 in ds master repo
* add log
* modify log
* resolve conflicts
* change hardcode to variable
* Add blocking task UI
* refactor old code to compatable with the newest branch
* add override annotation
* remove old code
* remove never used packages
* add missing packages declaration
* Refine the i18n description
* add UT
* resolve some code smell problems
* add fetch-depth
* enable unit test in sonar analysis
* disable unit test
* remove unused packages
* resolve duplicate codes
* add logs when task status changed and rename some variables
* add BLOCK & READY_BLOCK status and corresponding processing code
* revoke UI
* revert unit-test
Co-authored-by: Jave-Chen <baicai.chen@gmail.com>
3 years ago
yimaixinchen
a72742b9de
remove unuse import jsonFormat ( #8548 )
3 years ago
xiangzihao
891a002bea
[Feature-#8373][MasterServer] Dependent tasks can re-run automatically in the case of complement ( #8496 )
...
* first add feature_8373
* fix code smell
* add blank line
* fix some problems
* fix unit test error
3 years ago
caishunfeng
84f2cfec39
[Feature][Server] Custom timezone - Add timezone handler unified ( #8477 )
...
* date convert of timezone
* remove @JsonFormat
* add unit test
* fix time preview in scheduler
* optimization & add env config
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
JinYong Li
8d0a0f9c32
[Bug] [API] fix releaseWorkflowAndSchedule api and relation binding api ( #8464 )
...
* fix relation unbinding bug
* fix 8463
3 years ago
xiangzihao
08f642c756
[Fix-8337][Master] Process instance can not be kill when task is failure and can be retry ( #8387 )
...
* fix bug_8337
* fix bug_8337
* fix bug_8337
* fix bug_8337
* fix bug_8337
* fix bug_8337
* fix bug_8337
* test
* fix dev_bug_8337
* fix dev_bug_8337
* fix dev_bug_8337
* fix bug_8337
3 years ago
qianli2022
151100562f
[Feature][api] Task can run on kubernetes - k8s client and namespace #8247 ( #8252 )
...
* add db operator for namespace
* add k8s client
* add k8s controller
* add namespace ui
* add licenses
* add kubernetes model known-dependencies
* add new licenses
* fix SonarCloud Code Analysis bug
* remove license for standard protocol
* fix variable MemoryStr to memoryStr
Co-authored-by: qianl4 <qianl4@cicso.com>
3 years ago
Wenjun Ruan
66dcf2376a
Optimize some code ( #8324 )
3 years ago
JinYong Li
65744d4624
[Cherry-pick-dev][API] fix relation api bug ( #8207 )
...
* pick 8111
* fix relation delete (#8190 )
* pick 8190
3 years ago
zixi0825
4ba6d667fc
[Feature][DataQuality] Add Data quality Module #4283 ( #6718 )
...
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
* fix Constants bug
* remove the unused class
* add unit test
* fix code style error
* add unit test
* refactor data quality common entity
* fix code style error
* add unit test
* close e2e test
* fix code smell bug
* modify dataquality enum value to 14 in TaskType
* add data qualtiy task
* update
* add getDatasourceOptions interface
* fix checkstyle
* close e2e test
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* add unit test and alter dataquality task result
* fix checkstyle
* fix process service test error
* add unit test and fix code smells
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* change execute sql type name
* revert ui pom.xml
* fix data quality task error
* fix checkstyle error
* fix dq task src_connector_type ui select bug
* fix spark rw postgresql bug
* change mysql driver scope
* fix form-create json bug
* fix code smell
* fix DolphinException Bug
* fix ui validate rule and Alert title
* fix target connection param bug
* fix threshold validate change
* add rule input entry index
* change statistic_comparison_check logic
* remove check type change
* add DateExpressionReplaceUtil
* fix null point expetion
* fix null point expetion
* fix test error
* add more sql driver
* fix test error and remove DateExprReplaceUtil
* add get datasource tables and columns
* add get datasource tables and columns
* remove hive-jdbc in pom.xml
* fix code smells
* update sql
* change the pom.xml
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
* update sql
* [Feature][DataQuality] Add data quality task ui (#5054 )
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* fix dq task src_connector_type ui select bug
* fix threshold validate change
* remove check type change
* add get datasource tables and columns
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality module (#4830 )
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* update
* close e2e test
* fix spark rw postgresql bug
* change mysql driver scope
* add more sql driver
* remove hive-jdbc in pom.xml
* change the pom.xml
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality task backend (#4883 )
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* refactor data_quality_module
* add header license
* data quality module refactor
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix code smell
* fix check style
* fix unit test error
* task statistics value add unique code
* fix unit test error
* fix checkstyle error
* fix checkstyle
* fix security hotspot
* fix unit test error
* fix security hotspot
* fix check
* add data quality task error handling
* fix unit test error
* add unit test
* add unit test
* optimize data quality result alert
* fix unit test
* fix sql script error
* fix bug
* update sql script
* fix checkstyle
* add license
* fix checkstyle
* fix checkstyle
* fix unit test
* add jacoco dependencies
* fix unit test
* fix unit test
* add jacoco dependencies
* add unit test
* add unit test
* add license
* fix checkstyle
* fix pom
* fix checkstyle
* fix checkstyle
* merge dev
* fix ui error
* fix pom error
* fix pom error
* fix test error
* fix test error
* mssql-jdbc exclude azure-keyvault
* fix test error
* merge dev and add unit test
* add notes
* rollback the CollectionUtils
* fix
* update sql
* fix
* fix
* fix query rule page error
* change dq.jar path
* fix sql error
* fix ui error
* fix(dq): jar path&task enum description
* add notes on DataQualityApplication
* fix dq result jump error
* fix(ui): page condition
* feat(ui): add show error output path
* change version
* remove all chinese word in sql
* merge
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
3 years ago
Wenjun Ruan
f88def8ef7
[IMPROVEMENT-8178] Add Netty processor in Spring container ( #8179 )
...
* Add Netty processor in Spring container
3 years ago
wind
b5fa54b6be
[Bug-7865][MasterServer] retry logic optimization ( #8156 )
...
* submit task optimization
* cloneAndReset
* update
* tolerant task restart when init
* fix snoar check
* fix test
* delete unuse file
* taskInstance key
* code style
* skip complete when retry
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
springmonster
e0a99dd909
[Bug-8053] Fix CronUtils.getMaxCycle return null ( #8086 )
3 years ago
天仇
90ea6ebb6f
[Improvement-7907][refactor] Optimization for query task instances list when build dag flow ( #7915 )
...
Optimization for query task instances list when build dag flow
This closes #7907
3 years ago
Wenjun Ruan
ecaeadea33
Fix global params cannot transport into sub_process ( #7969 ) ( #7973 )
3 years ago
JinYong Li
24a97fda75
[Bug] [dolphinscheduler-api]support workflow instance to definition ( #7930 )
...
* add task save and binds workflow
* add task update with upstream
* support workflow instance to definition
* fix ut
3 years ago
xiangzihao
7395b980d3
[Fix-7767]Task instance is not deleted while deleting the process instance ( #7768 )
...
* fix bug_7767
* fix bug_7767
* fix bug_7767
* fix bug_7767
* fix bug_7767
3 years ago
xiangzihao
fa906e7d01
[Fix-7753][Master] Failed to pass global parameters between parent and sub process ( #7755 )
...
* fix bug_7753
* fix bug_7753
* fix bug_7753
Co-authored-by: SbloodyS <sbloodys@qq.com>
3 years ago
aaronlinv
707e3a6ec1
[Improvement][Logger] Logger server integrate into master and worker ( #6894 )
3 years ago
calvin
6b5db0ac5b
fix this issue ( #7683 )
3 years ago
wind
8808c0a700
[Bug-7474][MasterServer] fix failover when node host is null ( #7475 )
...
* fix failover when node host is null
* add failover execute thread
* worker handle dead server
* fix task instance failover time check
* fix upgrade sql
* failover logic update
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
wind
d75f22781b
fix workflow copy ( #7659 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
xiangzihao
3d9d91ccc3
[Feature-7576][Master] Optimize complement task's date ( #7585 )
...
* feature_7576
* feature_7576
* feature 7576
* feature 7576
3 years ago
wangxj3
9e763ad0d1
[Feature-#6422] [api-server] task group queue ( #7491 )
...
* add task group
* modify task group
* pull dev
* add license header
* modify code style
* fix code style
* fix sql error
* fix error
* fix test
* fix test
* fix test
* fix test
* fix code style
* fix ut
* code style
* fix unit test
* test ut
* ut
* add unittest
* test ut
* modify back ut
* majorization code
* fix conflict
* fix ut
* add task group api
* reset file
* fix ut
* fix lost column
* fix ut
* fix ut
* fix ut
* fix ut
* delete duplicate code
* fix code style 、name
* fix ut
* fix mapper
Co-authored-by: wangxj <wangxj31>
3 years ago
wind
d3533851a0
[Bug-7333][MasterServer] fix duplicate key task definition ( #7335 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
Kerwin
f1b85cf892
Fix the missing project information in the process instance timeout alarm ( #7324 )
3 years ago
Kerwin
3fadfca085
fix in the process alarm message, projectid is 0 ( #7298 )
3 years ago
kezhenxu94
67cc260d52
Split the components into individual package, refactor configurations to use Spring properties ( #7160 )
...
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
3 years ago
wind
a80a7b3fcf
[Feature-6963][MasterServer] unified cache manager ( #7187 )
...
* unified cache
* reduce db select
* checkstyle
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
JinYong Li
2792c35d68
[Bug] [API] queryProcessDefinitionByCode bug ( #7221 )
3 years ago
wind
ba2b2a67c2
[Feature-6988][MasterServer] add cache manager for workflow ( #7090 )
...
* add cache manager for workflow
* [DS-6988][MasterServer] add cache manager for workflow
* cache evict code optimization
* test
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
wangxj3
2b519f31e9
[Feature-#6422] task group queue ( #6722 )
...
* add task group
* modify task group
* pull dev
* add license header
* modify code style
* fix code style
* fix sql error
* fix error
* fix test
* fix test
* fix test
* fix test
* fix code style
* fix ut
* code style
* fix unit test
* test ut
* ut
* add unittest
* test ut
* modify back ut
* majorization code
* fix conflict
* fix ut
Co-authored-by: wangxj <wangxj31>
3 years ago
JinYong Li
b265297fc8
[Improvement][API] fix 6772 new api bug ( #7098 )
...
* add processDefinition releaseWorkflowAndSchedule
* add ProcessTaskRelationServiceImpl.moveTaskProcessRelation
* add dependency and subprocess judgement
* fix api bug
* fix codestyle
3 years ago
Sunny Lei
0dcff1425a
[DS-5229][feat] Implement server port custom config ( #6947 )
...
* [DS-5229][fix] server port custom config
This closes #5229
* [DS-5229][feat] Implement server port custom config
This closes #5229
* [Bug] [readme] Error link to Docker and k8s in readme apache#6802 (#6862 )
* [Bug] [readme] Error link to Docker and k8s in readme #6802
1、modify the error link
* [Bug] [readme] Error link to Docker and k8s in readme #6802
1、modify the error link in readme_zh_cn.md
* [DS-6829][WorkerServer] skip create log dir and print log in dryRun model (#6852 )
Co-authored-by: caishunfeng <534328519@qq.com>
* [DS-5229][feat] Implement server port custom config
Modify review suggestion
This closes #5229
Co-authored-by: GaoTianDuo <gaotianduo_yewu@cmss.chinamobile.com>
Co-authored-by: wind <caishunfeng@users.noreply.github.com>
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
wind
8e93939121
[DS-6963][MasterServer]add cache manager for tenant and user ( #6974 )
...
* cache manager for tenant and user
* add test
* fix checkstyle
* add dependencies
* fix sonar check
* use spring boot cache configuration
* add license
* check style
* check style
* add license
* change proxy name to processor
* add license header
* check style
* add license file
* add license file
* add license
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
wind
28872d8706
[DS-6891][MasterServer] reduce db operation when process instance running ( #6904 )
...
* remove taskInstanceCacheManager
* remove taskInstanceCacheManager
* [DS-6891][MasterServer] reduce db operation when process instance running
* [DS-6891][MasterServer] reduce db operation when process instance running
* fix test
* fix Transactional method call
* checkstyle
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
JinYong Li
4bec792160
[Fix][Common] rewrite code generate,fix bit shift ( #6914 )
...
* rewrite code generate,fix bit shift
* fix ut
* add algorithm from licenses file
* add algorithm from licenses file
* add algorithm from licenses file
* add algorithm from licenses file
* add algorithm from licenses file
* fix ut
3 years ago
wind
595e4843d0
[DS-6849][MasterServer] fetch more commands and handle in parallel ( #6850 )
...
* [DS-6849][MasterServer] fetch more commands and handle in parallel
* add return
* handle command with check
* add test
* delete master prefix
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
OS
1160b53940
[Feature][Master] change sub process id to code ( #6858 )
...
* feature 6597 change sub process id to code
* feature 6597 change sub process id to code
3 years ago
OS
880c1d28ff
[Feature][master] change sub process id to code. ( #6846 )
...
* 6597 change sub process id to code.
* 6597 change sub process id to code.
* 6597 change sub process id to code.
* 6597 change sub process id to code.
* 6597 change sub process id to code.
* 6597 change sub process id to code.
* code style
* fix ut process
* fix ut process
3 years ago
kezhenxu94
4114cb07f6
Add actuator and enable metrics, add Grafana configs ( #6840 )
3 years ago
mask
a998415afe
[Feature][datasource] Change DataSource Connection Pool from Druid to HiKariCP ( #6490 ) ( #6828 )
...
* [Feature][datasource] Change DataSource Connection Pool from Druid to HiKariCP (#6490 )
* fix pom dep scope
* filter alert
* filter alert
* filter alert
* fix checkstyle
* remove hikaricp version
* update pom
* add properties
* add properties
* rename properties
* fix e2e fail
* rename properties
3 years ago
Jiajie Zhong
6045728428
[ci] Add checkstyle prohibit rule System.out.println ( #6778 )
...
* [ci] Add checkstyle prohibit rule System.out.println
* Correct RegexpSingleline place
3 years ago
kezhenxu94
63602bf77c
Clean up the utility codes ( #6732 )
3 years ago
OS
24bd8a16e1
[Bug][Master] The task of dependent should use taskCode ( #6730 )
...
* fix #6682 The task of dependent should use taskCode
* fix #6682 The task of dependent should use taskCode
3 years ago
Wenjun Ruan
0afb780718
[Improvement-5807][Dao] Use h2 in UT ( #6500 )
...
* Use H2 in UT
* change scope
* Change profile name to h2
* Add datasource-h2.properties, datasource-postgresql.properties
* Fix UT
* Resolve conflict
* Remove startDatabase
3 years ago
kezhenxu94
5400abc74f
Refactor registry plugin and simplify its usage ( #6712 )
3 years ago
wangxj3
dc8b87e473
[Feature-#6268][server-master] Serial execte process ( #6185 )
...
* add serial processInstance
* fix bug
* add test
* fix code style
* fix style code
* add sql
* fix sql error
* add api
* add test
* fix code style
* modify api
* delete column , fix mapper
* fix unimport
* fix test
* fix bug of missing param for Python
* fix code style
* fix test
* fix test
Co-authored-by: wangxj <wangxj31>
3 years ago
wind
1ebab356f3
[DS-6660][Improvement] Change the registry default timeout ( #6663 )
...
* [DS-6660][Improvement] Change the registry default timeout
* [DS-6660][Improvement] Change the registry default timeout
* fix Double to Integer
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago