08335b1032
[Improvement][Task] Improved way to collect yarn job's appIds ( #12197 )
...
* Provide aop way as an optional way to collect yarn job's applicationId, and import new module `dolphinscheduler-aop` to place the aop code.
* Add user property `appId.collect` for user to decide how to collect applicationId.
* Add new environment configuration for each type of yarn tasks to support aop in `dolphinscheduler_env.sh`
* Update docs to declare how to use aop way.
* Update `LogUtils` to support fetch applicationId in different ways based on the user property.
Co-authored-by: gabrywu <gabrywu@apache.com>
2 years ago
e6da1ccf81
Add worker-group-refresh-interval in master config ( #12601 )
...
* Add worker-group-refresh-interval in master config
* Set interval cannot smaller than 10s
* Update dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/config/MasterConfig.java
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2 years ago
065d5caccc
Only expose necessary actuator endpoints ( #12571 )
2 years ago
489e7fe4e2
[Feature-10495][Resource Center] Resource Center Refactor ( #12076 )
...
* resource center refactor - S3 services connection
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2 years ago
04aa125ba2
[Refactor]migrate some task dao functions from ProcessServiceImpl to dao modules. ( #12505 )
...
* migrate some task DAO to dao modules.
* Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/repository/impl/ProcessInstanceMapDaoImpl.java
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2 years ago
4aa9d60683
[Bug][Master] Add the aws-java-sdk-s3 jar package to the master module ( #12259 ) ( #12512 )
...
Co-authored-by: qiquan.li <qiquan.li@linkflowtech.com>
2 years ago
dedff70f90
[Improvement] Merge spi.utils into common.utils ( #12502 )
...
* improve JSONUtil
* merge spi.utils into common.utils
* split constants && remove some copy utils method
* remove StringUtils
2 years ago
b936b882bb
[Refactor] Migrate all command-related interface functions from ProcessServiceImpl ( #12474 )
...
* migrate all command-related interface functions to CommonService
2 years ago
d46e4659d9
[Improvement][Style] Fix remaining style errors and wildcard imports ( #12412 )
...
* Fix remaining style errors and wildcard imports
* Upgrade spotless to the latest stable version, exclude .github folder from spotless check and fix dead link
* Fix spotless matching scope of markdown files
2 years ago
38b643f69b
make sure all failed task will save in errorTaskMap ( #12424 )
2 years ago
4ea7eec750
Use temurin Java docker images instead of deprecated ones ( #12334 )
2 years ago
bc80f34056
[Migrate][Test] Migrate all UT cases from jUnit 4 to 5 in master, worker, and remote modules ( #12362 )
...
* migrate junit4 -> junit5 in master, worker, and remote modules
2 years ago
2f37da0dbc
[Improvement][Test] Block the usage of powermock and move mockito dependencies from sub-modules to root pom ( #12311 )
...
* move mockito dependencies from sub-modules to root pom
* Add check in CI to block the usage of powermock
2 years ago
db1d319159
Reorganize some classes in common module, remove duplicate classes ( #12321 )
2 years ago
05099bd8f7
[Improvement-12296][Test] Close static mock ( #12300 )
2 years ago
165b9a58de
[DS-12131][master] Optimize the log printing of the master module acc… ( #12152 )
...
* [DS-12131][master] Optimize the log printing of the master module according to the log specification.
2 years ago
c87f6d89a4
[fix#12195] Change date to timestamp to support cross diff time zones ( #12239 )
...
* fix DS support cross time zone, use timestamp replace date
2 years ago
55388be21f
[Improvement][Master] LowerWeightRoundRobin doSelect may be null ( #12159 )
2 years ago
40090d459e
[Improvement] remove log-server and server module ( #12206 )
...
* remove log-server module
* remove dolphinscheduler-server module
2 years ago
aacabc6ec7
Remove workerGroup in registry ( #12217 )
2 years ago
e27c79974d
[fix-#11753] send alert error alert data id ( #11774 )
...
* [fix-#11753] send alert error alert data id
Co-authored-by: fuchanghai <changhai.fu@marketingforce.com>
2 years ago
d526dd89f6
[hotfix][UI] Remove SURVEIL and NEXT_LOOP in ui ( #12160 )
...
* Remove SURVEIL and NEXT_LOOP in ui
* Fix UT
2 years ago
cc82206c33
Remove powermock in dolphinscheduler-master module ( #12143 )
...
* Remove powermock related code in dolphinscheduler-master module
* Remove redundant comments, useless code and unnecessary stubbings
2 years ago
cf5a8894e2
[BUG][BE-12113]fix htrace dependency conflicts ( #12126 )
...
* fix htrace dependency conflicts
* replace htrace-core with hbase-noop-htrace
2 years ago
ada7cf71d5
[fix#12000]Cannot remove the WorkerGroup from the master service ( #12050 )
...
* [Bug] [Master] Cannot remove the WorkerGroup from the master service. #12000
* remove unnecessary locks
* Update dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/registry/ServerNodeManager.java
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2 years ago
5b384f3fab
[Feature-11473][Task]Support test task ( #11670 )
...
* [improve] support test tasks
* [improve] support test tasks
* [improve] support test tasks
* [improve] support test tasks
* [improve] support test tasks
* Update TaskExecuteThread.java
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* try solve e2e q
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* Update DataSource.java
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* Update messages_zh_CN.properties
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* Update messages.properties
* Update messages_en_US.properties
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
* [Feature] support test tasks
2 years ago
e938fdbe96
[Bug] [Master] WorkflowExecuteRunnable will face a infinite loop #11838 ( #11864 )
...
* [Bug] [Master] WorkflowExecuteRunnable will face a infinite loop #11838
2 years ago
6868876a29
fix workflow keep running when task fail ( #11930 )
2 years ago
d3a77c68e6
Make LogServiceClient Singleton ( #11777 )
2 years ago
009cb68011
[Improvement-11834] Upgrade docker base image to support python3.9 ( #11835 )
...
* use openjdk:8-jre-slim-bullseye instead of buster to support python3.9
* use openjdk:8-jre-slim-bullseye instead of buster across all components
2 years ago
3ca9680b20
[Fix][Monitor]Monitor UI not show DisakAvailable and MemoryUsage correctly ( #11870 )
2 years ago
105773f54d
Remove htrace dependencies ( #11881 )
...
* Remove htrace dependencies
* Update pom.xml
2 years ago
3664d85143
[Improve] Upgrade Hadoop to 3.2.4 ( #11832 )
2 years ago
2e61c76c22
[Improvement] Add remote task model ( #11767 )
2 years ago
efedc9a53c
Bump up dependencies to fix cves ( #11765 )
2 years ago
444c7e6eb7
[Improvement] Replace commons-lang 2 function invocations with commons-lang3 ( #11810 )
...
* [Improvement] Replace commons-lang 2 function invocations with commons-lang3
2 years ago
f8d46a26c1
[Feature-11530] add state history for process instance ( #11757 )
...
* [Feature] add state history for process instance (#97 )
* add state history for process instance
* upsertProcessInstance
* remove unuse method
* fix UT
Co-authored-by: caishunfeng <534328519@qq.com>
2 years ago
277f137358
Add Kubernetes configmap reload to all components ( #11730 )
2 years ago
f95cdd350a
reduce task dependencies ( #11731 )
2 years ago
67e7f88d8b
Refactor heart beat task, use json to serialize/deserialize ( #11702 )
...
* Refactor heart beat task, use json to serialize/deserialize
2 years ago
b96d69701a
[Improvement-#11613] Add spi priority factory ( #11614 )
...
* Add spi priority factory
* Add doc
* Add override log
* Use lombok
* Add comment
2 years ago
3f2ca7bca3
[Fix-9980] [Server] fix heartBeatTaskCount bug ( #11232 )
...
* fix heartBeat bug
* modify class name
* fix conflict
Co-authored-by: JinyLeeChina <jiny.li@foxmail.com>
2 years ago
abfef1a929
[improvement] Add two parameters in workergroup, and support the application of description display and other parameters ( #11542 )
...
* [Improvement] Improvement default worker group
* [Improvement] Improvement default worker group
* [Improvement] Improvement default worker group
* [Improvement] Improvement default worker group
* [Improvement] Improvement default worker group
* [improvement] The default worker grouping supports editing
* [improvement] The default worker grouping supports editing
* Update WorkerGroupServiceImpl.java
* Update WorkerGroupServiceImpl.java
* The default worker grouping supports editing
* update sql file
* update
* update
* update
* update
2 years ago
4d13a5104b
Fix kill task failed will cause the taskGroup cannot release and add taskGroup log ( #11469 )
2 years ago
a41c6824fc
Add dependent task instance log ( #11541 )
...
* Add dependent task instance log
* Optimize log
* Fix dependent task initialize failed will throw exception
2 years ago
8886d5a126
Fix quartz threadPriority config name error ( #11596 )
...
* Fix quartz threadPriority config name error
* Add batchTriggerAcquisitionMaxCount config
2 years ago
3516533017
Remove logger header in task log file ( #11555 )
2 years ago
f6ef628030
Support reading application properties from kubernetes config map ( #11402 )
2 years ago
4362fb8448
If the task processor is not found need to throw error rather than exception ( #11461 )
2 years ago
05589606a2
Set master's task running status in `runTask` to avoid the task group acquire failed, but the task status is in running ( #11451 )
2 years ago
Aaron Wang
Wenjun Ruan
kezhenxu94
HanayoZz
Yann Ann
liqiquan
caishunfeng
Eric Gao
rickchengx
sgw
xuhhui
fuchanghai
Tq
insist777
Yann Ann
Daniel Y
LiuBodong
Kengo Seki
JinYong Li