a17d0cc5d2
cherry-pick [Improvement][WorkerGroup]Remove workerGroup in registry #12217
2 years ago
58ef3cccd7
[3.1.1-prepare]cherry-pick [Improvement] remove log-server and server module #12206 ( #12562 )
2 years ago
f3277277f0
cherry-pick Refactor LogServiceClient Singleton to avoid repeat creation of NettyClient #11777 ( #12542 )
...
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
2 years ago
344293102d
[3.1.1-prepare][cherry-pick][Improvement] Add remote task model #11767 ( #12541 )
2 years ago
2402b8a6ef
Use temurin Java docker images instead of deprecated ones ( #12334 ) ( #12421 )
...
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2 years ago
d2e56af838
[Bug] [spark-sql] In spark-sql, select both SPARK1 and SPARK2 versions and execute ${SPARK_HOME2}/bin/spark-sql ( #11721 ) ( #12420 )
...
select different versions of spark-sql to execute different versions of spark-sql
Co-authored-by: limaiwang <limaiwang@163.com>
2 years ago
703f9991b4
[DS-12154][worker] Optimize the log printing of the worker module ( #12183 )
...
* [DS-12154][worker] Optimize the log printing of the worker module according to the log specification.
2 years ago
7e39396a76
[maven-release-plugin] prepare for next development iteration
2 years ago
ae33ba5947
[maven-release-plugin] prepare release 3.1.0
2 years ago
2a437607ae
[Improvement-11834] Upgrade docker base image to support python3.9 ( #11835 )
...
* use openjdk:8-jre-slim-bullseye instead of buster to support python3.9
* use openjdk:8-jre-slim-bullseye instead of buster across all components
2 years ago
0f3b42925f
[Fix][Monitor]Monitor UI not show DisakAvailable and MemoryUsage correctly ( #11870 )
2 years ago
52b79b017e
[Improvement] Replace commons-lang 2 function invocations with commons-lang3 ( #11810 )
...
* [Improvement] Replace commons-lang 2 function invocations with commons-lang3
2 years ago
f034a09d25
[Bug-11650][worker] #11650 fix SQL type task, stop task cause NPE ( #11668 ) ( #11958 )
...
Co-authored-by: 冯剑 <35831367+fengjian1129@users.noreply.github.com>
2 years ago
277f137358
Add Kubernetes configmap reload to all components ( #11730 )
2 years ago
67e7f88d8b
Refactor heart beat task, use json to serialize/deserialize ( #11702 )
...
* Refactor heart beat task, use json to serialize/deserialize
2 years ago
03e1e6fe45
[Bug] [Worker] Optimize the getAppId method to avoid worker OOM when kill task ( #11701 )
...
* Fix kill job may cause worker oom
2 years ago
1b120e3a59
Refactor worker execute task process ( #11540 )
...
* Refactor worker execute task process
2 years ago
3f2ca7bca3
[Fix-9980] [Server] fix heartBeatTaskCount bug ( #11232 )
...
* fix heartBeat bug
* modify class name
* fix conflict
Co-authored-by: JinyLeeChina <jiny.li@foxmail.com>
2 years ago
3516533017
Remove logger header in task log file ( #11555 )
2 years ago
9330d6cfcd
[Doc][Security] Update instructions on worker groups ( #11483 )
...
* Update instructions on worker groups
2 years ago
7ff34c3947
[Feature-7024] Add waiting strategy to support master/worker can recover from registry lost ( #11368 )
...
* Add waiting strategy to support master/worker can recover from registry lost
* throw exception when zookeeper registry start failed due to interrupted
2 years ago
0464123c2b
[Feature-11223] support stream task ( #11350 )
...
* add task execute type
* update task definition list paging
* update task instance list paging
* stream task start
* [Feature][UI] Some changes to execute task.
* Set the connection edge to dashed line.
* Add FLINK_STREAM task.
* add stream task
* flink savepoint and cancel
* fix query bug
* add stream task definition
* add task instance for stream task
* delete stream task definition state
* update api for stream task definition edit
* modify search for stream task instance
* add language
* delete task type search for stream task definition
* change task type search for stream task instance
* add jump button
* add savepoint
* add down log for stream task instance
* ui test
* stream task start
* run DAG
* [Fix][UI] Fix the stream task edgs not to be dashed when filling back.
* [Feature][UI] Remove some fields for FLINK_STREAM.
* add start modal
* add dryRun column for stream task instance
* fix duration
* fix pon
* fix build error
* Add success tip
* add auto sync for stream task instance
* remove forgien key for task instance
* license header
* UT fix
* modify locales
* recover common config
* fix UT
* add doc
Co-authored-by: Amy <amywang0104@163.com>
Co-authored-by: devosend <devosend@gmail.com>
2 years ago
8774415197
Split ExecutionStatus to WorkflowExecutionStatus and TaskExecutionStatus ( #11340 )
2 years ago
9ca1eb96c4
[Improvement][Metrics] Add metrics for alert server ( #11240 )
...
* [Improvement][Metrics] Add metrics for alert server (#11131 )
* Update related docs of metrics
* Add grafana demo dashboards for alert server metrics
* Refactor metric classes with UtilityClass annotation
* Refactor meter names in camelCase for checkstyle
2 years ago
bfff3a7c5d
fix error ( #11206 )
2 years ago
3701a24d15
[Improvement][Task Log] Task status log print description instead of code ( #11009 )
...
* use execution status instead of status code
2 years ago
5e9c7dad23
Add dolphinscheduler-bom to manage the dependency version ( #11025 )
2 years ago
a74d7ef665
[hotfix][Worker] Remove service dependency from worker module ( #11008 )
...
* worker remove service dependency
2 years ago
083ab2b5c9
Remove dao in worker ( #10994 )
2 years ago
2be1d4bf0a
Fix worker cannot shutdown due to resource close failed or heart beat check failed ( #10979 )
...
* Use try-with-resource to close resource, and add heart error threshold to avoid worker cannot close due to heart beat check failed
* Move heartbeat error threshold to applicaiton.yml
2 years ago
cade66a9b6
[Fix-10827] Fix network error cause worker cannot send message to master ( #10886 )
...
* Fix network error cause worker cannot send message to master
2 years ago
2f7281c2d2
[Feature][Metrics] Add resource download related metrics for workers ( #10749 )
...
* [Feature][Metrics] Add resource download related metrics for workers (#9324 )
* [Feature][Metrics] Fix bugs and add grafana demos for worker resource download metrics (#9324 )
* [Feature][Metrics] Add docs to resource related metrics (#9324 )
* [Feature][Metrics] Use tags to indicate status in metrics (#9324 )
* [Feature][Metrics] Fix demos, docs and remove redundant code (#9324 )
* [Feature][Metrics] Remove .pnpm-debug.log (#9324 )
* [Feature][Metrics] Fix style check (#9324 )
* [Feature][Metrics] Replace KB with bytes for the unit of resource file size in metrics (#9324 )
* [Feature][Metrics] Make code neat (#9324 )
2 years ago
f639a2eed4
[Fix-10854] Fix database restart may lost task instance status ( #10866 )
...
* Fix database update error doesn't rollback the task instance status
* Fix database error may cause workflow dead with running status
2 years ago
426567348e
Remove quartz in service ( #10748 )
...
* Remove quartz in service
2 years ago
67d14fb7b3
[Fix-10785] Fix state event handle error will not retry ( #10786 )
...
* Fix state event handle error will not retry
* Use state event handler to deal with the event
2 years ago
8f621ff98b
[Optimization] Calculate global parameter and local parameter at master. ( #10704 )
...
* Global parameter and local parameter calculation external expansion.
* k8s task ut fix.
* TimePlaceholderUtils import DateUtils fix
* follow the review comments to fix.
* follow the review comments to fix.
* e2e rerun
2 years ago
35b25da863
Validate master/worker config ( #10649 )
2 years ago
66624c5c86
[Bug] [Master] Worker failover will cause task cannot be failover ( #10631 )
...
* fix worker failover may lose event
2 years ago
719a9d4532
[Improvement][Worker] fixed naming of rpc package ( #10614 )
2 years ago
1111371c9a
add datasource health check to the healthcheck endpoint ( #10588 )
2 years ago
0f38217b12
fix_10514 ( #10568 )
2 years ago
db595b3eff
Optimize master log, use MDC to inject workflow instance id and task instance id in log ( #10516 )
...
* Optimize master log, add workflow instance id and task instance id in log
* Use MDC to set the workflow info in log4j
* Add workflowInstanceId and taskInstanceId in MDC
2 years ago
cc06eaaf54
[Improvement][Metrics] Apply micrometer naming convention to metrics ( #10477 )
...
* Apply micrometer naming convention to worker metrics
* Apply micrometer naming convention all current metrics
* Fix remaining metrics names, update English docs and add Chinese docs
* Fix metrics names in grafana-demo dashboards
2 years ago
ad2646ff1f
Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe ( #10479 )
...
* Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe
2 years ago
78c5fcc6ac
Add mysql registry plugin ( #10406 )
...
* Add mysql registry plugin
2 years ago
e21d7b1551
[Feature][metrics] Add master, worker metrics ( #10326 )
...
* Add mater metrics
* fix UT
* Add url to mysql profile
* Add worker metrics
* Update grafana config
* Add system metrics doc
* Add process failover counter
* Add metrics image
* Change jpg to png
* Add command insert metrics
* Fix UT
* Revert UT
2 years ago
2d3be6b36c
Add dolphinscheduler-scheduler module ( #10360 )
...
* Add dolphinscheduler-scheduler module
2 years ago
022e4886be
Remove quartz at WorkerServer ( #10358 )
...
* Remove quartz at WorkerServer
* move k8s and permission from dolphinscheduler-service to dolphinscheduler-api
2 years ago
d80cf21456
Clean up unused dependencies and packaging issues ( #9944 )
2 years ago
49979c658e
[Fix-8828] [Master] Assign tasks to worker optimization ( #9919 )
...
* fix 9584
* master recall
* fix ut
* update logger
* update delay queue
* fix ut
* remove sleep
Co-authored-by: 进勇 <lijinyong@cai-inc.com>
Co-authored-by: JinyLeeChina <jiny.li@foxmail.com>
2 years ago
Wenjun Ruan
Kerwin
sgw
caishunfeng
Daniel Y
LiuBodong
Kengo Seki
kezhenxu94
JinYong Li
Eric Gao
xuhhui
zhuxt2015
WangJPLeo
pinkhello
xiangzihao