85605bbef9
[Improvement-13653][Remote Logging] Support api-server to get task log from remote target ( #13654 )
2 years ago
68660ec96b
Refactor remote command ( #13809 )
...
* Refactor remote command
* Rename Command to Message
2 years ago
d1b6e6f02c
Take over task instance in master failover ( #13798 )
2 years ago
394805b2c7
[Feature][Metrics] Tag workflow related metrics with process definition code (workflow id) ( #13640 )
...
* Tag workflow related metrics with process definition code (workflow id)
* Clean up related metrics when deleting workflow definition
* Add license headers
* Update related UT cases
* Add an example in grafana-demo
* Add related docs
2 years ago
c9066e8de9
Use MDC to filter task instance log ( #13673 )
...
* Use MDC to collect task instance log
* Use MDCAutoClosableContext to remove the MDC key
2 years ago
7bf3e3cdd6
[improve-#13201] update pid during running ( #13206 )
2 years ago
2bd65fb2df
[Feature][Remote Logging] Add support for writing task logs to OSS ( #13332 )
2 years ago
8d12dc0702
[Improvement-13491] Use lombok @Slf4j annotation to generate logger ( #13509 )
2 years ago
de70421a1e
Remove the log path check ( #13280 )
2 years ago
14ec4a2398
Remove dao module in worker ( #13242 )
2 years ago
08335b1032
[Improvement][Task] Improved way to collect yarn job's appIds ( #12197 )
...
* Provide aop way as an optional way to collect yarn job's applicationId, and import new module `dolphinscheduler-aop` to place the aop code.
* Add user property `appId.collect` for user to decide how to collect applicationId.
* Add new environment configuration for each type of yarn tasks to support aop in `dolphinscheduler_env.sh`
* Update docs to declare how to use aop way.
* Update `LogUtils` to support fetch applicationId in different ways based on the user property.
Co-authored-by: gabrywu <gabrywu@apache.com>
2 years ago
dedff70f90
[Improvement] Merge spi.utils into common.utils ( #12502 )
...
* improve JSONUtil
* merge spi.utils into common.utils
* split constants && remove some copy utils method
* remove StringUtils
2 years ago
d46e4659d9
[Improvement][Style] Fix remaining style errors and wildcard imports ( #12412 )
...
* Fix remaining style errors and wildcard imports
* Upgrade spotless to the latest stable version, exclude .github folder from spotless check and fix dead link
* Fix spotless matching scope of markdown files
2 years ago
bc80f34056
[Migrate][Test] Migrate all UT cases from jUnit 4 to 5 in master, worker, and remote modules ( #12362 )
...
* migrate junit4 -> junit5 in master, worker, and remote modules
2 years ago
db1d319159
Reorganize some classes in common module, remove duplicate classes ( #12321 )
2 years ago
9ab79e064c
[Improvement][Test] Fully remove the usage of powermock from the whole project ( #12244 )
...
* Fully remove the usage of powermock from the whole project
* Upgrade org.reflections to 0.10.12
2 years ago
165b9a58de
[DS-12131][master] Optimize the log printing of the master module acc… ( #12152 )
...
* [DS-12131][master] Optimize the log printing of the master module according to the log specification.
2 years ago
c87f6d89a4
[fix#12195] Change date to timestamp to support cross diff time zones ( #12239 )
...
* fix DS support cross time zone, use timestamp replace date
2 years ago
d3a77c68e6
Make LogServiceClient Singleton ( #11777 )
2 years ago
03e1e6fe45
[Bug] [Worker] Optimize the getAppId method to avoid worker OOM when kill task ( #11701 )
...
* Fix kill job may cause worker oom
2 years ago
5fa3e7b1ed
Add pause to task instance status ( #11390 )
2 years ago
0464123c2b
[Feature-11223] support stream task ( #11350 )
...
* add task execute type
* update task definition list paging
* update task instance list paging
* stream task start
* [Feature][UI] Some changes to execute task.
* Set the connection edge to dashed line.
* Add FLINK_STREAM task.
* add stream task
* flink savepoint and cancel
* fix query bug
* add stream task definition
* add task instance for stream task
* delete stream task definition state
* update api for stream task definition edit
* modify search for stream task instance
* add language
* delete task type search for stream task definition
* change task type search for stream task instance
* add jump button
* add savepoint
* add down log for stream task instance
* ui test
* stream task start
* run DAG
* [Fix][UI] Fix the stream task edgs not to be dashed when filling back.
* [Feature][UI] Remove some fields for FLINK_STREAM.
* add start modal
* add dryRun column for stream task instance
* fix duration
* fix pon
* fix build error
* Add success tip
* add auto sync for stream task instance
* remove forgien key for task instance
* license header
* UT fix
* modify locales
* recover common config
* fix UT
* add doc
Co-authored-by: Amy <amywang0104@163.com>
Co-authored-by: devosend <devosend@gmail.com>
2 years ago
8774415197
Split ExecutionStatus to WorkflowExecutionStatus and TaskExecutionStatus ( #11340 )
2 years ago
3701a24d15
[Improvement][Task Log] Task status log print description instead of code ( #11009 )
...
* use execution status instead of status code
2 years ago
998e4d74dd
Kill task when we do master failover to avoid task lost ( #10997 )
2 years ago
cade66a9b6
[Fix-10827] Fix network error cause worker cannot send message to master ( #10886 )
...
* Fix network error cause worker cannot send message to master
2 years ago
aa8b88a875
[Feature-10871] add workflow executing data query ( #10875 )
...
* add workflow executing data query
* fix sonar check for interrupted
2 years ago
f639a2eed4
[Fix-10854] Fix database restart may lost task instance status ( #10866 )
...
* Fix database update error doesn't rollback the task instance status
* Fix database error may cause workflow dead with running status
2 years ago
67d14fb7b3
[Fix-10785] Fix state event handle error will not retry ( #10786 )
...
* Fix state event handle error will not retry
* Use state event handler to deal with the event
2 years ago
66624c5c86
[Bug] [Master] Worker failover will cause task cannot be failover ( #10631 )
...
* fix worker failover may lose event
2 years ago
f46faa02c3
[Fix-10452] Serial wait for policy recovery ( #10453 )
...
* Serial wait for policy recovery
* processInstance state check null
* add sendresult(Host host, Command command) method and replace the original.
2 years ago
ad2646ff1f
Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe ( #10479 )
...
* Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe
2 years ago
b0d9d3f9ab
Add some warning log in master ( #10383 )
...
* Add some warn log in master
* fix may skip sleep
2 years ago
e21d7b1551
[Feature][metrics] Add master, worker metrics ( #10326 )
...
* Add mater metrics
* fix UT
* Add url to mysql profile
* Add worker metrics
* Update grafana config
* Add system metrics doc
* Add process failover counter
* Add metrics image
* Change jpg to png
* Add command insert metrics
* Fix UT
* Revert UT
2 years ago
516757cc74
[Fix] [Worker] Fix worker will hang if fails to start ( #10342 )
...
* Fix worker will hang if fails to start
* Add .run to ignore
2 years ago
d80cf21456
Clean up unused dependencies and packaging issues ( #9944 )
2 years ago
49979c658e
[Fix-8828] [Master] Assign tasks to worker optimization ( #9919 )
...
* fix 9584
* master recall
* fix ut
* update logger
* update delay queue
* fix ut
* remove sleep
Co-authored-by: 进勇 <lijinyong@cai-inc.com>
Co-authored-by: JinyLeeChina <jiny.li@foxmail.com>
2 years ago
ca95d2f928
[Fix-9221] [alert-server] optimization and gracefully close ( #9246 )
...
* [Fix-9221] [alert-server] optimization and gracefully close
This closes #9221
* [Fix-9221] [alert-server] remove unused mock data
This closes #9221
* [Fix-9221] [alert-server] remove unused mock data
This closes #9221
* [Fix-9221] [alert-server] remove unnecessary Mockito stubbings
* [Fix-9221] [alert-server] init AlertPluginManager in AlertServer
* [Fix-9221] [alert-server] AlertServerTest add AlertPluginManager installPlugin
* [Fix-9221] [alert-server] replace @Eventlistener with @PostConstruct
* [Fix-9221] [alert-server] sonar check solution
* [Improvement-9221] [alert] update constructor injection and replace IStoppable with Closeable
Co-authored-by: guoshupei <guoshupei@lixiang.com>
3 years ago
ddf1ff98fa
[Improvement-7697][Master/Worker] Change the task ack to runnning callback ( #8719 )
...
* rebase dev
* change task state to dispatch if dispatch success
* update task host when dispatch
* add dispatch task event
* test
* check tenant after enable auto create
* handle dispatch state
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
a1bef43981
[Feature-8579][Task] Task back-end plugin optimization, new plugins only need to modify the plugin's own module ( #8584 )
...
* Task plug-in optimization, task-related classes are migrated to the task plugin module
* remove tasktype enum class.
* fix code style.
* fix code style
* fix code style
* fix ProcessServiceTest.testUpdateResourceInfo test class
* fix code style
* fix test task plugin manager check parameters null
* fix test ut
* update aws license
* change TaskTimeoutStrategy class
3 years ago
0d8079a816
[DS-7992][feat] Alert module judging strategy ( #8636 )
3 years ago
a72742b9de
remove unuse import jsonFormat ( #8548 )
3 years ago
84f2cfec39
[Feature][Server] Custom timezone - Add timezone handler unified ( #8477 )
...
* date convert of timezone
* remove @JsonFormat
* add unit test
* fix time preview in scheduler
* optimization & add env config
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
66dcf2376a
Optimize some code ( #8324 )
3 years ago
093815785b
Remove unused class ( #8299 )
3 years ago
0cfb2043cf
Improvement: improve Pong.java mistake ( #7822 )
3 years ago
a80a7b3fcf
[Feature-6963][MasterServer] unified cache manager ( #7187 )
...
* unified cache
* reduce db select
* checkstyle
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
2b519f31e9
[Feature-#6422] task group queue ( #6722 )
...
* add task group
* modify task group
* pull dev
* add license header
* modify code style
* fix code style
* fix sql error
* fix error
* fix test
* fix test
* fix test
* fix test
* fix code style
* fix ut
* code style
* fix unit test
* test ut
* ut
* add unittest
* test ut
* modify back ut
* majorization code
* fix conflict
* fix ut
Co-authored-by: wangxj <wangxj31>
3 years ago
8e93939121
[DS-6963][MasterServer]add cache manager for tenant and user ( #6974 )
...
* cache manager for tenant and user
* add test
* fix checkstyle
* add dependencies
* fix sonar check
* use spring boot cache configuration
* add license
* check style
* check style
* add license
* change proxy name to processor
* add license header
* check style
* add license file
* add license file
* add license
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
c78fd24776
bug about dolphinscheduler-remote SendAsync ( #6972 )
...
* bug about SendAsync
3 years ago
Rick Cheng
Wenjun Ruan
Eric Gao
fuchanghai
seedscoder
Aaron Wang
caishunfeng
Yann Ann
kezhenxu94
sgw
zhuxt2015
WangJPLeo
JinYong Li
guoshupei
Kerwin
wangyang
yimaixinchen
xuhhui
wind
wangxj3
lyq