DolphinScheduler

Commit Graph

Author	SHA1	Message	Date
JieguangZhou	a47b4d0672	[Feature][Task] Transfer files between tasks (#12552 ) * add data transfer between tasks * add delete DATA_TRANSFER API * convert Result to DeleteDataTransferResponse * add api UT * fix final * fix doc	2 years ago
Aaron Wang	08335b1032	[Improvement][Task] Improved way to collect yarn job's appIds (#12197 ) * Provide aop way as an optional way to collect yarn job's applicationId, and import new module `dolphinscheduler-aop` to place the aop code. * Add user property `appId.collect` for user to decide how to collect applicationId. * Add new environment configuration for each type of yarn tasks to support aop in `dolphinscheduler_env.sh` * Update docs to declare how to use aop way. * Update `LogUtils` to support fetch applicationId in different ways based on the user property. Co-authored-by: gabrywu <gabrywu@apache.com>	2 years ago
kezhenxu94	065d5caccc	Only expose necessary actuator endpoints (#12571 )	2 years ago
HanayoZz	489e7fe4e2	[Feature-10495][Resource Center] Resource Center Refactor (#12076 ) * resource center refactor - S3 services connection Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>	2 years ago
Wenjun Ruan	76a9e6d8a6	Add task executor threads full policy config in worker (#12510 )	2 years ago
Wenjun Ruan	a0d37fc7ae	Set tenantDir permission (#12486 )	2 years ago
caishunfeng	dedff70f90	[Improvement] Merge spi.utils into common.utils (#12502 ) * improve JSONUtil * merge spi.utils into common.utils * split constants && remove some copy utils method * remove StringUtils	2 years ago
fuchanghai	f0a34cddec	[improvement-#12416] Use CollectionUtils.isEmpty judge the collection isEmpty to avoid NPE (#12449 )	2 years ago
Eric Gao	d46e4659d9	[Improvement][Style] Fix remaining style errors and wildcard imports (#12412 ) * Fix remaining style errors and wildcard imports * Upgrade spotless to the latest stable version, exclude .github folder from spotless check and fix dead link * Fix spotless matching scope of markdown files	2 years ago
kezhenxu94	4ea7eec750	Use temurin Java docker images instead of deprecated ones (#12334 )	2 years ago
Yann Ann	bc80f34056	[Migrate][Test] Migrate all UT cases from jUnit 4 to 5 in master, worker, and remote modules (#12362 ) * migrate junit4 -> junit5 in master, worker, and remote modules	2 years ago
Eric Gao	2f37da0dbc	[Improvement][Test] Block the usage of powermock and move mockito dependencies from sub-modules to root pom (#12311 ) * move mockito dependencies from sub-modules to root pom * Add check in CI to block the usage of powermock	2 years ago
kezhenxu94	db1d319159	Reorganize some classes in common module, remove duplicate classes (#12321 )	2 years ago
Eric Gao	9ab79e064c	[Improvement][Test] Fully remove the usage of powermock from the whole project (#12244 ) * Fully remove the usage of powermock from the whole project * Upgrade org.reflections to 0.10.12	2 years ago
Yann Ann	c87f6d89a4	[fix#12195] Change date to timestamp to support cross diff time zones (#12239 ) * fix DS support cross time zone, use timestamp replace date	2 years ago
caishunfeng	40090d459e	[Improvement] remove log-server and server module (#12206 ) * remove log-server module * remove dolphinscheduler-server module	2 years ago
Wenjun Ruan	aacabc6ec7	Remove workerGroup in registry (#12217 )	2 years ago
sgw	962fa6dbd5	[DS-12154][worker] Optimize the log printing of the worker module (#12183 ) * [DS-12154][worker] Optimize the log printing of the worker module according to the log specification.	2 years ago
insist777	5b384f3fab	[Feature-11473][Task]Support test task (#11670 ) * [improve] support test tasks * [improve] support test tasks * [improve] support test tasks * [improve] support test tasks * [improve] support test tasks * Update TaskExecuteThread.java * try solve e2e q * try solve e2e q * try solve e2e q * try solve e2e q * try solve e2e q * try solve e2e q * try solve e2e q * try solve e2e q * try solve e2e q * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * Update DataSource.java * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * Update messages_zh_CN.properties * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * Update messages.properties * Update messages_en_US.properties * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks	2 years ago
Wenjun Ruan	d3a77c68e6	Make LogServiceClient Singleton (#11777 )	2 years ago
Daniel Y	009cb68011	[Improvement-11834] Upgrade docker base image to support python3.9 (#11835 ) * use openjdk:8-jre-slim-bullseye instead of buster to support python3.9 * use openjdk:8-jre-slim-bullseye instead of buster across all components	2 years ago
limaiwang	25b78a8003	[Bug] [spark-sql] In spark-sql, select both SPARK1 and SPARK2 versions and execute ${SPARK_HOME2}/bin/spark-sql (#11721 ) select different versions of spark-sql to execute different versions of spark-sql	2 years ago
LiuBodong	3ca9680b20	[Fix][Monitor]Monitor UI not show DisakAvailable and MemoryUsage correctly (#11870 )	2 years ago
caishunfeng	2e61c76c22	[Improvement] Add remote task model (#11767 )	2 years ago
Kengo Seki	444c7e6eb7	[Improvement] Replace commons-lang 2 function invocations with commons-lang3 (#11810 ) * [Improvement] Replace commons-lang 2 function invocations with commons-lang3	2 years ago
冯剑	45ebd528d6	[Bug-11650][worker] #11650 fix SQL type task, stop task cause NPE (#11668 )	2 years ago
kezhenxu94	277f137358	Add Kubernetes configmap reload to all components (#11730 )	2 years ago
Wenjun Ruan	67e7f88d8b	Refactor heart beat task, use json to serialize/deserialize (#11702 ) * Refactor heart beat task, use json to serialize/deserialize	2 years ago
Wenjun Ruan	03e1e6fe45	[Bug] [Worker] Optimize the getAppId method to avoid worker OOM when kill task (#11701 ) * Fix kill job may cause worker oom	2 years ago
Wenjun Ruan	1b120e3a59	Refactor worker execute task process (#11540 ) * Refactor worker execute task process	2 years ago
JinYong Li	3f2ca7bca3	[Fix-9980] [Server] fix heartBeatTaskCount bug (#11232 ) * fix heartBeat bug * modify class name * fix conflict Co-authored-by: JinyLeeChina <jiny.li@foxmail.com>	2 years ago
Wenjun Ruan	3516533017	Remove logger header in task log file (#11555 )	2 years ago
Eric Gao	9330d6cfcd	[Doc][Security] Update instructions on worker groups (#11483 ) * Update instructions on worker groups	2 years ago
Wenjun Ruan	7ff34c3947	[Feature-7024] Add waiting strategy to support master/worker can recover from registry lost (#11368 ) * Add waiting strategy to support master/worker can recover from registry lost * throw exception when zookeeper registry start failed due to interrupted	2 years ago
caishunfeng	0464123c2b	[Feature-11223] support stream task (#11350 ) * add task execute type * update task definition list paging * update task instance list paging * stream task start * [Feature][UI] Some changes to execute task. * Set the connection edge to dashed line. * Add FLINK_STREAM task. * add stream task * flink savepoint and cancel * fix query bug * add stream task definition * add task instance for stream task * delete stream task definition state * update api for stream task definition edit * modify search for stream task instance * add language * delete task type search for stream task definition * change task type search for stream task instance * add jump button * add savepoint * add down log for stream task instance * ui test * stream task start * run DAG * [Fix][UI] Fix the stream task edgs not to be dashed when filling back. * [Feature][UI] Remove some fields for FLINK_STREAM. * add start modal * add dryRun column for stream task instance * fix duration * fix pon * fix build error * Add success tip * add auto sync for stream task instance * remove forgien key for task instance * license header * UT fix * modify locales * recover common config * fix UT * add doc Co-authored-by: Amy <amywang0104@163.com> Co-authored-by: devosend <devosend@gmail.com>	2 years ago
Wenjun Ruan	8774415197	Split ExecutionStatus to WorkflowExecutionStatus and TaskExecutionStatus (#11340 )	2 years ago
Eric Gao	9ca1eb96c4	[Improvement][Metrics] Add metrics for alert server (#11240 ) * [Improvement][Metrics] Add metrics for alert server (#11131) * Update related docs of metrics * Add grafana demo dashboards for alert server metrics * Refactor metric classes with UtilityClass annotation * Refactor meter names in camelCase for checkstyle	2 years ago
xuhhui	bfff3a7c5d	fix error (#11206 )	2 years ago
zhuxt2015	3701a24d15	[Improvement][Task Log] Task status log print description instead of code (#11009 ) * use execution status instead of status code	2 years ago
Wenjun Ruan	5e9c7dad23	Add dolphinscheduler-bom to manage the dependency version (#11025 )	2 years ago
zhuxt2015	a74d7ef665	[hotfix][Worker] Remove service dependency from worker module (#11008 ) * worker remove service dependency	2 years ago
Wenjun Ruan	083ab2b5c9	Remove dao in worker (#10994 )	2 years ago
Wenjun Ruan	2be1d4bf0a	Fix worker cannot shutdown due to resource close failed or heart beat check failed (#10979 ) * Use try-with-resource to close resource, and add heart error threshold to avoid worker cannot close due to heart beat check failed * Move heartbeat error threshold to applicaiton.yml	2 years ago
Wenjun Ruan	cade66a9b6	[Fix-10827] Fix network error cause worker cannot send message to master (#10886 ) * Fix network error cause worker cannot send message to master	2 years ago
Eric Gao	2f7281c2d2	[Feature][Metrics] Add resource download related metrics for workers (#10749 ) * [Feature][Metrics] Add resource download related metrics for workers (#9324) * [Feature][Metrics] Fix bugs and add grafana demos for worker resource download metrics (#9324) * [Feature][Metrics] Add docs to resource related metrics (#9324) * [Feature][Metrics] Use tags to indicate status in metrics (#9324) * [Feature][Metrics] Fix demos, docs and remove redundant code (#9324) * [Feature][Metrics] Remove .pnpm-debug.log (#9324) * [Feature][Metrics] Fix style check (#9324) * [Feature][Metrics] Replace KB with bytes for the unit of resource file size in metrics (#9324) * [Feature][Metrics] Make code neat (#9324)	2 years ago
Wenjun Ruan	f639a2eed4	[Fix-10854] Fix database restart may lost task instance status (#10866 ) * Fix database update error doesn't rollback the task instance status * Fix database error may cause workflow dead with running status	2 years ago
Wenjun Ruan	426567348e	Remove quartz in service (#10748 ) * Remove quartz in service	2 years ago
Wenjun Ruan	67d14fb7b3	[Fix-10785] Fix state event handle error will not retry (#10786 ) * Fix state event handle error will not retry * Use state event handler to deal with the event	2 years ago
WangJPLeo	8f621ff98b	[Optimization] Calculate global parameter and local parameter at master. (#10704 ) * Global parameter and local parameter calculation external expansion. * k8s task ut fix. * TimePlaceholderUtils import DateUtils fix * follow the review comments to fix. * follow the review comments to fix. * e2e rerun	2 years ago
Wenjun Ruan	35b25da863	Validate master/worker config (#10649 )	2 years ago

1 2 3

107 Commits (72e50b25f1eac15e5a9041c9cae0b7c57aca496e)