DolphinScheduler

Commit Graph

Author	SHA1	Message	Date
caishunfeng	2e61c76c22	[Improvement] Add remote task model (#11767 )	2 years ago
kezhenxu94	efedc9a53c	Bump up dependencies to fix cves (#11765 )	2 years ago
caishunfeng	f95cdd350a	reduce task dependencies (#11731 )	2 years ago
Wenjun Ruan	03e1e6fe45	[Bug] [Worker] Optimize the getAppId method to avoid worker OOM when kill task (#11701 ) * Fix kill job may cause worker oom	2 years ago
Wenjun Ruan	1b120e3a59	Refactor worker execute task process (#11540 ) * Refactor worker execute task process	2 years ago
Wenjun Ruan	b96d69701a	[Improvement-#11613] Add spi priority factory (#11614 ) * Add spi priority factory * Add doc * Add override log * Use lombok * Add comment	2 years ago
Wenjun Ruan	a41c6824fc	Add dependent task instance log (#11541 ) * Add dependent task instance log * Optimize log * Fix dependent task initialize failed will throw exception	2 years ago
kezhenxu94	13e4720aa2	Upgrade spring boot and swagger (#11574 )	2 years ago
kezhenxu94	f6ef628030	Support reading application properties from kubernetes config map (#11402 )	2 years ago
Kerwin	b79bad19e5	[Fix-11051][Task]Fix the process exitValue exception when the process times out and th… (#11099 )	2 years ago
Wenjun Ruan	5fa3e7b1ed	Add pause to task instance status (#11390 )	2 years ago
Stalary	6345a13cdc	[Fix-10392][SqlTask]: Fix list param error when use sql task (#11285 ) * FIX: fix list param in. * FIX: code bug * FIX: fix one value * FIX: header * FIX: ci * FIX: fix ci * MOD: for review * MOD: for review	2 years ago
caishunfeng	0464123c2b	[Feature-11223] support stream task (#11350 ) * add task execute type * update task definition list paging * update task instance list paging * stream task start * [Feature][UI] Some changes to execute task. * Set the connection edge to dashed line. * Add FLINK_STREAM task. * add stream task * flink savepoint and cancel * fix query bug * add stream task definition * add task instance for stream task * delete stream task definition state * update api for stream task definition edit * modify search for stream task instance * add language * delete task type search for stream task definition * change task type search for stream task instance * add jump button * add savepoint * add down log for stream task instance * ui test * stream task start * run DAG * [Fix][UI] Fix the stream task edgs not to be dashed when filling back. * [Feature][UI] Remove some fields for FLINK_STREAM. * add start modal * add dryRun column for stream task instance * fix duration * fix pon * fix build error * Add success tip * add auto sync for stream task instance * remove forgien key for task instance * license header * UT fix * modify locales * recover common config * fix UT * add doc Co-authored-by: Amy <amywang0104@163.com> Co-authored-by: devosend <devosend@gmail.com>	2 years ago
Wenjun Ruan	8774415197	Split ExecutionStatus to WorkflowExecutionStatus and TaskExecutionStatus (#11340 )	2 years ago
microeastcowboy	9a53c6ac34	fix work oom when task logs's size is very large (#11224 )	2 years ago
xuhhui	bfff3a7c5d	fix error (#11206 )	2 years ago
caishunfeng	818648df7d	[Bug-11071] fix local params (#11184 ) * fix local params * add CI case	2 years ago
Wenjun Ruan	052ceaacd0	Add is success in loop task status (#11169 )	2 years ago
Wenjun Ruan	c7789bf049	Add base loop task execotor and http template parser (#11137 ) * Add base loop task execotor and http template parser * Add JsonPathUtils	2 years ago
Eric Gao	a38fa34d43	[Feature][Task Plugin] Enable users to switch endpoints in zeppelin tasks (#10925 ) * [Feature][Task Plugin] Enable users to switch endpoints in zeppelin tasks (#9814)	2 years ago
chuxing	427c58546c	[fix][plugin]: fix spark sql use custom parma. (#10821 ) * [fix][plugin]: fix spark sql use custom parma. * [fix][plugin] roll back part of the code Co-authored-by: huaangcg <huangcg@getui.com>	2 years ago
WangJPLeo	8f621ff98b	[Optimization] Calculate global parameter and local parameter at master. (#10704 ) * Global parameter and local parameter calculation external expansion. * k8s task ut fix. * TimePlaceholderUtils import DateUtils fix * follow the review comments to fix. * follow the review comments to fix. * e2e rerun	2 years ago
Wenjun Ruan	66624c5c86	[Bug] [Master] Worker failover will cause task cannot be failover (#10631 ) * fix worker failover may lose event	2 years ago
Wenjun Ruan	db595b3eff	Optimize master log, use MDC to inject workflow instance id and task instance id in log (#10516 ) * Optimize master log, add workflow instance id and task instance id in log * Use MDC to set the workflow info in log4j * Add workflowInstanceId and taskInstanceId in MDC	2 years ago
guodong	2c227ab384	[Feature-10363][server] S3 Resource center supports bucket customization (#10433 ) * [Feature-10363][server] S3 Resource center supports bucket customization * Modify the configuration. Sort out HDFS and S3 configurations to distinguish them * Modify the configuration comment to clarify the configuration meaning * Modify official documents to add missing configuration descriptions and clarify configuration meanings and usage * Modify the configuration file in e2E to the correct definition, dolphinscheduler-e2e/dolphinscheduler-e2e-case/src/test/resources/docker/file-manage/common.properties * Modified code to support bucket customization * Check that the bucket on personal permission, Otherwise, the startup fails * S3 increase support resource. Storage. Upload. Base. The base path path * Change S3Utils implementation style to be consistent with HadoopUtils * [Feature-10363][server] S3 Resource center supports bucket customization * Modify code review comments	2 years ago
xiangzihao	bbba20d39a	change to common lang3 and fix checkstyle (#10380 )	3 years ago
xiangzihao	b1fb17119b	[Feature-6758][Task] Add limit resource usage for tasks base on cgroup (#10373 )	3 years ago
kezhenxu94	d80cf21456	Clean up unused dependencies and packaging issues (#9944 )	3 years ago
He Zhao	5bb1eb04fc	[Feature][plugin] Add k8s task in task plugin (#9425 ) * [Feature][plugin] Add k8s task in task plugin * [Feature][plugin] fix dos and code problems * [Feature][plugin] refactor some code based on sonar * [Feature][UI] front-end for k8s task plugin * [Feature][plugin] delete some front files * [Feature][plugin] update document * Update docs/docs/zh/guide/task/k8s.md * Update docs/docs/en/guide/task/k8s.md * Update docs/docs/en/guide/task/k8s.md * Update docs/docs/en/guide/task/k8s.md * [Feature][UI] front-end change from review comments * [Feature][UI] replace get namespace list api * [Feature][plugin] change file name * Add kubernetes to zh task list Co-authored-by: hezhao2 <hezhao2@cisco.com> Co-authored-by: William Tong <weitong@cisco.com> Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>	3 years ago
Eric Gao	80369363c9	[feat][task plugin] Add Jupyter task plugin (#9872 ) Co-authored-by: Amy0104 <97265214+Amy0104@users.noreply.github.com>	3 years ago
Kerwin	0e7c98d465	[Feature-9177][Task] The sql task supports configuring segmentation notation to provide execution of multiple statements (#9917 ) * Support sql segmentation to execute multiple functions.	3 years ago
sq-q	ed425d2d2d	[Feature-9772][plugin/ui] support SparkSQL Task (#9790 ) * [refactor] Dolphinscheduler sparkSQL * [refactor] Dolphinscheduler SparkSQL * [refactor] Dolphinscheduler SparkSQL * [refactor] Dolphinscheduler SparkSQL * [refactor] Dolphinscheduler SparkSQL * [refactor] Dolphinscheduler SparkSQL * [refactor] Dolphinscheduler SparkSQL * [refactor] Refactor ui code and add sparksql test cases * [refactor] refactor dolphinscheduler SparkSQL * [refactor] refactor dolphinscheduler plugin-sparkSQL * [refactor] refactor dolphinscheduler plugin-SparkSQL * [refactor] dolphinscheduler plugin-SparkTaskTest * [refactor] dolphinscheduler plugin-SparkTask * [refactor] dolphinscheduler plugin-Spark * [refactor] dolphinscheduler plugin-SparkTask-SparkSQL * [refactor] dolphinscheduler plugin-spark-SparkTask * [refactor] dolphinscheduler plugin-spark-SparkTask redefine code	3 years ago
exmy	36f01155b5	[Improvement][server] varPool support syntax #{setValue(key=value)} (#9586 )	3 years ago
Tq	c5b7e5adff	[Bug] [API-9558]fix homepage task instance count method to use submit time to recount (#9559 ) * fix homepage task instance count method to use submit time to recount * fix homepage task instance count method to use submit time to recount * fix homepage task instance count method to use submit time to recount * fix homepage task instance count method JUNIT * fix homepage task instance count method JUNIT * fix homepage task instance count method JUNIT	3 years ago
Eric Gao	a5bbf7852d	[Feature][Task-Plugin]Add zeppelin task-plugin to support Apache Zeppelin (#9327 )	3 years ago
Tq	04d4e4e0c4	fix the OUT direct param could pass to the current script (#9463 )	3 years ago
caishunfeng	b285ccf930	[Future-9396]Support output parameters transfer from parent workflow to child work flow (#9410 ) * [Future-9396]Support output parameters transfer from parent workflow to child work flow * fix note	3 years ago
nobolity	037692517a	[Fix-9251] [WORKER] reslove the sql task about of add the udf resource failed (#9319 ) * feat(resource manager): extend s3 to the storage of ds 1.fix some spell question 2.extend the type of storage 3.add the s3utils to manager resource 4.automatic inject the storage in addition to your config * fix(resource manager): update the dependency * fix(resource manager): extend s3 to the storage of ds fix the constant of hadooputils * fix(resource manager): extend s3 to the storage of ds 1.fix some spell question 2.delete the import * * fix(resource manager): merge the unitTest: 1.TenantServiceImpl 2.ResourceServiceImpl 3.UserServiceImpl * fix(resource manager): extend s3 to the storage of ds merge the resourceServiceTest * fix(resource manager): test cancel the test method createTenant verifyTenant * fix(resource manager): merge the code follow the check-result of sonar * fix(resource manager): extend s3 to the storage of ds fit the spell question * fix(resource manager): extend s3 to the storage of ds revert the common.properties * fix(resource manager): extend s3 to the storage of ds update the storageConfig with None * fix(resource manager): extend s3 to the storage of ds fix the judge of resourceType * fix(resource manager): extend s3 to the storage of ds undo the compile-mysql * fix(resource manager): extend s3 to the storage of ds delete hadoop aws * fix(resource manager): extend s3 to the storage of ds update the know-dependencies to delete aws 1.7.4 update the e2e file-manager common.properties * fix(resource manager): extend s3 to the storage of ds update the aws-region * fix(resource manager): extend s3 to the storage of ds fix the storageconfig init * fix(resource manager): update e2e docker-compose update e2e docker-compose * fix(resource manager): extend s3 to the storage of ds revent the e2e common.proprites print the resource type in propertyUtil * fix(resource manager): extend s3 to the storage of ds 1.println the properties * fix(resource manager): println the s3 info * fix(resource manager): extend s3 to the storage of ds delete the info and upgrade the s3 info to e2e * fix(resource manager): extend s3 to the storage of ds add the bucket init * fix(resource manager): extend s3 to the storage of ds 1.fix some spell question 2.delete the import * * fix(resource manager): extend s3 to the storage of ds upgrade the s3 endpoint * fix(resource manager): withPathStyleAccessEnabled(true) * fix(resource manager): extend s3 to the storage of ds 1.fix some spell question 2.delete the import * * fix(resource manager): upgrade the s3client builder * fix(resource manager): correct the s3 point to s3client * fix(resource manager): update the constant BUCKET_NAME * fix(resource manager): e2e s3 endpoint -> s3:9000 * fix(resource manager): extend s3 to the storage of ds 1.fix some spell question 2.delete the import * * style(resource manager): add info to createBucket * style(resource manager): debug the log * ci(resource manager): test test s3 * ci(ci): add INSERT INTO dolphinscheduler.t_ds_tenant (id, tenant_code, description, queue_id, create_time, update_time) VALUES(1, 'root', NULL, 1, NULL, NULL); to h2.sql * fix(resource manager): update the h2 sql * fix(resource manager): solve to delete the tenant * style(resource manager): merge the style end delete the unuse s3 config * fix(resource manager): extend s3 to the storage of ds UPDATE the rename resources when s3 * fix(resource manager): extend s3 to the storage of ds 1.fix the code style of QuartzImpl * fix(resource manager): extend s3 to the storage of ds 1.impoort restore_type to CommonUtils * fix(resource manager): update the work thread * fix(resource manager): update the baseTaskProcessor * fix(resource manager): upgrade dolphinscheduler-standalone-server.xml * fix(resource manager): add user Info to dolphinscheduler_h2.sql * fix(resource manager): merge the resourceType to NONE * style(upgrade the log level to info): * fix(resource manager): sysnc the h2.sql * fix(resource manager): update the merge the user tenant * fix(resource manager): merge the resourcesServiceImpl * fix(resource manager): when the storage is s3 ,that the directory can't be renamed * fix(resource manager): in s3 ,the directory cannot be renamed * fix(resource manager): delete the deleteRenameDirectory in E2E * fix(resource manager): check the style and recoverd the test * fix(resource manager): delete the log.print(LoginUser) * fix(server): fix the udf serialize * fix(master task): update the udfTest to update the json string * fix(test): update the udfFuncTest * fix(common): syn the common.properties * fix(udfTest): upgrade the udfTest * fix(common): revent the common.properties	3 years ago
Tq	2a4fa9cdb1	[BUG][WORKER-9349]fix param priority (#9379 ) * fix param priority * fix params priority code logic	3 years ago
caishunfeng	ddf1ff98fa	[Improvement-7697][Master/Worker] Change the task ack to runnning callback (#8719 ) * rebase dev * change task state to dispatch if dispatch success * update task host when dispatch * add dispatch task event * test * check tenant after enable auto create * handle dispatch state Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
nobolity	0e3cafec1d	[Feature-8612][RESOURCE] extend s3 to the storage of ds (#8637 ) * feat(resource manager): extend s3 to the storage of ds 1.fix some spell question 2.extend the type of storage 3.add the s3utils to manager resource 4.automatic inject the storage in addition to your config * fix(resource manager): update the dependency * fix(resource manager): extend s3 to the storage of ds fix the constant of hadooputils * fix(resource manager): extend s3 to the storage of ds 1.fix some spell question 2.delete the import * * fix(resource manager): merge the unitTest: 1.TenantServiceImpl 2.ResourceServiceImpl 3.UserServiceImpl * fix(resource manager): extend s3 to the storage of ds merge the resourceServiceTest * fix(resource manager): test cancel the test method createTenant verifyTenant * fix(resource manager): merge the code follow the check-result of sonar * fix(resource manager): extend s3 to the storage of ds fit the spell question * fix(resource manager): extend s3 to the storage of ds revert the common.properties * fix(resource manager): extend s3 to the storage of ds update the storageConfig with None * fix(resource manager): extend s3 to the storage of ds fix the judge of resourceType * fix(resource manager): extend s3 to the storage of ds undo the compile-mysql * fix(resource manager): extend s3 to the storage of ds delete hadoop aws * fix(resource manager): extend s3 to the storage of ds update the know-dependencies to delete aws 1.7.4 update the e2e file-manager common.properties * fix(resource manager): extend s3 to the storage of ds update the aws-region * fix(resource manager): extend s3 to the storage of ds fix the storageconfig init * fix(resource manager): update e2e docker-compose update e2e docker-compose * fix(resource manager): extend s3 to the storage of ds revent the e2e common.proprites print the resource type in propertyUtil * fix(resource manager): extend s3 to the storage of ds 1.println the properties * fix(resource manager): println the s3 info * fix(resource manager): extend s3 to the storage of ds delete the info and upgrade the s3 info to e2e * fix(resource manager): extend s3 to the storage of ds add the bucket init * fix(resource manager): extend s3 to the storage of ds 1.fix some spell question 2.delete the import * * fix(resource manager): extend s3 to the storage of ds upgrade the s3 endpoint * fix(resource manager): withPathStyleAccessEnabled(true) * fix(resource manager): extend s3 to the storage of ds 1.fix some spell question 2.delete the import * * fix(resource manager): upgrade the s3client builder * fix(resource manager): correct the s3 point to s3client * fix(resource manager): update the constant BUCKET_NAME * fix(resource manager): e2e s3 endpoint -> s3:9000 * fix(resource manager): extend s3 to the storage of ds 1.fix some spell question 2.delete the import * * style(resource manager): add info to createBucket * style(resource manager): debug the log * ci(resource manager): test test s3 * ci(ci): add INSERT INTO dolphinscheduler.t_ds_tenant (id, tenant_code, description, queue_id, create_time, update_time) VALUES(1, 'root', NULL, 1, NULL, NULL); to h2.sql * fix(resource manager): update the h2 sql * fix(resource manager): solve to delete the tenant * style(resource manager): merge the style end delete the unuse s3 config * fix(resource manager): extend s3 to the storage of ds UPDATE the rename resources when s3 * fix(resource manager): extend s3 to the storage of ds 1.fix the code style of QuartzImpl * fix(resource manager): extend s3 to the storage of ds 1.impoort restore_type to CommonUtils * fix(resource manager): update the work thread * fix(resource manager): update the baseTaskProcessor * fix(resource manager): upgrade dolphinscheduler-standalone-server.xml * fix(resource manager): add user Info to dolphinscheduler_h2.sql * fix(resource manager): merge the resourceType to NONE * style(upgrade the log level to info): * fix(resource manager): sysnc the h2.sql * fix(resource manager): update the merge the user tenant * fix(resource manager): merge the resourcesServiceImpl * fix(resource manager): when the storage is s3 ,that the directory can't be renamed * fix(resource manager): in s3 ,the directory cannot be renamed * fix(resource manager): delete the deleteRenameDirectory in E2E * fix(resource manager): check the style and recoverd the test * fix(resource manager): delete the log.print(LoginUser)	3 years ago
caishunfeng	852597c9bf	[Bug-9070][Server]fix taskExecutionContext json convert error (#9072 ) * [Bug-9070][Server]fix taskExecutionContext json convert error * test * add license header Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
Kerwin	a1bef43981	[Feature-8579][Task] Task back-end plugin optimization, new plugins only need to modify the plugin's own module (#8584 ) * Task plug-in optimization, task-related classes are migrated to the task plugin module * remove tasktype enum class. * fix code style. * fix code style * fix code style * fix ProcessServiceTest.testUpdateResourceInfo test class * fix code style * fix test task plugin manager check parameters null * fix test ut * update aws license * change TaskTimeoutStrategy class	3 years ago
Ivan0626	60ddede6a6	fixbug: varPool value is error (#7724 )	3 years ago
ShuiMuNianHuaLP	806333a11c	fix sudo.enable=false Is invalid (#8388 )	3 years ago
Shiwen Cheng	0057cf3fe7	[Improvement][task-plugin] Remove unreasonable log (#8115 )	3 years ago
wind	e4cdd82b7c	[Improvement][TaskLog] Unified task log (#7831 ) * task logger * update * log format * remove master config task-logger * udpate log level Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
wangxj3	cc77963522	[FIX-#6727][worker-server] Fix procedure params bug (#7680 ) * fix Procedure param error * code style Co-authored-by: wangxj <wangxj31>	3 years ago
J·Y	6ae46f2c1b	[Improvement] Merge redundant codes (#7550 )	3 years ago
mask	0aea5ae11a	[feature-6071][plugin] add datasource plugin. (#6071 ) (#6361 ) * [feature-6071][plugin] add datasource plugin. (#6071) * remove the dependency of datasource in common and task-plugin module. * the new datasource-api module provides data source connection acquisition. * add sqlserver data source plug-in. * add ds license * fix check style * update spi pom * frx conflict resolution * fix check style * fix datasource ut * fix code check style * fix code check style * fix datasource ut * add pg hive mysql oracle clickhouse db2 datasource * fix datasource package * update datasource pom * update datasource pom * update datasource pom * update datasource pom * update pom * update pom dep * update pom dep * fix license * fix ut * fix ut * fix ut * fix ut * fix ut * fix ut * fix ut * fix ut * fix ut * fix ut * fix ut * fix ut * fix ut * add oracle package * modify comments * conflict resolution * rename log comment * rename log comment	3 years ago

1 2 3

119 Commits (1f365819a642e20aebe54da4b275f00f13719eab)