DolphinScheduler

Commit Graph

Author	SHA1	Message	Date
caishunfeng	6d0e367f69	cherry-pick [Bug-8110][WorkerServer] kill all running task before worker stop (#8508 )	3 years ago
caishunfeng	2dec94d3b4	[cherry-pick-2.0.4] [Bug] [dolphinscheduler-server] task run error when worker group name contains uppercase letters (#8459 ) * remove worker group lowercase (#8448) * add try catch (#8433) Co-authored-by: xiangzihao <460888207@qq.com>	3 years ago
xiangzihao	21a1d9cd9e	[Fix-8337] [Master] Process instance can not be kill when task is failure and can be retry (#8347 ) * fix bug_8337 * change kill logic * change kill logic	3 years ago
xiangzihao	9bfff90c66	[Fix-8367][Master] execute COMPLEMENT_DATA command always missing some ack event (#8407 ) * fix bug_8367 * fix bug_8367 * fix bug_8367 * fix bug_8367	3 years ago
Wenjun Ruan	de476edc38	Fix deadlock in master/worker caused by close method (#8361 )	3 years ago
xiangzihao	a322f9cf27	[Fix-8316][Master] Stop dependent node error (#8323 ) * fix bug_8316 * fix bug_8316 * fix bug_8316	3 years ago
zwZjut	d2bd95d453	[2.0.3][Bug] [dolphinscheduler-remote] process is always running: netty writeAndFlush without retry when failed leads to worker response to master failed (#8081 ) * to #8046 * to #8046 * to #8046 * to #8046 * to #8046 Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com>	3 years ago
wind	2d3c50cbae	[Bug-7942][Config] update config of master and worker (#7975 ) Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
wind	1f0aeabb92	[cherry-pick-2.0.3]Cherry pick to 2.0.3 (#8041 ) * [cherry-pick-2.0.3]Correct typo in English README #7695 * Update NOTICE year (#7784) * [BUG] Repair JDBC connection of Oracle (#7883) * [cherry-pick-2.0.3][Fix-7538] [server] Fix when there is a forbidden node in dag, the execution flow is abnormal #7613 * [cherry-pick-2.0.3][Fix-7825] Remedy the value of create time and update time to be current time when importing a process json file. #7828 * [FIX-7732][fix] fix column 'is_directory' of table `t_ds_resources` type error in PG database (#7898) Fix column 'is_directory' of table t_ds_resources type error in PG database This closes #7732 Co-authored-by: Brennan Fox <brnnnfx@users.noreply.github.com> Co-authored-by: Kirs <acm_master@163.com> Co-authored-by: X&Z <980813351@qq.com> Co-authored-by: caishunfeng <534328519@qq.com> Co-authored-by: 天仇 <532066967@qq.com>	3 years ago
wind	99cfd4864a	fix zk root (#7998 ) Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
JinYong Li	ceae42721f	[cherry-pick-7930][Improvement]support instance to definition (#7939 ) * Update ProcessDefinitionMapper.xml (#7931) Resolves process definition list issue（data duplication ） * pick 7930 * Update WorkflowExecuteThread.java * fix pg_ddl * fix pg_ddl Co-authored-by: mazhong <316422240@qq.com> Co-authored-by: BaoLiang <29528966+lenboo@users.noreply.github.com>	3 years ago
zwZjut	889274562e	to #7981 (#7982 ) Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com>	3 years ago
wind	90a33e06eb	[cherry-pick-2.0.3] cache manager (#7882 ) * cherry-pick [DS-6963][MasterServer]add cache manager for tenant and user * [cherry-pick][Feature-6988][MasterServer] add cache manager for workflow * [Feature][MasterServer] add cache for workGroup and schedule #6987 (#7161) * to #6987 * to #6987 * to #6987 * to #6987 * to #6987 * to #6987: @Param -> @Name * to #6987: fix Sonar * to #6987: fix Sonar Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> * [cherry-pick][Feature-6963][MasterServer] unified cache manager * [cherry-pick][Bug-7292][ApiServer] fix cache error when standalone * [cherry-pick][Fix-7245] [API] Fix spelEvaluationException: EL1008E: Property or field 'all' cannot be found * bugfix Co-authored-by: zwZjut <zw15068835817@163.com> Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> Co-authored-by: Kerwin <37063904+zhuangchong@users.noreply.github.com> Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
wind	46a5d17bf6	[cherry-pick-2.0.3][Improvement][TaskLog] Unified task log #7831 (#7926 ) * [cherry-pick][Improvement][TaskLog] Unified task log #7831 * [cherry-pick][Improvement][TaskLog] Unified task log #7831 * fix thread name Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
zwZjut	8dc529d35f	[dolphinscheduler-server] process instance is always running (#7914 ) * #7698 * #7698 * #7698 * fix workflow is always running * fix workflow is always running * fix workflow is always running * #7700 #7698 * add license * fix ut * #7700 #7698 Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com>	3 years ago
wind	5592e7bb7b	[Bug-7788][MasterServer] fix submit duplicate tasks sometimes when retry (#7808 ) (#7866 ) * [Bug-7788] fix submit duplicate tasks sometimes when retry * add exist check when add task to standby list * update * put queue contain judge first Co-authored-by: caishunfeng <534328519@qq.com> Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
kezhenxu94	4d525536f6	Remove init script when gaining a new DB connection (#7748 )	3 years ago
wind	85b3b2d29d	[DS-7686][Server]fix restart server after kill force (#7689 ) * [DS-7686][Server]fix restart server after kill force * update registry logic Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
zwZjut	32fb3f84df	[Bug] [dolphinscheduler-server] memory leak of logger #7661 (#7665 ) * [Feature][dolphinscheduler-api] parse traceId in http header for Cross system delivery to #7237 (#7238) * to #7237 * rerun test Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> * chery-pick `05aef27` and handle conflicts * to #7065: fix ExecutorService and schedulerService (#7072) Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> * [Feature][dolphinscheduler-api] access control of taskDefinition and taskInstance in project to #7081 (#7082) * to #7081 * fix #7081 * to #7081 Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> * chery-pick `8ebe060` and handle conflicts * cherry-pick `1f18444` and handle conflicts * fix #6807: dolphinscheduler.zookeeper.env_vars - > dolphinscheduler.registry.env_vars (#6808) Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> Co-authored-by: Kirs <acm_master@163.com> * add default constructor (#6780) Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> * to #7108 (#7109) * add conf && pick #7562 to worker * to #7661 * to #7661 * to #7661:fix ut Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> Co-authored-by: Kirs <acm_master@163.com>	3 years ago
zwZjut	df40cdc116	[Bug] [deploy on k8s] add default config of quartz and zookeeper in common configmap && pick #7562 fix to worker #7639 (#7641 ) * [Feature][dolphinscheduler-api] parse traceId in http header for Cross system delivery to #7237 (#7238) * to #7237 * rerun test Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> * chery-pick `05aef27` and handle conflicts * to #7065: fix ExecutorService and schedulerService (#7072) Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> * [Feature][dolphinscheduler-api] access control of taskDefinition and taskInstance in project to #7081 (#7082) * to #7081 * fix #7081 * to #7081 Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> * chery-pick `8ebe060` and handle conflicts * cherry-pick `1f18444` and handle conflicts * fix #6807: dolphinscheduler.zookeeper.env_vars - > dolphinscheduler.registry.env_vars (#6808) Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> Co-authored-by: Kirs <acm_master@163.com> * add default constructor (#6780) Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> * to #7108 (#7109) * add conf && pick #7562 to worker Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> Co-authored-by: Kirs <acm_master@163.com>	3 years ago
wind	744577f252	remove taskInstancnCacheManager (#7636 ) Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
Jiajie Zhong	9e068d23e2	[cherry-pick][python] Make dolphinscheduler python API works to 2.0.2 (#7608 ) * [cherry-pick][python] Make it work to 2.0.2 * Remove unused ProcessExecutionTypeEnum * Add queryByName to project * Add checkTenantExists to tenant * Add queryByTenantCode to tenant * Add queryQueueName to queue * Add all content from dev branch * Add gitignore * Add pydolphinscheduler content * Add ds-py to bin test * Py merge to 202 * Fix version * Fix missing variable * Add py4j as known deps * Fix core database bug	3 years ago
wind	8c5f364796	[cherry-pick-2.0.2][BUG-7554] Fix database lose data in standalone (#7621 ) * [cherry-pick][BUG-7554] Fix database lose data in standalone * Fix thread starts too early Co-authored-by: caishunfeng <534328519@qq.com> Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>	3 years ago
zwZjut	6c819ee16d	[Bug] [dolphinscheduler-server] workflow is always running when worker sendResult success but master not received (#7610 ) * [Feature][dolphinscheduler-api] parse traceId in http header for Cross system delivery to #7237 (#7238) * to #7237 * rerun test Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> * chery-pick `05aef27` and handle conflicts * to #7065: fix ExecutorService and schedulerService (#7072) Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> * [Feature][dolphinscheduler-api] access control of taskDefinition and taskInstance in project to #7081 (#7082) * to #7081 * fix #7081 * to #7081 Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> * chery-pick `8ebe060` and handle conflicts * cherry-pick `1f18444` and handle conflicts * fix #6807: dolphinscheduler.zookeeper.env_vars - > dolphinscheduler.registry.env_vars (#6808) Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> Co-authored-by: Kirs <acm_master@163.com> * add default constructor (#6780) Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> * to #7108 (#7109) * to #7609 Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> Co-authored-by: Kirs <acm_master@163.com>	3 years ago
zwZjut	f8942bf798	[Improvement][MasterServer] event response handle parallel (#7560 ) * [Feature][dolphinscheduler-api] parse traceId in http header for Cross system delivery to #7237 (#7238) * to #7237 * rerun test Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> * chery-pick `05aef27` and handle conflicts * to #7065: fix ExecutorService and schedulerService (#7072) Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> * [Feature][dolphinscheduler-api] access control of taskDefinition and taskInstance in project to #7081 (#7082) * to #7081 * fix #7081 * to #7081 Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> * chery-pick `8ebe060` and handle conflicts * cherry-pick `1f18444` and handle conflicts * fix #6807: dolphinscheduler.zookeeper.env_vars - > dolphinscheduler.registry.env_vars (#6808) Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> Co-authored-by: Kirs <acm_master@163.com> * add default constructor (#6780) Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> * to #7108 (#7109) * to #7450 * to #7450: fix parallel bug * add index * expose config to user * fix bug * fix bug * add delay delete * fix bug * add License * fix ut * fix ut * fix name Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> Co-authored-by: Kirs <acm_master@163.com>	3 years ago
BaoLiang	8fa61cf050	[FIX-7287] change description of 'master.exec.threads' (#7588 ) * [2.0.2] #7287 change description of 'master.exec.threads' * [2.0.2] #7287 change description of 'master.exec.threads'	3 years ago
xiangzihao	5c297a594c	fix 7534 (#7567 ) Co-authored-by: SbloodyS <sbloodys@qq.com>	3 years ago
SbloodyS	ab1467e622	cherry-pick bug_7537 (#7555 )	3 years ago
zwZjut	c956185e00	[Feature] 2.0.2-prepare bug fix of Pressure tests #7511 (#7540 ) * [Feature][dolphinscheduler-api] parse traceId in http header for Cross system delivery to #7237 (#7238) * to #7237 * rerun test Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> * chery-pick `05aef27` and handle conflicts * to #7065: fix ExecutorService and schedulerService (#7072) Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> * [Feature][dolphinscheduler-api] access control of taskDefinition and taskInstance in project to #7081 (#7082) * to #7081 * fix #7081 * to #7081 Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> * chery-pick `8ebe060` and handle conflicts * cherry-pick `1f18444` and handle conflicts * fix #6807: dolphinscheduler.zookeeper.env_vars - > dolphinscheduler.registry.env_vars (#6808) Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> Co-authored-by: Kirs <acm_master@163.com> * add default constructor (#6780) Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> * to #7108 (#7109) * to #7511 * to #7511 * to #7511 * to #7511 Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> Co-authored-by: Kirs <acm_master@163.com>	3 years ago
SbloodyS	0b6f3fb0c3	[Cherry-pick-2.0.2[Feature-7460][]Feature-7451][Fix-7392] (#7483 ) * remove date+1 when the data is supplemented (#7452) * resolve conflict * [Fix-7392][dolphinscheduler-datasource] Add hive datasource failed (#7393) * fix bug_7392 * fix bug_7392 * fix bug_7392 Co-authored-by: SbloodyS <sbloodys@qq.com> * cherry-pick-2.0.2 Co-authored-by: SbloodyS <sbloodys@qq.com>	3 years ago
zwZjut	255670fda9	[Feature][dev-2.0.2-prepare] cherry-pick pull requests from dev to 2.0.2-prepare #7357 (#7358 ) * [Feature][dolphinscheduler-api] parse traceId in http header for Cross system delivery to #7237 (#7238) * to #7237 * rerun test Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> * chery-pick `05aef27` and handle conflicts * to #7065: fix ExecutorService and schedulerService (#7072) Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> * [Feature][dolphinscheduler-api] access control of taskDefinition and taskInstance in project to #7081 (#7082) * to #7081 * fix #7081 * to #7081 Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> * chery-pick `8ebe060` and handle conflicts * cherry-pick `1f18444` and handle conflicts * fix #6807: dolphinscheduler.zookeeper.env_vars - > dolphinscheduler.registry.env_vars (#6808) Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> Co-authored-by: Kirs <acm_master@163.com> * add default constructor (#6780) Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> * to #7108 (#7109) Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com> Co-authored-by: Kirs <acm_master@163.com>	3 years ago
BaoLiang	921f388077	[FIX-7338] set h2 for default DATABASE_TYPE for standalone-server (#7339 ) * [Bug] [dolphinscheduler-api] The pages of project management, workflow instance and task instance are accessed slowly #7061 (#7139) 1、I found that the SQL behind this interface is very slow. The main table is large, but there is no index to use. By add the index, the all mode in the query plan can be changed into a more efficient ref mode. If this optimization still fails to meet the requirements, please contact me and I will continue to optimize。 2、modify index start_time_index in init script and upgrade scripts. * [BUG][SQL] fix the process list page load slowly. (#7166) * fix #7601 move ddl to 2.0.1 * fix #7601 move ddl to 2.0.1 * [2.0.1] remove unused foreign_key * [2.0.1] add version in sql * [2.0.1] change serial for postgre * [2.0.1] set h2 for default DATABASE_TYPE for standalone-server Co-authored-by: GaoTianDuo <gaotianduo_yewu@cmss.chinamobile.com>	3 years ago
wind	79ab27299e	[cherry-pick-2.0.1][Bug-7319][MasterServer] fix NPE when switch else branch is empty (#7314 ) * fix taskNode NPE * fix switch node NPE when no else branch Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
wind	5dad635ff0	[cherry-pick-2.0.1][Bug-7307][MasterServer] timeout event was filtered (#7303 ) * remove task filter when task timeout * get lastest task instance info Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
wind	41c05b53c7	[cherry-pick-2.0.1][Bug][MasterServer] fix timeout event (#7283 ) * fix timeout alert * fix timeout Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
wind	9a307dc71a	fix failover logic (#7276 )	3 years ago
kezhenxu94	a8edeac1e3	Fix env vars are not passed in from docker-compose (#7240 )	3 years ago
wind	30848478bf	fix timeout (#7224 ) Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
wind	c2d0acd713	fix timeout (#7222 ) Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
Kerwin	b4c422e977	fix waterdrop task plugin (#7216 )	3 years ago
kezhenxu94	e9d27015c3	Add missing property for API and worker (#7198 )	3 years ago
Jiajie Zhong	1e0757bd8e	Change default database from mysql to pg (#7190 )	3 years ago
wind	0b376f7f90	[2.0.1-cherry-pick] fix bug : Tez state, complement data kill, dependent node (#7188 ) * [Fix-7037][dolphincheduler-server] process instance can not be killed when complement data (#7140) * fix bug 7037 * update processComplementData Co-authored-by: SbloodyS <sbloodys@qq.com> * [Fix-5551][common] Fix ExecutionStatus for hive-on-Tez application (#6289) * add ENDED state * Update HadoopUtils.java * [Fix-7093][dolphincheduler-server] fix dependent node stuck in running (#7116) * fix bug 7093 * remove redundant else Co-authored-by: SbloodyS <sbloodys@qq.com> Co-authored-by: SbloodyS <460888207@qq.com> Co-authored-by: SbloodyS <sbloodys@qq.com> Co-authored-by: Manhua <kevinjmh@qq.com>	3 years ago
kezhenxu94	449c098aef	Fix quartz not work, worker / alert server should not listen or 8080 port (#7159 )	3 years ago
Jiajie Zhong	423d7ea3a1	[deploy] Add export database type for before daemon file (#7151 ) * [deploy] Use install_config.conf export datasource properties Directly user install_config.conf to set spring datasource properties, delete datasource.properties file for more clearly * Remove unused properties	3 years ago
wind	36fd0c83f5	[Fix-7019][MasterServer] Fix the issue with failing to save and run the process that contains the switch task. (#7046 ) (#7059 ) Co-authored-by: Hua Jiang <jianghuachinacom@163.com>	3 years ago
Kirs	2bf3b22d64	[2.0.1-cherrypick] SPI(Task ,Alert ,DataSource) (#7041 ) * data source spi * Fix datasource not work in tests and standalone server, and run all tests by default (#6743) * Refactor alert plugins and simplify its usage (#6723) * [Fix-6792] resolve API and alert port conflicts (#6792) (#6815) * [Fix-6792]Multiple ApplicationRunners on classpath make SpringApplication.run only run one (#6792) * [Feature][datasource] Change DataSource Connection Pool from Druid to HiKariCP (#6490) (#6828) * [Feature][datasource] Change DataSource Connection Pool from Druid to HiKariCP (#6490) * fix pom dep scope * filter alert * filter alert * filter alert * fix checkstyle * remove hikaricp version * update pom * add properties * add properties * rename properties * fix e2e fail * rename properties * [Fix-6792] resolve API and alert port conflicts (#6792) (#6815) [Fix-6792]Multiple ApplicationRunners on classpath make SpringApplication.run only run one (#6792) * fix ut fail * fix license * fix license * update package * update package * remove unused license Co-authored-by: mask <39329477+narcasserun@users.noreply.github.com> Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>	3 years ago
wind	19cb1237b3	[cherry-pick][DS-7004][MasterServer]fix master still work when it lose zk connection (#7058 )	3 years ago
Kirs	6821c7a9ac	[2.0.1-prepare#6712]Refactor registry plugin and simplify its usage (#6999 ) * [2.0.1-prepare#6712]Refactor registry plugin and simplify its usage (#6712) * upgrade version and fix dep err * Clean up the utility codes (#6732) * remove code style check * upgrade docker version Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>	3 years ago
OS	652b9230a7	[BUG][Master] process cannot finish and its status is always running (#6886 ) * fix 6882: process cannot finish and its status is always running * fix 6882: process cannot finish and its status is always running	3 years ago

1 2 3 4 5 ...

658 Commits (e608f4ad42915e14b822ec4539889c00a4c0ed5f)