DolphinScheduler

Commit Graph

Author	SHA1	Message	Date
Wenjun Ruan	66624c5c86	[Bug] [Master] Worker failover will cause task cannot be failover (#10631 ) * fix worker failover may lose event	3 years ago
Wenjun Ruan	3ecbee3885	Optimize MasterServer, add MasterRPCService (#10371 ) * Optimize MasterServer, avoid NPE	3 years ago
kezhenxu94	d80cf21456	Clean up unused dependencies and packaging issues (#9944 )	3 years ago
Eric Gao	fd6b43bc81	[Dev] Switch version in pom.xml to dev-SNAPSHOT (#9223 ) (#9299 )	3 years ago
Kerwin	a1bef43981	[Feature-8579][Task] Task back-end plugin optimization, new plugins only need to modify the plugin's own module (#8584 ) * Task plug-in optimization, task-related classes are migrated to the task plugin module * remove tasktype enum class. * fix code style. * fix code style * fix code style * fix ProcessServiceTest.testUpdateResourceInfo test class * fix code style * fix test task plugin manager check parameters null * fix test ut * update aws license * change TaskTimeoutStrategy class	3 years ago
caishunfeng	84f2cfec39	[Feature][Server] Custom timezone - Add timezone handler unified (#8477 ) * date convert of timezone * remove @JsonFormat * add unit test * fix time preview in scheduler * optimization & add env config Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
zixi0825	4ba6d667fc	[Feature][DataQuality] Add Data quality Module #4283 (#6718 ) * add data quality module * add license * add package configuration in dist pom * fix license and jar import bug * replace apache/skywalking-eyes@9bd5feb SHA * refacotr jbdc-connector and writer * modify parameter name in HiveConnector * fix checkstyle error * fix checkstyle error in dolphinschesuler-dist * fix checkstyle error in dolphinschesuler-dist * fix checkstyle error in dolphinschesuler-dist * fix duplicate code bug * fix code style bug * fix code smells * add dq relevant enums and parameter * replace apache/skywalking-eyes@9bd5feb SHA * fix Constants bug * remove the unused class * add unit test * fix code style error * add unit test * refactor data quality common entity * fix code style error * add unit test * close e2e test * fix code smell bug * modify dataquality enum value to 14 in TaskType * add data qualtiy task * update * add getDatasourceOptions interface * fix checkstyle * close e2e test * add data quality task ui * update skywalking-eyes SHA * fix style * fix eslint error * fix eslint error * test e2e * add unit test and alter dataquality task result * fix checkstyle * fix process service test error * add unit test and fix code smells * fix checkstyle error * fix unit test error * fix checkstyle error * change execute sql type name * revert ui pom.xml * fix data quality task error * fix checkstyle error * fix dq task src_connector_type ui select bug * fix spark rw postgresql bug * change mysql driver scope * fix form-create json bug * fix code smell * fix DolphinException Bug * fix ui validate rule and Alert title * fix target connection param bug * fix threshold validate change * add rule input entry index * change statistic_comparison_check logic * remove check type change * add DateExpressionReplaceUtil * fix null point expetion * fix null point expetion * fix test error * add more sql driver * fix test error and remove DateExprReplaceUtil * add get datasource tables and columns * add get datasource tables and columns * remove hive-jdbc in pom.xml * fix code smells * update sql * change the pom.xml * optimize multi_table_accuracy ui * fix v-show error * fix code smells * update sql * [Feature][DataQuality] Add data quality task ui (#5054) * add data quality task ui * update skywalking-eyes SHA * fix style * fix eslint error * fix eslint error * test e2e * fix dq task src_connector_type ui select bug * fix threshold validate change * remove check type change * add get datasource tables and columns * optimize multi_table_accuracy ui * fix v-show error * fix code smells Co-authored-by: sunchaohe <sunzhaohe@linklogis.com> * [Feature][DataQuality] Add data quality module (#4830) * add data quality module * add license * add package configuration in dist pom * fix license and jar import bug * replace apache/skywalking-eyes@9bd5feb SHA * refacotr jbdc-connector and writer * modify parameter name in HiveConnector * fix checkstyle error * fix checkstyle error in dolphinschesuler-dist * fix checkstyle error in dolphinschesuler-dist * fix checkstyle error in dolphinschesuler-dist * fix duplicate code bug * fix code style bug * fix code smells * update * close e2e test * fix spark rw postgresql bug * change mysql driver scope * add more sql driver * remove hive-jdbc in pom.xml * change the pom.xml Co-authored-by: sunchaohe <sunzhaohe@linklogis.com> * [Feature][DataQuality] Add data quality task backend (#4883) * add dq relevant enums and parameter * replace apache/skywalking-eyes@9bd5feb SHA Co-authored-by: sunchaohe <sunzhaohe@linklogis.com> * refactor data_quality_module * add header license * data quality module refactor * fix unit test error * fix checkstyle error * fix unit test error * fix checkstyle error * fix unit test error * fix code smell * fix check style * fix unit test error * task statistics value add unique code * fix unit test error * fix checkstyle error * fix checkstyle * fix security hotspot * fix unit test error * fix security hotspot * fix check * add data quality task error handling * fix unit test error * add unit test * add unit test * optimize data quality result alert * fix unit test * fix sql script error * fix bug * update sql script * fix checkstyle * add license * fix checkstyle * fix checkstyle * fix unit test * add jacoco dependencies * fix unit test * fix unit test * add jacoco dependencies * add unit test * add unit test * add license * fix checkstyle * fix pom * fix checkstyle * fix checkstyle * merge dev * fix ui error * fix pom error * fix pom error * fix test error * fix test error * mssql-jdbc exclude azure-keyvault * fix test error * merge dev and add unit test * add notes * rollback the CollectionUtils * fix * update sql * fix * fix * fix query rule page error * change dq.jar path * fix sql error * fix ui error * fix(dq): jar path&task enum description * add notes on DataQualityApplication * fix dq result jump error * fix(ui): page condition * feat(ui): add show error output path * change version * remove all chinese word in sql * merge Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>	3 years ago
Kerwin	103ccb01da	change version to 2.0.4-SNAPSHOT (#8188 )	3 years ago
wind	e4cdd82b7c	[Improvement][TaskLog] Unified task log (#7831 ) * task logger * update * log format * remove master config task-logger * udpate log level Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
aaronlinv	707e3a6ec1	[Improvement][Logger] Logger server integrate into master and worker (#6894 )	3 years ago
kezhenxu94	67cc260d52	Split the components into individual package, refactor configurations to use Spring properties (#7160 ) * Split the components into individual package A follow-up PR will be made to build dedicated Docker images for each component, so that every component Docker image has minimal jars, which is easy to maintain and good for security fixes. * Split the components into individual package A follow-up PR will be made to build dedicated Docker images for each component, so that every component Docker image has minimal jars, which is easy to maintain and good for security fixes. * Split the components into individual package A follow-up PR will be made to build dedicated Docker images for each component, so that every component Docker image has minimal jars, which is easy to maintain and good for security fixes.	3 years ago
wind	3a351da752	fix failover logic (#7275 ) Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
zwZjut	f577b5bd30	to #7228 (#7249 ) Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com>	3 years ago
wind	a80a7b3fcf	[Feature-6963][MasterServer] unified cache manager (#7187 ) * unified cache * reduce db select * checkstyle Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
wind	ff9bc806ac	[Bug-7206] [MasterServer] fix process isntance always running when task timeout (#7207 ) * fix timeout * add task timeout map to avoid repeated timeout event * split task check list for retry and timeout Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
Kerwin	36ebfe0984	[Implement][server]TaskProcessor code optimization (#7146 ) * TaskProcessor code optimization * fix code style * update Collectors.toMap	3 years ago
SbloodyS	d83735ab51	[Fix-7037][dolphincheduler-server] process instance can not be killed when complement data (#7140 ) * fix bug 7037 * update processComplementData Co-authored-by: SbloodyS <sbloodys@qq.com>	3 years ago
SbloodyS	8d39bf14b1	[Fix-7093][dolphincheduler-server] fix dependent node stuck in running (#7116 ) * fix bug 7093 * remove redundant else Co-authored-by: SbloodyS <sbloodys@qq.com>	3 years ago
zwZjut	af9ed35295	[Feature][MasterServer] add cache for workGroup and schedule #6987 (#7161 ) * to #6987 * to #6987 * to #6987 * to #6987 * to #6987 * to #6987: @Param -> @Name * to #6987: fix Sonar * to #6987: fix Sonar Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com>	3 years ago
wind	ba2b2a67c2	[Feature-6988][MasterServer] add cache manager for workflow (#7090 ) * add cache manager for workflow * [DS-6988][MasterServer] add cache manager for workflow * cache evict code optimization * test Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
wangxj3	2b519f31e9	[Feature-#6422] task group queue (#6722 ) * add task group * modify task group * pull dev * add license header * modify code style * fix code style * fix sql error * fix error * fix test * fix test * fix test * fix test * fix code style * fix ut * code style * fix unit test * test ut * ut * add unittest * test ut * modify back ut * majorization code * fix conflict * fix ut Co-authored-by: wangxj <wangxj31>	3 years ago
Kerwin	9fffbdb316	add TaskProcessorFactoryTest Ignore. (#7130 )	3 years ago
zwZjut	c42531e39c	to #7108 (#7109 )	3 years ago
kezhenxu94	2ed3c0c3a7	Recover unit tests that have not been run for some time (#7054 )	3 years ago
Sunny Lei	0dcff1425a	[DS-5229][feat] Implement server port custom config (#6947 ) * [DS-5229][fix] server port custom config This closes #5229 * [DS-5229][feat] Implement server port custom config This closes #5229 * [Bug] [readme] Error link to Docker and k8s in readme apache#6802 (#6862) * [Bug] [readme] Error link to Docker and k8s in readme #6802 1、modify the error link * [Bug] [readme] Error link to Docker and k8s in readme #6802 1、modify the error link in readme_zh_cn.md * [DS-6829][WorkerServer] skip create log dir and print log in dryRun model (#6852) Co-authored-by: caishunfeng <534328519@qq.com> * [DS-5229][feat] Implement server port custom config Modify review suggestion This closes #5229 Co-authored-by: GaoTianDuo <gaotianduo_yewu@cmss.chinamobile.com> Co-authored-by: wind <caishunfeng@users.noreply.github.com> Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
zwZjut	100a9ba3fc	to #7055 : tenant-auto-create: true (#7056 ) Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com>	3 years ago
zhang	c2585a1170	[BUG] master task log can't output repair (#7029 ) * update task_definition_log field typ update task_definition_log field task_params type,modify text to longtext to be consistent with the main table * update t_ds_task_definition_log field task_params update t_ds_task_definition_log field task_params * [FIX:211128] master task log Can't output repair Co-authored-by: crane <zhangbq3306@163.com>	3 years ago
wind	be3bd4c83d	[DS-7004][MasterServer]fix master still work when it lose zk connection (#7045 ) Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
Hua Jiang	a0955a078a	[Fix-7019][MasterServer] Fix the issue with failing to save and run the process that contains the switch task. (#7046 )	3 years ago
wind	8e93939121	[DS-6963][MasterServer]add cache manager for tenant and user (#6974 ) * cache manager for tenant and user * add test * fix checkstyle * add dependencies * fix sonar check * use spring boot cache configuration * add license * check style * check style * add license * change proxy name to processor * add license header * check style * add license file * add license file * add license Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
wind	e6fe39ea47	[DS-6961][MasterServer] batch dispatch (#6962 ) * [DS-6961][MasterServer] batch dispatch * fix test Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
kezhenxu94	2de4a3cbb5	Add a demo docker-compose for testing the meter module (#6938 )	3 years ago
wind	e6239e808b	[DS-6829][WorkerServer] skip create log dir and print log in dryRun model (#6852 ) Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
wind	28872d8706	[DS-6891][MasterServer] reduce db operation when process instance running (#6904 ) * remove taskInstanceCacheManager * remove taskInstanceCacheManager * [DS-6891][MasterServer] reduce db operation when process instance running * [DS-6891][MasterServer] reduce db operation when process instance running * fix test * fix Transactional method call * checkstyle Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
wind	595e4843d0	[DS-6849][MasterServer] fetch more commands and handle in parallel (#6850 ) * [DS-6849][MasterServer] fetch more commands and handle in parallel * add return * handle command with check * add test * delete master prefix Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
OS	653eae2419	[BUG][Master] process cannot finish and its status is always running (#6886 ) * fix 6882: process cannot finish and its status is always running * fix 6882: process cannot finish and its status is always running	3 years ago
chouc	f564687a89	[Improvement][Log] optimize task log path to reduce task running time (#6731 ) * [Improvement][Log] optimize task log path to reduce task running time * Reduce the creation of log directories Co-authored-by: Kirs <acm_master@163.com>	3 years ago
KyoYang	94f6c5c27e	fix complement data retry bug; issues: [Bug] [Master] A bug on task retry mechanism #6613 (#6843 ) Co-authored-by: yangqiyu <yangqiyu@deepexi.com>	3 years ago
kezhenxu94	4114cb07f6	Add actuator and enable metrics, add Grafana configs (#6840 )	3 years ago
Jiajie Zhong	6045728428	[ci] Add checkstyle prohibit rule System.out.println (#6778 ) * [ci] Add checkstyle prohibit rule System.out.println * Correct RegexpSingleline place	3 years ago
wind	5741c758b3	[DS-6771][MasterServer] fix failover worker interrupt (#6801 ) Co-authored-by: caishunfeng <534328519@qq.com> Co-authored-by: Kirs <acm_master@163.com> Co-authored-by: OS <29528966+lenboo@users.noreply.github.com>	3 years ago
mask	e36d18b588	[Fix-6792] resolve API and alert port conflicts (#6792 ) (#6815 ) * [Fix-6792]Multiple ApplicationRunners on classpath make SpringApplication.run only run one (#6792)	3 years ago
kezhenxu94	11844415b0	Refactor datasource and task plugins and simplify its usage (#6767 )	3 years ago
mask	0aea5ae11a	[feature-6071][plugin] add datasource plugin. (#6071 ) (#6361 ) * [feature-6071][plugin] add datasource plugin. (#6071) * remove the dependency of datasource in common and task-plugin module. * the new datasource-api module provides data source connection acquisition. * add sqlserver data source plug-in. * add ds license * fix check style * update spi pom * frx conflict resolution * fix check style * fix datasource ut * fix code check style * fix code check style * fix datasource ut * add pg hive mysql oracle clickhouse db2 datasource * fix datasource package * update datasource pom * update datasource pom * update datasource pom * update datasource pom * update pom * update pom dep * update pom dep * fix license * fix ut * fix ut * fix ut * fix ut * fix ut * fix ut * fix ut * fix ut * fix ut * fix ut * fix ut * fix ut * fix ut * add oracle package * modify comments * conflict resolution * rename log comment * rename log comment	3 years ago
kezhenxu94	05bae0b226	Refactor alert plugins and simplify its usage (#6723 )	3 years ago
kezhenxu94	02350ea37b	Fix datasource not work in tests and standalone server, and run all tests by default (#6743 )	3 years ago
kezhenxu94	63602bf77c	Clean up the utility codes (#6732 )	3 years ago
wind	b6824b4741	[DS-6737][MasterServer] fix event handle twice (#6738 ) Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
OS	24bd8a16e1	[Bug][Master] The task of dependent should use taskCode (#6730 ) * fix #6682 The task of dependent should use taskCode * fix #6682 The task of dependent should use taskCode	3 years ago
Wenjun Ruan	0afb780718	[Improvement-5807][Dao] Use h2 in UT (#6500 ) * Use H2 in UT * change scope * Change profile name to h2 * Add datasource-h2.properties, datasource-postgresql.properties * Fix UT * Resolve conflict * Remove startDatabase	3 years ago

1 2 3 4 5 ...

737 Commits (499e5b1307665928f2a17e878d091ac593adbfd8)