DolphinScheduler

Commit Graph

Author	SHA1	Message	Date
xiangzihao	7f41a96fc1	[Fix-9525] [Worker] Environment did not work as expected (#9527 ) * fix #9525 * change to ${PYTHON_HOME} * remove import * fix ut error	3 years ago
Tq	04d4e4e0c4	fix the OUT direct param could pass to the current script (#9463 )	3 years ago
caishunfeng	b285ccf930	[Future-9396]Support output parameters transfer from parent workflow to child work flow (#9410 ) * [Future-9396]Support output parameters transfer from parent workflow to child work flow * fix note	3 years ago
Kerwin	923f3f38e3	[Fix-9316] [Task] Configure DB2 data source SQL script execution report ResultSet has been closed exception in SQL task (#9317 ) * fix db2 error in the sql task * update limit in sql task	3 years ago
nobolity	037692517a	[Fix-9251] [WORKER] reslove the sql task about of add the udf resource failed (#9319 ) * feat(resource manager): extend s3 to the storage of ds 1.fix some spell question 2.extend the type of storage 3.add the s3utils to manager resource 4.automatic inject the storage in addition to your config * fix(resource manager): update the dependency * fix(resource manager): extend s3 to the storage of ds fix the constant of hadooputils * fix(resource manager): extend s3 to the storage of ds 1.fix some spell question 2.delete the import * * fix(resource manager): merge the unitTest: 1.TenantServiceImpl 2.ResourceServiceImpl 3.UserServiceImpl * fix(resource manager): extend s3 to the storage of ds merge the resourceServiceTest * fix(resource manager): test cancel the test method createTenant verifyTenant * fix(resource manager): merge the code follow the check-result of sonar * fix(resource manager): extend s3 to the storage of ds fit the spell question * fix(resource manager): extend s3 to the storage of ds revert the common.properties * fix(resource manager): extend s3 to the storage of ds update the storageConfig with None * fix(resource manager): extend s3 to the storage of ds fix the judge of resourceType * fix(resource manager): extend s3 to the storage of ds undo the compile-mysql * fix(resource manager): extend s3 to the storage of ds delete hadoop aws * fix(resource manager): extend s3 to the storage of ds update the know-dependencies to delete aws 1.7.4 update the e2e file-manager common.properties * fix(resource manager): extend s3 to the storage of ds update the aws-region * fix(resource manager): extend s3 to the storage of ds fix the storageconfig init * fix(resource manager): update e2e docker-compose update e2e docker-compose * fix(resource manager): extend s3 to the storage of ds revent the e2e common.proprites print the resource type in propertyUtil * fix(resource manager): extend s3 to the storage of ds 1.println the properties * fix(resource manager): println the s3 info * fix(resource manager): extend s3 to the storage of ds delete the info and upgrade the s3 info to e2e * fix(resource manager): extend s3 to the storage of ds add the bucket init * fix(resource manager): extend s3 to the storage of ds 1.fix some spell question 2.delete the import * * fix(resource manager): extend s3 to the storage of ds upgrade the s3 endpoint * fix(resource manager): withPathStyleAccessEnabled(true) * fix(resource manager): extend s3 to the storage of ds 1.fix some spell question 2.delete the import * * fix(resource manager): upgrade the s3client builder * fix(resource manager): correct the s3 point to s3client * fix(resource manager): update the constant BUCKET_NAME * fix(resource manager): e2e s3 endpoint -> s3:9000 * fix(resource manager): extend s3 to the storage of ds 1.fix some spell question 2.delete the import * * style(resource manager): add info to createBucket * style(resource manager): debug the log * ci(resource manager): test test s3 * ci(ci): add INSERT INTO dolphinscheduler.t_ds_tenant (id, tenant_code, description, queue_id, create_time, update_time) VALUES(1, 'root', NULL, 1, NULL, NULL); to h2.sql * fix(resource manager): update the h2 sql * fix(resource manager): solve to delete the tenant * style(resource manager): merge the style end delete the unuse s3 config * fix(resource manager): extend s3 to the storage of ds UPDATE the rename resources when s3 * fix(resource manager): extend s3 to the storage of ds 1.fix the code style of QuartzImpl * fix(resource manager): extend s3 to the storage of ds 1.impoort restore_type to CommonUtils * fix(resource manager): update the work thread * fix(resource manager): update the baseTaskProcessor * fix(resource manager): upgrade dolphinscheduler-standalone-server.xml * fix(resource manager): add user Info to dolphinscheduler_h2.sql * fix(resource manager): merge the resourceType to NONE * style(upgrade the log level to info): * fix(resource manager): sysnc the h2.sql * fix(resource manager): update the merge the user tenant * fix(resource manager): merge the resourcesServiceImpl * fix(resource manager): when the storage is s3 ,that the directory can't be renamed * fix(resource manager): in s3 ,the directory cannot be renamed * fix(resource manager): delete the deleteRenameDirectory in E2E * fix(resource manager): check the style and recoverd the test * fix(resource manager): delete the log.print(LoginUser) * fix(server): fix the udf serialize * fix(master task): update the udfTest to update the json string * fix(test): update the udfFuncTest * fix(common): syn the common.properties * fix(udfTest): upgrade the udfTest * fix(common): revent the common.properties	3 years ago
Tq	2a4fa9cdb1	[BUG][WORKER-9349]fix param priority (#9379 ) * fix param priority * fix params priority code logic	3 years ago
Eric Gao	fd6b43bc81	[Dev] Switch version in pom.xml to dev-SNAPSHOT (#9223 ) (#9299 )	3 years ago
caishunfeng	ddf1ff98fa	[Improvement-7697][Master/Worker] Change the task ack to runnning callback (#8719 ) * rebase dev * change task state to dispatch if dispatch success * update task host when dispatch * add dispatch task event * test * check tenant after enable auto create * handle dispatch state Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
Kerwin	d7d756e7b0	Fix DataX and Sqoop Task execute NullPointerException (#9192 )	3 years ago
nobolity	0e3cafec1d	[Feature-8612][RESOURCE] extend s3 to the storage of ds (#8637 ) * feat(resource manager): extend s3 to the storage of ds 1.fix some spell question 2.extend the type of storage 3.add the s3utils to manager resource 4.automatic inject the storage in addition to your config * fix(resource manager): update the dependency * fix(resource manager): extend s3 to the storage of ds fix the constant of hadooputils * fix(resource manager): extend s3 to the storage of ds 1.fix some spell question 2.delete the import * * fix(resource manager): merge the unitTest: 1.TenantServiceImpl 2.ResourceServiceImpl 3.UserServiceImpl * fix(resource manager): extend s3 to the storage of ds merge the resourceServiceTest * fix(resource manager): test cancel the test method createTenant verifyTenant * fix(resource manager): merge the code follow the check-result of sonar * fix(resource manager): extend s3 to the storage of ds fit the spell question * fix(resource manager): extend s3 to the storage of ds revert the common.properties * fix(resource manager): extend s3 to the storage of ds update the storageConfig with None * fix(resource manager): extend s3 to the storage of ds fix the judge of resourceType * fix(resource manager): extend s3 to the storage of ds undo the compile-mysql * fix(resource manager): extend s3 to the storage of ds delete hadoop aws * fix(resource manager): extend s3 to the storage of ds update the know-dependencies to delete aws 1.7.4 update the e2e file-manager common.properties * fix(resource manager): extend s3 to the storage of ds update the aws-region * fix(resource manager): extend s3 to the storage of ds fix the storageconfig init * fix(resource manager): update e2e docker-compose update e2e docker-compose * fix(resource manager): extend s3 to the storage of ds revent the e2e common.proprites print the resource type in propertyUtil * fix(resource manager): extend s3 to the storage of ds 1.println the properties * fix(resource manager): println the s3 info * fix(resource manager): extend s3 to the storage of ds delete the info and upgrade the s3 info to e2e * fix(resource manager): extend s3 to the storage of ds add the bucket init * fix(resource manager): extend s3 to the storage of ds 1.fix some spell question 2.delete the import * * fix(resource manager): extend s3 to the storage of ds upgrade the s3 endpoint * fix(resource manager): withPathStyleAccessEnabled(true) * fix(resource manager): extend s3 to the storage of ds 1.fix some spell question 2.delete the import * * fix(resource manager): upgrade the s3client builder * fix(resource manager): correct the s3 point to s3client * fix(resource manager): update the constant BUCKET_NAME * fix(resource manager): e2e s3 endpoint -> s3:9000 * fix(resource manager): extend s3 to the storage of ds 1.fix some spell question 2.delete the import * * style(resource manager): add info to createBucket * style(resource manager): debug the log * ci(resource manager): test test s3 * ci(ci): add INSERT INTO dolphinscheduler.t_ds_tenant (id, tenant_code, description, queue_id, create_time, update_time) VALUES(1, 'root', NULL, 1, NULL, NULL); to h2.sql * fix(resource manager): update the h2 sql * fix(resource manager): solve to delete the tenant * style(resource manager): merge the style end delete the unuse s3 config * fix(resource manager): extend s3 to the storage of ds UPDATE the rename resources when s3 * fix(resource manager): extend s3 to the storage of ds 1.fix the code style of QuartzImpl * fix(resource manager): extend s3 to the storage of ds 1.impoort restore_type to CommonUtils * fix(resource manager): update the work thread * fix(resource manager): update the baseTaskProcessor * fix(resource manager): upgrade dolphinscheduler-standalone-server.xml * fix(resource manager): add user Info to dolphinscheduler_h2.sql * fix(resource manager): merge the resourceType to NONE * style(upgrade the log level to info): * fix(resource manager): sysnc the h2.sql * fix(resource manager): update the merge the user tenant * fix(resource manager): merge the resourcesServiceImpl * fix(resource manager): when the storage is s3 ,that the directory can't be renamed * fix(resource manager): in s3 ,the directory cannot be renamed * fix(resource manager): delete the deleteRenameDirectory in E2E * fix(resource manager): check the style and recoverd the test * fix(resource manager): delete the log.print(LoginUser)	3 years ago
Kerwin	25fc1dcb5f	Fix the exception of task getting external resource data during execution (#9134 )	3 years ago
caishunfeng	852597c9bf	[Bug-9070][Server]fix taskExecutionContext json convert error (#9072 ) * [Bug-9070][Server]fix taskExecutionContext json convert error * test * add license header Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
zixi0825	63a10037a6	fix package bug (#9051 )	3 years ago
Kerwin	a1bef43981	[Feature-8579][Task] Task back-end plugin optimization, new plugins only need to modify the plugin's own module (#8584 ) * Task plug-in optimization, task-related classes are migrated to the task plugin module * remove tasktype enum class. * fix code style. * fix code style * fix code style * fix ProcessServiceTest.testUpdateResourceInfo test class * fix code style * fix test task plugin manager check parameters null * fix test ut * update aws license * change TaskTimeoutStrategy class	3 years ago
zixi0825	6a02b15e40	[Fix][data-quality] change data-quality task main jar path (#8752 ) * feat(data-quality) change data-quality task main jar path * feat(data-quality) change data-quality task main jar path Co-authored-by: zixi0825 <sunzhaohe@linklogis.com>	3 years ago
ronyang1985	20ee9a37fe	[Feature-8485][task] Add EMR task plugin to support submit task to AWS EMR (#8503 ) Co-authored-by: Wenjun Ruan <wenjun@apache.org> Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>	3 years ago
yimaixinchen	a72742b9de	remove unuse import jsonFormat (#8548 )	3 years ago
caishunfeng	84f2cfec39	[Feature][Server] Custom timezone - Add timezone handler unified (#8477 ) * date convert of timezone * remove @JsonFormat * add unit test * fix time preview in scheduler * optimization & add env config Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
Yao WANG	dd1397aaed	[Bug] [Task-plugin] ShellTask add the parentfolder exist check and create if not exit (#8526 ) * check the parent dir exist and create it * fix lint Co-authored-by: ywang46 <ywang46@paypal.com>	3 years ago
Ivan0626	60ddede6a6	fixbug: varPool value is error (#7724 )	3 years ago
ShuiMuNianHuaLP	806333a11c	fix sudo.enable=false Is invalid (#8388 )	3 years ago
zixi0825	4ba6d667fc	[Feature][DataQuality] Add Data quality Module #4283 (#6718 ) * add data quality module * add license * add package configuration in dist pom * fix license and jar import bug * replace apache/skywalking-eyes@9bd5feb SHA * refacotr jbdc-connector and writer * modify parameter name in HiveConnector * fix checkstyle error * fix checkstyle error in dolphinschesuler-dist * fix checkstyle error in dolphinschesuler-dist * fix checkstyle error in dolphinschesuler-dist * fix duplicate code bug * fix code style bug * fix code smells * add dq relevant enums and parameter * replace apache/skywalking-eyes@9bd5feb SHA * fix Constants bug * remove the unused class * add unit test * fix code style error * add unit test * refactor data quality common entity * fix code style error * add unit test * close e2e test * fix code smell bug * modify dataquality enum value to 14 in TaskType * add data qualtiy task * update * add getDatasourceOptions interface * fix checkstyle * close e2e test * add data quality task ui * update skywalking-eyes SHA * fix style * fix eslint error * fix eslint error * test e2e * add unit test and alter dataquality task result * fix checkstyle * fix process service test error * add unit test and fix code smells * fix checkstyle error * fix unit test error * fix checkstyle error * change execute sql type name * revert ui pom.xml * fix data quality task error * fix checkstyle error * fix dq task src_connector_type ui select bug * fix spark rw postgresql bug * change mysql driver scope * fix form-create json bug * fix code smell * fix DolphinException Bug * fix ui validate rule and Alert title * fix target connection param bug * fix threshold validate change * add rule input entry index * change statistic_comparison_check logic * remove check type change * add DateExpressionReplaceUtil * fix null point expetion * fix null point expetion * fix test error * add more sql driver * fix test error and remove DateExprReplaceUtil * add get datasource tables and columns * add get datasource tables and columns * remove hive-jdbc in pom.xml * fix code smells * update sql * change the pom.xml * optimize multi_table_accuracy ui * fix v-show error * fix code smells * update sql * [Feature][DataQuality] Add data quality task ui (#5054) * add data quality task ui * update skywalking-eyes SHA * fix style * fix eslint error * fix eslint error * test e2e * fix dq task src_connector_type ui select bug * fix threshold validate change * remove check type change * add get datasource tables and columns * optimize multi_table_accuracy ui * fix v-show error * fix code smells Co-authored-by: sunchaohe <sunzhaohe@linklogis.com> * [Feature][DataQuality] Add data quality module (#4830) * add data quality module * add license * add package configuration in dist pom * fix license and jar import bug * replace apache/skywalking-eyes@9bd5feb SHA * refacotr jbdc-connector and writer * modify parameter name in HiveConnector * fix checkstyle error * fix checkstyle error in dolphinschesuler-dist * fix checkstyle error in dolphinschesuler-dist * fix checkstyle error in dolphinschesuler-dist * fix duplicate code bug * fix code style bug * fix code smells * update * close e2e test * fix spark rw postgresql bug * change mysql driver scope * add more sql driver * remove hive-jdbc in pom.xml * change the pom.xml Co-authored-by: sunchaohe <sunzhaohe@linklogis.com> * [Feature][DataQuality] Add data quality task backend (#4883) * add dq relevant enums and parameter * replace apache/skywalking-eyes@9bd5feb SHA Co-authored-by: sunchaohe <sunzhaohe@linklogis.com> * refactor data_quality_module * add header license * data quality module refactor * fix unit test error * fix checkstyle error * fix unit test error * fix checkstyle error * fix unit test error * fix code smell * fix check style * fix unit test error * task statistics value add unique code * fix unit test error * fix checkstyle error * fix checkstyle * fix security hotspot * fix unit test error * fix security hotspot * fix check * add data quality task error handling * fix unit test error * add unit test * add unit test * optimize data quality result alert * fix unit test * fix sql script error * fix bug * update sql script * fix checkstyle * add license * fix checkstyle * fix checkstyle * fix unit test * add jacoco dependencies * fix unit test * fix unit test * add jacoco dependencies * add unit test * add unit test * add license * fix checkstyle * fix pom * fix checkstyle * fix checkstyle * merge dev * fix ui error * fix pom error * fix pom error * fix test error * fix test error * mssql-jdbc exclude azure-keyvault * fix test error * merge dev and add unit test * add notes * rollback the CollectionUtils * fix * update sql * fix * fix * fix query rule page error * change dq.jar path * fix sql error * fix ui error * fix(dq): jar path&task enum description * add notes on DataQualityApplication * fix dq result jump error * fix(ui): page condition * feat(ui): add show error output path * change version * remove all chinese word in sql * merge Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>	3 years ago
Wenjun Ruan	a1cc2fd11a	[task] Python task support PYTHON_HOME (#8196 )	3 years ago
Kerwin	103ccb01da	change version to 2.0.4-SNAPSHOT (#8188 )	3 years ago
Shiwen Cheng	0057cf3fe7	[Improvement][task-plugin] Remove unreasonable log (#8115 )	3 years ago
seagle	f1a0215978	[bug] Fix datax task type error parse clickhouse reader sql (#8091 )	3 years ago
wind	e4cdd82b7c	[Improvement][TaskLog] Unified task log (#7831 ) * task logger * update * log format * remove master config task-logger * udpate log level Co-authored-by: caishunfeng <534328519@qq.com>	3 years ago
Kerwin	5172862a0a	Add seatunnel task plugin (#7131 )	3 years ago
J·Y	a4168ae24e	[Improvement] The spark version is associated with the spark command (#7544 ) * [Improvement] The spark version is associated with the spark command	3 years ago
xiangzihao	c58dbefaa5	[Fix-7459][Sqoop] running error in parallel (#7752 ) * fix bug_7459 * fix bug_7459 * fix bug_7459 * fix bug_7459 * fix bug_7459 Co-authored-by: SbloodyS <sbloodys@qq.com>	3 years ago
wangxj3	cc77963522	[FIX-#6727][worker-server] Fix procedure params bug (#7680 ) * fix Procedure param error * code style Co-authored-by: wangxj <wangxj31>	3 years ago
J·Y	6ae46f2c1b	[Improvement] Merge redundant codes (#7550 )	3 years ago
Zonglei Dong	7e0010de35	[Refactor] Rename class file, unified word spelling. (#7261 )	3 years ago
kezhenxu94	67cc260d52	Split the components into individual package, refactor configurations to use Spring properties (#7160 ) * Split the components into individual package A follow-up PR will be made to build dedicated Docker images for each component, so that every component Docker image has minimal jars, which is easy to maintain and good for security fixes. * Split the components into individual package A follow-up PR will be made to build dedicated Docker images for each component, so that every component Docker image has minimal jars, which is easy to maintain and good for security fixes. * Split the components into individual package A follow-up PR will be made to build dedicated Docker images for each component, so that every component Docker image has minimal jars, which is easy to maintain and good for security fixes.	3 years ago
Zonglei Dong	0b9792db76	[Refactor]Rename class file, DatasourceUtil to DataSourceUtils. (#7181 ) * Rename class file, DatasourceUtil to DataSourceUtils.	3 years ago
张俊杰	463fc76bb7	[Fix-6809][dolphinscheduler-task-sql] Fix sql task can't send alert (#7089 )	3 years ago
kezhenxu94	2ed3c0c3a7	Recover unit tests that have not been run for some time (#7054 )	3 years ago
kezhenxu94	4114cb07f6	Add actuator and enable metrics, add Grafana configs (#6840 )	3 years ago
kezhenxu94	11844415b0	Refactor datasource and task plugins and simplify its usage (#6767 )	3 years ago
mask	0aea5ae11a	[feature-6071][plugin] add datasource plugin. (#6071 ) (#6361 ) * [feature-6071][plugin] add datasource plugin. (#6071) * remove the dependency of datasource in common and task-plugin module. * the new datasource-api module provides data source connection acquisition. * add sqlserver data source plug-in. * add ds license * fix check style * update spi pom * frx conflict resolution * fix check style * fix datasource ut * fix code check style * fix code check style * fix datasource ut * add pg hive mysql oracle clickhouse db2 datasource * fix datasource package * update datasource pom * update datasource pom * update datasource pom * update datasource pom * update pom * update pom dep * update pom dep * fix license * fix ut * fix ut * fix ut * fix ut * fix ut * fix ut * fix ut * fix ut * fix ut * fix ut * fix ut * fix ut * fix ut * add oracle package * modify comments * conflict resolution * rename log comment * rename log comment	3 years ago
mask	9b1984c244	[fix-6570][task] add sql task udf resource load (#6570 ) (#6571 ) * [fix-6570][task] add sql task udf resource load (#6570) -add sql task udf resource load -Fix UdfFuncRequest serialization * fix checkstyle * add defaultFS in sqlTaskContext * fix checkstyle	3 years ago
Kirs	09657d8576	[#6337 ]When the SQL result reaches the limit value, increase the log result prompt (#6528 )	3 years ago
Hua Jiang	8200bc1452	[Fix-6529] Improve the python task to solve the issue about getting the environment variables. (#6531 ) * modify config * improve python task * improve python task	3 years ago
Kirs	bddf3e6f2c	[FIX-#6514]Task Class Load Err (#6513 ) * [FIX-#6412]Task Class Load Err * Task Class Load Err	3 years ago
Hua Jiang	a80fca70f7	[Fix-6497][Server] Fix the missing code about the feature of supporting multiple environments. (#6509 )	3 years ago
百岁	d284e44d52	Rename tis to pigeon from backend layer (#6437 ) * add mock getTaskLogName() for UT test pass * make task-tis rename to task-pigeon * rename module name 'tis' to pigeon * rename module name 'tis' to pigeon,make sanity check pass * remove execute console.log()	3 years ago
Kirs	1843e986f5	[BUG][Task-API]load codec class fail (#6411 )	3 years ago
Kirs	67508d8d1b	update project version to 2.0.0-SNAPSHOT (#6383 )	3 years ago
mask	a62a7a7216	[Fix][Task] sql limit param no default value (#6337 ) (#6387 ) * add default query sql limit	3 years ago
百岁	545173faa3	add mock getTaskLogName() for UT test pass (#6369 )	3 years ago

1 2 3 4

178 Commits (cc82206c33cc60cd71ebfe7b327e8cc4110bd15d)