DolphinScheduler

Commit Graph

Author	SHA1	Message	Date
Simon	71619ed29b	Update pom.xml	5 years ago
Simon	b5a296f4cd	Update pom.xml	5 years ago
Simon	094d09ba74	Update pom.xml	5 years ago
eights	822addac30	add sqoop task UT to pom	5 years ago
simon824	fad09075f9	remove fastjson maven dependence	5 years ago
simon824	6e907f150c	1. specified jackson Date format. 2. remove getMapper method,replace with toList() 3. rewrite toList method,support constructCollectionType. 4. parseObject(resp, Map.class) replace to toMap() method 5. rename JacksonSerialize to JsonSerializer rewrite createArrayNode,createObjectNode method will Affect too many lines ，i will fix it in the next pr.	5 years ago
bao liang	422fe5136d	[license] add license and notice (#2916 ) * fix bug: send task result to master until success. * fix bug: send task result to master until success. * add sleep * set null if send result failed * set null if send result failed * change max length of username to 39 * test for gc * change version to 1.3.0 * check and add some license files * check and add some license files * revert Co-authored-by: baoliang <baoliang@analysys.com.cn>	5 years ago
CalvinKirs	b21259d600	add MicroBenchMark model (#2851 ) * add MicroBenchMark model * add log * fix propertyName * small change * add ut * format * add @Test * fix maven compile error * remove enum set method * exclusion microbench ut * small change * Use assembly instead of shade Co-authored-by: dailidong <dailidong66@gmail.com>	5 years ago
simon	4a10157506	ut fix	5 years ago
lgcareer	ef4c1e7930	add source license full text into source license (#2840 ) * add source license full text into source license * exclude ScriptRunner.java on rat check	5 years ago
qiaozhanwei	0130da2bdf	Fix [Bug] process definition json worker group convert #2794 (#2795 ) * add LoggerServerTest UT * add LoggerServerTest UT * add LoggerServerTest UT add RemoveTaskLogRequestCommandTest UT add RemoveTaskLogResponseCommandTest * master select worker filter high load worker #2704 * master select worker filter high load worker #2704 * master select worker filter high load worker #2704 * master select worker filter high load worker #2704 * master select worker filter high load worker #2704 * master select worker filter high load worker #2704 * add not worker log and remove worker invalid property * process definition json worker group convert #2794 * process definition json worker group convert #2794 * process definition json worker group convert #2794 * process definition json worker group convert #2794 * process definition json worker group convert #2794 * process definition json worker group convert #2794 Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>	5 years ago
gabry.wu	c28a697ee8	add RetryerUtils to retry async & sync function gracefully (#2697 ) * add RetryerUtils tool class for async & sync retry invoking * add RetryerUtils tool class for async & sync retry invoking * add guava-retrying-2.0.0.jar to known-dependencies.txt * fix javadoc * fix code smell Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>	5 years ago
qiaozhanwei	f4583d78e9	add UT (#2717 ) * add LoggerServerTest UT * add LoggerServerTest UT * add LoggerServerTest UT add RemoveTaskLogRequestCommandTest UT add RemoveTaskLogResponseCommandTest Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>	5 years ago
qiaozhanwei	689c216050	add LoggerServerTest UT (#2713 ) * add LoggerServerTest UT * add LoggerServerTest UT Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>	5 years ago
xiaozhaoying	de3a81cab5	support ali nail(ding talk) , issue:2458 (#2594 ) * Create TextMessage.java * Create DingTalkUtils.java * Update alert.properties * Update Constants.java * Update Constants.java * Create DingTalkMsgFormatter.java * Update alert.properties * Create DingTalkMsgFormatterTest.java * Create DingTalkUtilsTest.java * Update DingTalkUtils.java * Create DingTalkManager.java * Update EmailAlertPlugin.java * feature: test send msg with proxy * fix:delete comment and add exception info * Update DingTalkUtilsTest.java * Update DingTalkUtils.java * Update DingTalkManager.java * Update EmailAlertPlugin.java * Update DingTalkUtilsTest.java * Update DingTalkMsgFormatter.java * Update DingTalkUtils.java * Update DingTalkMsgFormatterTest.java * Update DingTalkUtilsTest.java * Update TextMessage.java * Update DingTalkManager.java * Delete TextMessage.java * fix: add test case * fix:add http client test case * fix:delete ignore test case * Update pom.xml Co-authored-by: dailidong <dailidong66@gmail.com>	5 years ago
kezhenxu94	6e08b29254	Add a script to check the license (#2552 ) * Add a script to check the license ### Motivation Check licenses in the distribution package ### Modification 1. Add a script to check licenses, and list known licenses. 1. Remove unused dep license, lombok. 1. Remove unnecessary ignored patterns in .gitignore. 1. Add missing config items to apache-rat plugin, which just checked a small part of files before. 1. Add check to GitHub Actions process ### Result 1. Newly-added dependencies should be checked and confirmed. 1. Closes #1578 * Separate rat and dependency check * Tee dependencies to stdout * Add more debug messages * Check light-weight task first	5 years ago
hgaol	906ffa09a9	resolve code checks	5 years ago
hgaol	85f3ac10a0	feat: add plugin management for alert service	5 years ago
qiaozhanwei	a12103b08c	no valid worker group，master can kill task directly (#2541 ) * dispatch task fail will set task status failed * 1，no worker condition , master will while ture wait for worker startup 2，worker response task status sync wait for result * 1，no worker condition , master will while ture wait for worker startup 2，worker response task status sync wait for result * 1，no worker condition , master will while ture wait for worker startup 2，worker response task status sync wait for result * 1，no worker condition , master will while ture wait for worker startup 2，worker response task status sync wait for result * 1，no worker condition , master will while ture wait for worker startup 2，worker response task status sync wait for result * 1，no worker condition , master will while ture wait for worker startup 2，worker response task status sync wait for result * 1，no worker condition , master will while ture wait for worker startup 2，worker response task status sync wait for result * 1，no worker condition , master will while ture wait for worker startup 2，worker response task status sync wait for result * 1，no worker condition , master will while ture wait for worker startup 2，worker response task status sync wait for result * 1，no worker condition , master will while ture wait for worker startup 2，worker response task status sync wait for result * 1，no worker condition , master will while ture wait for worker startup 2，worker response task status sync wait for result * 1，task status statistics and process status statistics bug fix (#2357) 2，worker group bug fix * 1，task status statistics and process status statistics bug fix (#2357) 2，worker group bug fix * 1，task status statistics and process status statistics bug fix (#2357) 2，worker group bug fix * 1，task status statistics and process status statistics bug fix (#2357) 2，worker group bug fix * send mail error, #2466 bug fix * send mail error, #2466 bug fix * send mail error, #2466 bug fix * send mail error, #2466 bug fix * #2486 bug fix * host and workergroup compatible * EnterpriseWeChatUtils modify * EnterpriseWeChatUtils modify * EnterpriseWeChatUtils modify * #2499 bug fix * add comment * revert comment * revert comment * #2499 buf fix * #2499 bug fix * #2499 bug fix * #2499 bug fix * #2499 bug fix * #2499 bug fix * #2499 bug fix * no valid worker group，master can kill task directly * no valid worker group，master can kill task directly * no valid worker group，master can kill task directly * no valid worker group，master can kill task directly * no valid worker group，master can kill task directly * no valid worker group，master can kill task directly * no valid worker group，master can kill task directly * no valid worker group，master can kill task directly * no valid worker group，master can kill task directly Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>	5 years ago
zixi0825	64d335d9f2	copy process feature #2412 (#2471 ) * Update pom.xml (#2467) * add copy process * add copy process 2 * add copy process 3 * add copy process 4 * add copy process 5 Co-authored-by: dailidong <dailidong66@gmail.com> Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>	5 years ago
lgcareer	44770a8767	[fix #2442 ] It will give a error tip that resource not exist exwhen update resources. (#2483 ) * fix #2442 and remove unavailable code * revert verifyResourceName method * Add ServiceException * add ServiceExceptionTest * update ServiceExceptionTest * add ServiceExceptionTest in pom	5 years ago
dailidong	eec92ed1fc	simplify and optimize config (#2469 ) * [Refactor worker] simplify and optimize config (#2386) * simplify config * simplify config * simplify and optimize config * Update HadoopUtils.java optimize HadoopUtils * Update HadoopUtilsTest.java * Update HadoopUtilsTest.java * Update HadoopUtilsTest.java * Update HttpUtils.java * Update pom.xml * Update HadoopUtilsTest.java * Update HadoopUtilsTest.java * Update HadoopUtilsTest.java * Update HadoopUtilsTest.java * Update HadoopUtilsTest.java * Update HadoopUtilsTest.java	5 years ago
dailidong	0b405b6fc8	Update pom.xml	5 years ago
dailidong	dc36c77ba7	Update pom.xml (#2467 )	5 years ago
dailidong	0f54149ae3	exclude HadoopUtils Test (#2465 ) * Update pom.xml * Update pom.xml	5 years ago
bao liang	4946e88872	move dependent and conditions to master server (#2423 ) * move condition and dependent to master * move conditions to master * move conditions and dependent task to master * move conditions to master * update test * add log * add test for dependent task * add test for dependent task * update * update * refactor complexity code * refactor complexity code * add conditions task test * add conditions task test * update * update host to host:port * update logback.xml Co-authored-by: baoliang <baoliang@analysys.com.cn>	5 years ago
zixi0825	ca66cfec17	solve code spell problem (#2419 ) * solve code spell problem * remove controller test * add controller test * sovle spell problem and sovle conflict * sovle spell problem and sovle conflict * sovle spell problem and add test Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>	5 years ago
Han Gao	3c9ba0a3f3	api server exception management and code optimization (#397 ) (#2397 ) * api server exception management and code optimization (#397) * add unit test for exception handler. * fix some style and naming issues Co-authored-by: dailidong <dailidong66@gmail.com>	5 years ago
qiaozhanwei	ff535e766d	update UT in pom.xml	5 years ago
qiaozhanwei	9d06fe7755	update UT in pom.xml	5 years ago
qiaozhanwei	b2c92deace	update UT in pom.xml	5 years ago
qiaozhanwei	66d587c40d	update UT in pom.xml	5 years ago
qiaozhanwei	b1a773f341	update UT in pom.xml	5 years ago
qiaozhanwei	9de9f18c08	update UT in pom.xml	5 years ago
qiaozhanwei	4160c21d8f	update UT in pom.xml	5 years ago
qiaozhanwei	b581f02a21	add dao other mapper add in pom.xml	5 years ago
qiaozhanwei	f02e7e8b37	pom modify	5 years ago
qiaozhanwei	f65ab56d2d	UT modify	5 years ago
liwenhe1993	9dfc93e0a2	Exclude .mvn/jvm.config and .mvn/wrapper/maven-wrapper.properties (#2387 ) * Remove .helmignore file * Exclude .mvn/jvm.config and .mvn/wrapper/maven-wrapper.properties When running rat, exclude .mvn/jvm.config and .mvn/wrapper/maven-wrapper.properties	5 years ago
qiaozhanwei	af2b3187b9	update pom.xml	5 years ago
qiaozhanwei	433b41dd7f	refactor-worker merge to dev bug fix	5 years ago
qiaozhanwei	94713cef79	refactor-worker merge to dev bug fix	5 years ago
zixi0825	69e000b542	#2282 fix workflow dependent bug (#2329 ) * fix workflow dependent bug * fix workflow dependent bug 2 * fix workflow dependent bug 2 Co-authored-by: sunchaohe <sunzhaohe@linklogis.com> Co-authored-by: dailidong <dailidong66@gmail.com>	5 years ago
liwenhe1993	526e5c91f1	Connection mode of adding Sid to Oracle (#2254 ) * Connection mode of adding Sid to Oracle * Remove code * Add asf * Add unit test * Add unit test * Add unit test * solve the conflict	5 years ago
qiaozhanwei	68cb81fdf5	1，add UT in pom 2，refactor TaskUpdateQueue (#2326 ) * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * add license * modify javadoc error * TaskExecutionContext create modify * buildAckCommand taskInstanceId not set modify * java doc error modify * add comment * ExecutorManager interface add generic type * add TaskInstanceCacheManager receive Worker report result * TaskInstance setExecutePath * add TaskInstanceCacheManager to receive Worker Task result report * TaskInstanceCacheManager add remove method * add license * add dispatcht task method * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * 1，worker TaskPros use TaskExecutionContext replase 2，Master kill Task , KillTaskProcessor modify * worker remove db * ShellTask modify * master persistence processId and appIds * master persistence processId and appIds * master add kill task logic * master add kill task logic * master add kill task logic * javadoc error modify * remove chinese log * executeDirectly method add Override * remote module modify * TaskKillResponseProcessor command type modify * create buildKillCommand * host add host:port format * host add host:port format * TaskAckProcessor modify * TaskAckProcessor modify * task prioriry refator * remove ITaskQueue * task prioriry refator * remove ITaskQueue * TaskPriority refactor * remove logs * WorkerServer refactor * MasterSchedulerService modify * WorkerConfig listen port modify * modify master and worker listen port * cancelTaskInstance set TaskExecutionContext host,logPath,executePath * cancelTaskInstance set TaskExecutionContext host,logPath,executePath * Encapsulate the parameters required by sqltask * 1，Encapsulate the parameters required by sqltask 2，SQLTask optimization * AbstractTask modify * ProcedureTask optimization * MasterSchedulerService modify * TaskUpdateQueueConsumer modify * test * DataxTask process run debug * DataxTask process run debug * add protobuf dependency，MR、Spark task etc need this * TaskUpdateQueueConsumer modify * TaskExecutionContextBuilder set TaskInstance workgroup * WorkerGroupService queryAllGroup modify query available work group * 1，get workergroup from zk modify 2，SpringConnectionFactory repeat load modify * master and worker register ip use OSUtils.getHost() * ProcessInstance host set ip:port format * worker fault tolerance modify * Constants and .env modify * master fault tolerant bug modify * UT add pom.xml * timing online modify * when taskResponse is faster than taskAck to db，task state will error add async queue and new a thread reslove this problem * TaskExecutionContext set host * 1，TaskManager refactor 2， api start load server dolphinschedule-daemon.sh modify * 1，TaskManager refactor 2， api start load server dolphinschedule-daemon.sh modify * add UT in pom.xml * revert dolphinscheduler-daemon.sh * ZookeeperRegister use common.properties zookeeperRoot path * api start exclude org.apache.dolphinscheduler.server.* * ZookeeperRegister use common.properties zookeeperRoot path * 1，api start load server filter 2，SHELL task exitStatusCode modify * java doc error modify * java doc error modify * remove todo * add UT in pom * 1，add UT in pom 2，refactor TaskUpdateQueue Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>	5 years ago
t1mon	efde8fd4e7	Refactor BaseDataSource and unified handling of JDBC resource releases. (#2184 ) * Refactor BaseDataSource.isConnectable(). * Add BaseDataSource test. * Unified handling of JDBC resource releases. * Fix sonar check error. * Remove useless logger. * Refactor BaseDataSource.getJdbcUrl(). * Modify ConnectionUtils.releaseResource(). * Add javadoc * Remove useless method. * Add BaseDataSourceTest UT	5 years ago
qiaozhanwei	f112415b13	TaskManager refactor (#2302 ) * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * add license * modify javadoc error * TaskExecutionContext create modify * buildAckCommand taskInstanceId not set modify * java doc error modify * add comment * ExecutorManager interface add generic type * add TaskInstanceCacheManager receive Worker report result * TaskInstance setExecutePath * add TaskInstanceCacheManager to receive Worker Task result report * TaskInstanceCacheManager add remove method * add license * add dispatcht task method * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * 1，worker TaskPros use TaskExecutionContext replase 2，Master kill Task , KillTaskProcessor modify * worker remove db * ShellTask modify * master persistence processId and appIds * master persistence processId and appIds * master add kill task logic * master add kill task logic * master add kill task logic * javadoc error modify * remove chinese log * executeDirectly method add Override * remote module modify * TaskKillResponseProcessor command type modify * create buildKillCommand * host add host:port format * host add host:port format * TaskAckProcessor modify * TaskAckProcessor modify * task prioriry refator * remove ITaskQueue * task prioriry refator * remove ITaskQueue * TaskPriority refactor * remove logs * WorkerServer refactor * MasterSchedulerService modify * WorkerConfig listen port modify * modify master and worker listen port * cancelTaskInstance set TaskExecutionContext host,logPath,executePath * cancelTaskInstance set TaskExecutionContext host,logPath,executePath * Encapsulate the parameters required by sqltask * 1，Encapsulate the parameters required by sqltask 2，SQLTask optimization * AbstractTask modify * ProcedureTask optimization * MasterSchedulerService modify * TaskUpdateQueueConsumer modify * test * DataxTask process run debug * DataxTask process run debug * add protobuf dependency，MR、Spark task etc need this * TaskUpdateQueueConsumer modify * TaskExecutionContextBuilder set TaskInstance workgroup * WorkerGroupService queryAllGroup modify query available work group * 1，get workergroup from zk modify 2，SpringConnectionFactory repeat load modify * master and worker register ip use OSUtils.getHost() * ProcessInstance host set ip:port format * worker fault tolerance modify * Constants and .env modify * master fault tolerant bug modify * UT add pom.xml * timing online modify * when taskResponse is faster than taskAck to db，task state will error add async queue and new a thread reslove this problem * TaskExecutionContext set host * 1，TaskManager refactor 2， api start load server dolphinschedule-daemon.sh modify * 1，TaskManager refactor 2， api start load server dolphinschedule-daemon.sh modify * add UT in pom.xml * revert dolphinscheduler-daemon.sh Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>	5 years ago
qiaozhanwei	052f9d10bf	UT Coverage rate test (#2276 ) * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * add license * modify javadoc error * TaskExecutionContext create modify * buildAckCommand taskInstanceId not set modify * java doc error modify * add comment * ExecutorManager interface add generic type * add TaskInstanceCacheManager receive Worker report result * TaskInstance setExecutePath * add TaskInstanceCacheManager to receive Worker Task result report * TaskInstanceCacheManager add remove method * add license * add dispatcht task method * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * 1，worker TaskPros use TaskExecutionContext replase 2，Master kill Task , KillTaskProcessor modify * worker remove db * ShellTask modify * master persistence processId and appIds * master persistence processId and appIds * master add kill task logic * master add kill task logic * master add kill task logic * javadoc error modify * remove chinese log * executeDirectly method add Override * remote module modify * TaskKillResponseProcessor command type modify * create buildKillCommand * host add host:port format * host add host:port format * TaskAckProcessor modify * TaskAckProcessor modify * task prioriry refator * remove ITaskQueue * task prioriry refator * remove ITaskQueue * TaskPriority refactor * remove logs * WorkerServer refactor * MasterSchedulerService modify * WorkerConfig listen port modify * modify master and worker listen port * cancelTaskInstance set TaskExecutionContext host,logPath,executePath * cancelTaskInstance set TaskExecutionContext host,logPath,executePath * Encapsulate the parameters required by sqltask * 1，Encapsulate the parameters required by sqltask 2，SQLTask optimization * AbstractTask modify * ProcedureTask optimization * MasterSchedulerService modify * TaskUpdateQueueConsumer modify * test * DataxTask process run debug * DataxTask process run debug * add protobuf dependency，MR、Spark task etc need this * TaskUpdateQueueConsumer modify * TaskExecutionContextBuilder set TaskInstance workgroup * WorkerGroupService queryAllGroup modify query available work group * 1，get workergroup from zk modify 2，SpringConnectionFactory repeat load modify * master and worker register ip use OSUtils.getHost() * ProcessInstance host set ip:port format * worker fault tolerance modify * Constants and .env modify * master fault tolerant bug modify * UT add pom.xml Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>	5 years ago
khadgarmage	cddf803ba0	license and checkoutv2 fix (#2261 )	5 years ago
jin	74bd33301f	feat:optimizing code (#2245 ) * feat:optimizing code * add licence and format code	5 years ago

... 4 5 6 7 8

399 Commits (04c68e31e8382b5beb944a86581546ffc4737c23)