DolphinScheduler

Commit Graph

Author	SHA1	Message	Date
dailidong	b5ff3ad39e	Delete ReleaseNotes.md (#2338 )	5 years ago
dailidong	07cc10c34b	[Refactor worker] correct config error and remove unuse properties that may affect performance (#2337 ) * update logback * update log * refactor worker registry (#2107) * Refactor worker (#2115) * refactor worker registry * refactor master server * Modify workgroupid parameter name (#2105) * Delete worker group management page * Modify workgroupid parameter name * Refactor worker (#2121) * refactor worker registry * refactor master server * refactor MasterSchedulerService * cancelTaskInstance set TaskExecutionContext host,logPath,executePath (#2126) * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * add license * modify javadoc error * TaskExecutionContext create modify * buildAckCommand taskInstanceId not set modify * java doc error modify * add comment * ExecutorManager interface add generic type * add TaskInstanceCacheManager receive Worker report result * TaskInstance setExecutePath * add TaskInstanceCacheManager to receive Worker Task result report * TaskInstanceCacheManager add remove method * add license * add dispatcht task method * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * 1，worker TaskPros use TaskExecutionContext replase 2，Master kill Task , KillTaskProcessor modify * worker remove db * ShellTask modify * master persistence processId and appIds * master persistence processId and appIds * master add kill task logic * master add kill task logic * master add kill task logic * javadoc error modify * remove chinese log * executeDirectly method add Override * remote module modify * TaskKillResponseProcessor command type modify * create buildKillCommand * host add host:port format * host add host:port format * TaskAckProcessor modify * TaskAckProcessor modify * task prioriry refator * remove ITaskQueue * task prioriry refator * remove ITaskQueue * TaskPriority refactor * remove logs * WorkerServer refactor * MasterSchedulerService modify * WorkerConfig listen port modify * modify master and worker listen port * cancelTaskInstance set TaskExecutionContext host,logPath,executePath * cancelTaskInstance set TaskExecutionContext host,logPath,executePath Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn> * not exist in openjdk,just delete * add master and worker properties * add master and worker properties * add master and worker properties * fix cpu 100% bug * simplify master、 worker、alert、dao properties * add master and worker properties * add master and worker properties * add master and worker properties * Optimize code , reduce cost time from 60ms to 0.5ms * merge code form remote * remove zookeeper as task queue relevant config * correct config error and remove unuse properties that may affect performance * correct config error and remove unuse properties that may affect performance Co-authored-by: Tboy <guo.jiwei@immomo.com> Co-authored-by: break60 <790061044@qq.com> Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com> Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>	5 years ago
break60	520c40b626	Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev	5 years ago
Rubik-W	4440b35c4e	fix #2334 (#2335 ) Co-authored-by: dailidong <dailidong66@gmail.com>	5 years ago
songgg	f7e89e4561	some annotation optimizations (#2333 ) Co-authored-by: songqh <songquanhe@foxmail.com>	5 years ago
qiaozhanwei	c2cf8fb7b1	test	5 years ago
dailidong	195dfb508c	remove zookeeper as task queue relevant config (#2330 ) * update logback * update log * refactor worker registry (#2107) * Refactor worker (#2115) * refactor worker registry * refactor master server * Modify workgroupid parameter name (#2105) * Delete worker group management page * Modify workgroupid parameter name * Refactor worker (#2121) * refactor worker registry * refactor master server * refactor MasterSchedulerService * cancelTaskInstance set TaskExecutionContext host,logPath,executePath (#2126) * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * add license * modify javadoc error * TaskExecutionContext create modify * buildAckCommand taskInstanceId not set modify * java doc error modify * add comment * ExecutorManager interface add generic type * add TaskInstanceCacheManager receive Worker report result * TaskInstance setExecutePath * add TaskInstanceCacheManager to receive Worker Task result report * TaskInstanceCacheManager add remove method * add license * add dispatcht task method * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * 1，worker TaskPros use TaskExecutionContext replase 2，Master kill Task , KillTaskProcessor modify * worker remove db * ShellTask modify * master persistence processId and appIds * master persistence processId and appIds * master add kill task logic * master add kill task logic * master add kill task logic * javadoc error modify * remove chinese log * executeDirectly method add Override * remote module modify * TaskKillResponseProcessor command type modify * create buildKillCommand * host add host:port format * host add host:port format * TaskAckProcessor modify * TaskAckProcessor modify * task prioriry refator * remove ITaskQueue * task prioriry refator * remove ITaskQueue * TaskPriority refactor * remove logs * WorkerServer refactor * MasterSchedulerService modify * WorkerConfig listen port modify * modify master and worker listen port * cancelTaskInstance set TaskExecutionContext host,logPath,executePath * cancelTaskInstance set TaskExecutionContext host,logPath,executePath Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn> * not exist in openjdk,just delete * add master and worker properties * add master and worker properties * add master and worker properties * fix cpu 100% bug * simplify master、 worker、alert、dao properties * add master and worker properties * add master and worker properties * add master and worker properties * Optimize code , reduce cost time from 60ms to 0.5ms * merge code form remote * remove zookeeper as task queue relevant config Co-authored-by: Tboy <guo.jiwei@immomo.com> Co-authored-by: break60 <790061044@qq.com> Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com> Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>	5 years ago
Tboy	3bafa97f96	Refactor worker (#2331 ) * let quartz use the same datasource * move master/worker config from dao.properties to each config add master/worker registry test * move mybatis config from application.properties to SpringConnectionFactory * move mybatis-plus config from application.properties to SpringConnectionFactory * refactor TaskCallbackService * add ZookeeperNodeManagerTest * add NettyExecutorManagerTest * refactor TaskKillProcessor * add RandomSelectorTest, RoundRobinSelectorTest, TaskCallbackServiceTest * add RoundRobinHostManagerTest, ExecutorDispatcherTest * refactor task response service * add TaskResponseServiceTest * modify close method for MasterSchedulerService	5 years ago
break60	3edca1465d	Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev	5 years ago
liwenhe1993	526e5c91f1	Connection mode of adding Sid to Oracle (#2254 ) * Connection mode of adding Sid to Oracle * Remove code * Add asf * Add unit test * Add unit test * Add unit test * solve the conflict	5 years ago
qiaozhanwei	68cb81fdf5	1，add UT in pom 2，refactor TaskUpdateQueue (#2326 ) * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * add license * modify javadoc error * TaskExecutionContext create modify * buildAckCommand taskInstanceId not set modify * java doc error modify * add comment * ExecutorManager interface add generic type * add TaskInstanceCacheManager receive Worker report result * TaskInstance setExecutePath * add TaskInstanceCacheManager to receive Worker Task result report * TaskInstanceCacheManager add remove method * add license * add dispatcht task method * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * 1，worker TaskPros use TaskExecutionContext replase 2，Master kill Task , KillTaskProcessor modify * worker remove db * ShellTask modify * master persistence processId and appIds * master persistence processId and appIds * master add kill task logic * master add kill task logic * master add kill task logic * javadoc error modify * remove chinese log * executeDirectly method add Override * remote module modify * TaskKillResponseProcessor command type modify * create buildKillCommand * host add host:port format * host add host:port format * TaskAckProcessor modify * TaskAckProcessor modify * task prioriry refator * remove ITaskQueue * task prioriry refator * remove ITaskQueue * TaskPriority refactor * remove logs * WorkerServer refactor * MasterSchedulerService modify * WorkerConfig listen port modify * modify master and worker listen port * cancelTaskInstance set TaskExecutionContext host,logPath,executePath * cancelTaskInstance set TaskExecutionContext host,logPath,executePath * Encapsulate the parameters required by sqltask * 1，Encapsulate the parameters required by sqltask 2，SQLTask optimization * AbstractTask modify * ProcedureTask optimization * MasterSchedulerService modify * TaskUpdateQueueConsumer modify * test * DataxTask process run debug * DataxTask process run debug * add protobuf dependency，MR、Spark task etc need this * TaskUpdateQueueConsumer modify * TaskExecutionContextBuilder set TaskInstance workgroup * WorkerGroupService queryAllGroup modify query available work group * 1，get workergroup from zk modify 2，SpringConnectionFactory repeat load modify * master and worker register ip use OSUtils.getHost() * ProcessInstance host set ip:port format * worker fault tolerance modify * Constants and .env modify * master fault tolerant bug modify * UT add pom.xml * timing online modify * when taskResponse is faster than taskAck to db，task state will error add async queue and new a thread reslove this problem * TaskExecutionContext set host * 1，TaskManager refactor 2， api start load server dolphinschedule-daemon.sh modify * 1，TaskManager refactor 2， api start load server dolphinschedule-daemon.sh modify * add UT in pom.xml * revert dolphinscheduler-daemon.sh * ZookeeperRegister use common.properties zookeeperRoot path * api start exclude org.apache.dolphinscheduler.server.* * ZookeeperRegister use common.properties zookeeperRoot path * 1，api start load server filter 2，SHELL task exitStatusCode modify * java doc error modify * java doc error modify * remove todo * add UT in pom * 1，add UT in pom 2，refactor TaskUpdateQueue Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>	5 years ago
liwenhe1993	6cf7e6c115	Remove .helmignore file (#2328 )	5 years ago
liwenhe1993	f6ca5480dd	Support kubernetes deployment (#2153 ) * Support kubernetes deployment * Support kubernetes deployment	5 years ago
t1mon	d4735334a1	Fix SqlTask kerberos load scope unreasonable. #2178 (#2321 ) * Optimize PropertyUtils instantiation. * Fix info error. * Fix SqlTask kerberos load scope unreasonable. #2178	5 years ago
break60	570f8f523d	Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev	5 years ago
Rubik-W	a851168a35	fix：Optimize content returned by interface (#2294 )	5 years ago
break60	2aa4ac8884	Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev	5 years ago
lgcareer	38c965ddc9	new feature for #404 add resource tree function (#2323 ) * add create resource directory * add create resource directory * update the resource test * add upgrade sql in version 1.2.2 * Adding request parameter id to update queryResourceListPaging * set isDirectory value is false default * add full name to update updateResource * remove request parameter isDirectory to update createResource method * update queryResourceListPaging with change get to post * update updateResource method with remove fullName * File management list modification (#1976) * add resource component * add resource tree visitor * return json string * update queryResourceList * upload file need fullName * add method rootNode * Shell task resources and authorization resources (#1989) * File management list modification * Shell task resources and authorization resources * download resource when execute task * download resource when execute task * update authorization type * download resource when execute task * Spark task resource changes (#1990) * File management list modification * Shell task resources and authorization resources * Spark task resource changes * download resource when execute task * update udf function service * add resource type in ResourceComponent * UDF resource tree and change DAG style (#2019) * File management list modification * Shell task resources and authorization resources * Spark task resource changes * UDF resource tree and change DAG style * add deleteIds method in ResourceMapper and ResourceMapperTest * Add comments on class and method * add queryResourceByName method in controller * update verify-name with change name to full name * update queryResource with add parameter pid * update queryResource with add parameter pid * add resource ids in process definition and delete resource need judge whether it is used by any process definition * Breadcrumb development (#2033) * File management list modification * Shell task resources and authorization resources * Spark task resource changes * UDF resource tree and change DAG style * Breadcrumb development * Breadcrumb development * Resource tree bug fix (#2040) * File management list modification * Shell task resources and authorization resources * Spark task resource changes * UDF resource tree and change DAG style * Breadcrumb development * Breadcrumb development * Resource tree bug fix * update resource service test * Fix github action rerun failed * add status of PARENT_RESOURCE_NOT_EXIST * Fix github action rerun failed (#2067) * update resource service test * Fix github action rerun failed * add status of PARENT_RESOURCE_NOT_EXIST * Change crumb position * Change crumb position (#2068) * build resource process definition map * UDF changed to multiple choice * UDF changed to multiple choice (#2077) * Change crumb position * UDF changed to multiple choice * build resource process definition map (#2076) * update resource service test * Fix github action rerun failed * add status of PARENT_RESOURCE_NOT_EXIST * build resource process definition map * update resource name also need update all the children full name * need add queryResource * update resource name also need update all the children full name (#2096) * update resource service test * Fix github action rerun failed * add status of PARENT_RESOURCE_NOT_EXIST * build resource process definition map * update resource name also need update all the children full name * need add queryResource * Limit customization file content to no more than 3000 lines * Limit customization file content to no more than 3000 lines(#2128) * Limit customization file content to no more than 3000 lines(#2128) (#2140) * Change crumb position * UDF changed to multiple choice * Limit customization file content to no more than 3000 lines * Limit customization file content to no more than 3000 lines(#2128) * add queryResourceJarList * add queryResourceJarList * add queryResourceJarList * add queryResourceJarList (#2192) * update resource service test * Fix github action rerun failed * add status of PARENT_RESOURCE_NOT_EXIST * build resource process definition map * update resource name also need update all the children full name * need add queryResource * add queryResourceJarList * add queryResourceJarList * add queryResourceJarList * Modify the main jar package * Modify the main jar package (#2200) * Change crumb position * UDF changed to multiple choice * Limit customization file content to no more than 3000 lines * Limit customization file content to no more than 3000 lines(#2128) * Modify the main jar package * add resource filter in order to get filtered resource * add comments of resource filter * update list children by resource * choose main jar with resource tree (#2220) * update resource service test * Fix github action rerun failed * add status of PARENT_RESOURCE_NOT_EXIST * build resource process definition map * update resource name also need update all the children full name * need add queryResource * add queryResourceJarList * add queryResourceJarList * add queryResourceJarList * add resource filter in order to get filtered resource * add comments of resource filter * update list children by resource * Return null if query resource list is empty * update queryResource method change parameter pid to id * getResouDelete checksum and modify parameter namerceId * revert .env * remove parameter pid * Delete request interface * go back to the last page * jar interface call * Fix issue #2234 and #2228 * change resource name with full name * Fix issue #2234 and #2228 (#2246) * update resource service test * Fix github action rerun failed * add status of PARENT_RESOURCE_NOT_EXIST * build resource process definition map * update resource name also need update all the children full name * need add queryResource * add queryResourceJarList * add queryResourceJarList * add queryResourceJarList * add resource filter in order to get filtered resource * add comments of resource filter * update list children by resource * Return null if query resource list is empty * update queryResource method change parameter pid to id * revert .env * remove parameter pid * Fix issue #2234 and #2228 * change resource name with full name * Fix list query value error * remove unauth-file with authorize-resource-tree * Repair data cannot be echoed * Repair data cannot be echoed * execute mr and spark task need query resource name before * Authorized resource interface replacement * Authorized resource interface replacement * Filter UDF resources * Change parameters * need query all authorized directory children when create task * Change normalize.scss import method and animation.scss license modification * Delete file list update processing * It's fixed that resource not deleted in hdfs when delete it. * add tooltips * add tooltips (#2310) * Echo workflow name and modify udf management name * [new feature]add resource tree function * revert front code in order to be same as dev branch * revert front code in order to be same as dev branch * revert common.properties and application.properties * add super method * update flink parameter test * update flink parameter and unit test * update resource service test * If resource list is empty,need init it * update flink parameter test Co-authored-by: break60 <790061044@qq.com> Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com> Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>	5 years ago
gabry.wu	48d7612cd5	Adapting partial code(file name start with S #2 ) to the sonar cloud rule (#2270 )	5 years ago
liwenhe1993	c706b21c62	taskProps.getScheduleTime() may be null, but there is no check if it … (#2256 ) * taskProps.getScheduleTime() may be null, but there is no check if it is null or not * Add unit test	5 years ago
break60	7780153bdd	Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev	5 years ago
break60	c52c92e015	Resource tree code merge (#2322 ) * Change DOM label * Change name to lowercase * Limit customization file content to no more than 3000 lines * dd branch flow node verification * datax * datax add custom * Change normalize.scss import method and animation.scss license modification * Resource tree code merge	5 years ago
Tboy	23ba035182	add TaskResponseServiceTest (#2325 ) * let quartz use the same datasource * move master/worker config from dao.properties to each config add master/worker registry test * move mybatis config from application.properties to SpringConnectionFactory * move mybatis-plus config from application.properties to SpringConnectionFactory * refactor TaskCallbackService * add ZookeeperNodeManagerTest * add NettyExecutorManagerTest * refactor TaskKillProcessor * add RandomSelectorTest, RoundRobinSelectorTest, TaskCallbackServiceTest * add RoundRobinHostManagerTest, ExecutorDispatcherTest * refactor task response service * add TaskResponseServiceTest	5 years ago
break60	64b28a5171	Resource tree code merge	5 years ago
break60	fd250ff3d3	Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev	5 years ago
lgcareer	af8efe5184	add worker group and remove worker group id in dll (#2313 )	5 years ago
dailidong	7e9f45e514	[Refactor worker] Optimize code , reduce cost time from 60ms to 0.5ms (#2318 ) * update logback * update log * refactor worker registry (#2107) * Refactor worker (#2115) * refactor worker registry * refactor master server * Modify workgroupid parameter name (#2105) * Delete worker group management page * Modify workgroupid parameter name * Refactor worker (#2121) * refactor worker registry * refactor master server * refactor MasterSchedulerService * cancelTaskInstance set TaskExecutionContext host,logPath,executePath (#2126) * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * add license * modify javadoc error * TaskExecutionContext create modify * buildAckCommand taskInstanceId not set modify * java doc error modify * add comment * ExecutorManager interface add generic type * add TaskInstanceCacheManager receive Worker report result * TaskInstance setExecutePath * add TaskInstanceCacheManager to receive Worker Task result report * TaskInstanceCacheManager add remove method * add license * add dispatcht task method * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * 1，worker TaskPros use TaskExecutionContext replase 2，Master kill Task , KillTaskProcessor modify * worker remove db * ShellTask modify * master persistence processId and appIds * master persistence processId and appIds * master add kill task logic * master add kill task logic * master add kill task logic * javadoc error modify * remove chinese log * executeDirectly method add Override * remote module modify * TaskKillResponseProcessor command type modify * create buildKillCommand * host add host:port format * host add host:port format * TaskAckProcessor modify * TaskAckProcessor modify * task prioriry refator * remove ITaskQueue * task prioriry refator * remove ITaskQueue * TaskPriority refactor * remove logs * WorkerServer refactor * MasterSchedulerService modify * WorkerConfig listen port modify * modify master and worker listen port * cancelTaskInstance set TaskExecutionContext host,logPath,executePath * cancelTaskInstance set TaskExecutionContext host,logPath,executePath Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn> * not exist in openjdk,just delete * add master and worker properties * add master and worker properties * add master and worker properties * fix cpu 100% bug * simplify master、 worker、alert、dao properties * add master and worker properties * add master and worker properties * add master and worker properties * Optimize code , reduce cost time from 60ms to 0.5ms * merge code form remote Co-authored-by: Tboy <guo.jiwei@immomo.com> Co-authored-by: break60 <790061044@qq.com> Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com> Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>	5 years ago
Tboy	5730bfe2be	Refactor worker (#2319 ) * let quartz use the same datasource * move master/worker config from dao.properties to each config add master/worker registry test * move mybatis config from application.properties to SpringConnectionFactory * move mybatis-plus config from application.properties to SpringConnectionFactory * refactor TaskCallbackService * add ZookeeperNodeManagerTest * add NettyExecutorManagerTest * refactor TaskKillProcessor * add RandomSelectorTest, RoundRobinSelectorTest, TaskCallbackServiceTest * add RoundRobinHostManagerTest, ExecutorDispatcherTest * refactor task response service	5 years ago
gabry.wu	eef8bf4e6d	Adapting partial code(file name start with P #2 ) to the sonar cloud rule (#2260 )	5 years ago
孙朝和	d8cba5ff1b	resolve shell task build command nullpointexception (#2267 ) Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>	5 years ago
t1mon	efde8fd4e7	Refactor BaseDataSource and unified handling of JDBC resource releases. (#2184 ) * Refactor BaseDataSource.isConnectable(). * Add BaseDataSource test. * Unified handling of JDBC resource releases. * Fix sonar check error. * Remove useless logger. * Refactor BaseDataSource.getJdbcUrl(). * Modify ConnectionUtils.releaseResource(). * Add javadoc * Remove useless method. * Add BaseDataSourceTest UT	5 years ago
gabry.wu	d83a94a305	Adapting partial code(file name start with T) to the sonar cloud rule (#2271 )	5 years ago
tswstarplanet	74525c28b5	Decouple DruidDataSource in ConnectionFactory and cache DataSource instance (#2232 ) * Decouple DruidDataSource in ConnectionFactory and cache DataSource instance * init dataSource first * fix code smell * add unit test * add unit test * add unit test * add unit test * fix unit test * delete useless logger	5 years ago
gabry.wu	42f9385ee4	Adapting partial code(file name start with W) to the sonar cloud rule (#2287 )	5 years ago
WeiYe	a1b8386c8c	Update test code logger class name (#2303 )	5 years ago
Tboy	e720ca5463	Refactor worker (#2312 ) * let quartz use the same datasource * move master/worker config from dao.properties to each config add master/worker registry test * move mybatis config from application.properties to SpringConnectionFactory * move mybatis-plus config from application.properties to SpringConnectionFactory * refactor TaskCallbackService * add ZookeeperNodeManagerTest * add NettyExecutorManagerTest * refactor TaskKillProcessor * add RandomSelectorTest, RoundRobinSelectorTest, TaskCallbackServiceTest * add RoundRobinHostManagerTest, ExecutorDispatcherTest	5 years ago
xingchun-chen	5edc2af311	e2e add project and workflow case (#2309 ) * add LoginTest license * Delete useless packages * e2e add project and workflow case * e2e add project and workflow case * e2e add project and workflow case * e2e add project and workflow case * e2e add project and workflow case * e2e add project and workflow case * e2e add project and workflow case * e2e add project and workflow case * e2e add project and workflow case * e2e add project and workflow case * e2e add project and workflow case * e2e add project and workflow case Co-authored-by: chenxingchun <438044805@qq.com>	5 years ago
qiaozhanwei	03e29b9311	1，ZookeeperRegister use common.properties zookeeperRoot path 2，api start exclude org.apache.dolphinscheduler.server.* (#2307 ) * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * add license * modify javadoc error * TaskExecutionContext create modify * buildAckCommand taskInstanceId not set modify * java doc error modify * add comment * ExecutorManager interface add generic type * add TaskInstanceCacheManager receive Worker report result * TaskInstance setExecutePath * add TaskInstanceCacheManager to receive Worker Task result report * TaskInstanceCacheManager add remove method * add license * add dispatcht task method * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * 1，worker TaskPros use TaskExecutionContext replase 2，Master kill Task , KillTaskProcessor modify * worker remove db * ShellTask modify * master persistence processId and appIds * master persistence processId and appIds * master add kill task logic * master add kill task logic * master add kill task logic * javadoc error modify * remove chinese log * executeDirectly method add Override * remote module modify * TaskKillResponseProcessor command type modify * create buildKillCommand * host add host:port format * host add host:port format * TaskAckProcessor modify * TaskAckProcessor modify * task prioriry refator * remove ITaskQueue * task prioriry refator * remove ITaskQueue * TaskPriority refactor * remove logs * WorkerServer refactor * MasterSchedulerService modify * WorkerConfig listen port modify * modify master and worker listen port * cancelTaskInstance set TaskExecutionContext host,logPath,executePath * cancelTaskInstance set TaskExecutionContext host,logPath,executePath * Encapsulate the parameters required by sqltask * 1，Encapsulate the parameters required by sqltask 2，SQLTask optimization * AbstractTask modify * ProcedureTask optimization * MasterSchedulerService modify * TaskUpdateQueueConsumer modify * test * DataxTask process run debug * DataxTask process run debug * add protobuf dependency，MR、Spark task etc need this * TaskUpdateQueueConsumer modify * TaskExecutionContextBuilder set TaskInstance workgroup * WorkerGroupService queryAllGroup modify query available work group * 1，get workergroup from zk modify 2，SpringConnectionFactory repeat load modify * master and worker register ip use OSUtils.getHost() * ProcessInstance host set ip:port format * worker fault tolerance modify * Constants and .env modify * master fault tolerant bug modify * UT add pom.xml * timing online modify * when taskResponse is faster than taskAck to db，task state will error add async queue and new a thread reslove this problem * TaskExecutionContext set host * 1，TaskManager refactor 2， api start load server dolphinschedule-daemon.sh modify * 1，TaskManager refactor 2， api start load server dolphinschedule-daemon.sh modify * add UT in pom.xml * revert dolphinscheduler-daemon.sh * ZookeeperRegister use common.properties zookeeperRoot path * api start exclude org.apache.dolphinscheduler.server.* * ZookeeperRegister use common.properties zookeeperRoot path * 1，api start load server filter 2，SHELL task exitStatusCode modify * java doc error modify * java doc error modify * remove todo Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>	5 years ago
dailidong	ace907e7dd	[refactor-worker] simplify master、 worker、alert、dao properties (#2304 ) * update logback * update log * refactor worker registry (#2107) * Refactor worker (#2115) * refactor worker registry * refactor master server * Modify workgroupid parameter name (#2105) * Delete worker group management page * Modify workgroupid parameter name * Refactor worker (#2121) * refactor worker registry * refactor master server * refactor MasterSchedulerService * cancelTaskInstance set TaskExecutionContext host,logPath,executePath (#2126) * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * add license * modify javadoc error * TaskExecutionContext create modify * buildAckCommand taskInstanceId not set modify * java doc error modify * add comment * ExecutorManager interface add generic type * add TaskInstanceCacheManager receive Worker report result * TaskInstance setExecutePath * add TaskInstanceCacheManager to receive Worker Task result report * TaskInstanceCacheManager add remove method * add license * add dispatcht task method * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * 1，worker TaskPros use TaskExecutionContext replase 2，Master kill Task , KillTaskProcessor modify * worker remove db * ShellTask modify * master persistence processId and appIds * master persistence processId and appIds * master add kill task logic * master add kill task logic * master add kill task logic * javadoc error modify * remove chinese log * executeDirectly method add Override * remote module modify * TaskKillResponseProcessor command type modify * create buildKillCommand * host add host:port format * host add host:port format * TaskAckProcessor modify * TaskAckProcessor modify * task prioriry refator * remove ITaskQueue * task prioriry refator * remove ITaskQueue * TaskPriority refactor * remove logs * WorkerServer refactor * MasterSchedulerService modify * WorkerConfig listen port modify * modify master and worker listen port * cancelTaskInstance set TaskExecutionContext host,logPath,executePath * cancelTaskInstance set TaskExecutionContext host,logPath,executePath Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn> * not exist in openjdk,just delete * add master and worker properties * add master and worker properties * add master and worker properties * fix cpu 100% bug * simplify master、 worker、alert、dao properties * add master and worker properties * add master and worker properties * add master and worker properties Co-authored-by: Tboy <guo.jiwei@immomo.com> Co-authored-by: break60 <790061044@qq.com> Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com> Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>	5 years ago
qiaozhanwei	f112415b13	TaskManager refactor (#2302 ) * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * add license * modify javadoc error * TaskExecutionContext create modify * buildAckCommand taskInstanceId not set modify * java doc error modify * add comment * ExecutorManager interface add generic type * add TaskInstanceCacheManager receive Worker report result * TaskInstance setExecutePath * add TaskInstanceCacheManager to receive Worker Task result report * TaskInstanceCacheManager add remove method * add license * add dispatcht task method * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * 1，worker TaskPros use TaskExecutionContext replase 2，Master kill Task , KillTaskProcessor modify * worker remove db * ShellTask modify * master persistence processId and appIds * master persistence processId and appIds * master add kill task logic * master add kill task logic * master add kill task logic * javadoc error modify * remove chinese log * executeDirectly method add Override * remote module modify * TaskKillResponseProcessor command type modify * create buildKillCommand * host add host:port format * host add host:port format * TaskAckProcessor modify * TaskAckProcessor modify * task prioriry refator * remove ITaskQueue * task prioriry refator * remove ITaskQueue * TaskPriority refactor * remove logs * WorkerServer refactor * MasterSchedulerService modify * WorkerConfig listen port modify * modify master and worker listen port * cancelTaskInstance set TaskExecutionContext host,logPath,executePath * cancelTaskInstance set TaskExecutionContext host,logPath,executePath * Encapsulate the parameters required by sqltask * 1，Encapsulate the parameters required by sqltask 2，SQLTask optimization * AbstractTask modify * ProcedureTask optimization * MasterSchedulerService modify * TaskUpdateQueueConsumer modify * test * DataxTask process run debug * DataxTask process run debug * add protobuf dependency，MR、Spark task etc need this * TaskUpdateQueueConsumer modify * TaskExecutionContextBuilder set TaskInstance workgroup * WorkerGroupService queryAllGroup modify query available work group * 1，get workergroup from zk modify 2，SpringConnectionFactory repeat load modify * master and worker register ip use OSUtils.getHost() * ProcessInstance host set ip:port format * worker fault tolerance modify * Constants and .env modify * master fault tolerant bug modify * UT add pom.xml * timing online modify * when taskResponse is faster than taskAck to db，task state will error add async queue and new a thread reslove this problem * TaskExecutionContext set host * 1，TaskManager refactor 2， api start load server dolphinschedule-daemon.sh modify * 1，TaskManager refactor 2， api start load server dolphinschedule-daemon.sh modify * add UT in pom.xml * revert dolphinscheduler-daemon.sh Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>	5 years ago
break60	0d70e065ba	Change normalize.scss import method and animation.scss license modification (#2295 ) * Change DOM label * Change name to lowercase * Limit customization file content to no more than 3000 lines * dd branch flow node verification * datax * datax add custom * Change normalize.scss import method and animation.scss license modification	5 years ago
qiaozhanwei	62f7d21bda	add async queue and new a thread reslove taskResponse is faster than taskAck to db (#2297 ) * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * add license * modify javadoc error * TaskExecutionContext create modify * buildAckCommand taskInstanceId not set modify * java doc error modify * add comment * ExecutorManager interface add generic type * add TaskInstanceCacheManager receive Worker report result * TaskInstance setExecutePath * add TaskInstanceCacheManager to receive Worker Task result report * TaskInstanceCacheManager add remove method * add license * add dispatcht task method * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * 1，worker TaskPros use TaskExecutionContext replase 2，Master kill Task , KillTaskProcessor modify * worker remove db * ShellTask modify * master persistence processId and appIds * master persistence processId and appIds * master add kill task logic * master add kill task logic * master add kill task logic * javadoc error modify * remove chinese log * executeDirectly method add Override * remote module modify * TaskKillResponseProcessor command type modify * create buildKillCommand * host add host:port format * host add host:port format * TaskAckProcessor modify * TaskAckProcessor modify * task prioriry refator * remove ITaskQueue * task prioriry refator * remove ITaskQueue * TaskPriority refactor * remove logs * WorkerServer refactor * MasterSchedulerService modify * WorkerConfig listen port modify * modify master and worker listen port * cancelTaskInstance set TaskExecutionContext host,logPath,executePath * cancelTaskInstance set TaskExecutionContext host,logPath,executePath * Encapsulate the parameters required by sqltask * 1，Encapsulate the parameters required by sqltask 2，SQLTask optimization * AbstractTask modify * ProcedureTask optimization * MasterSchedulerService modify * TaskUpdateQueueConsumer modify * test * DataxTask process run debug * DataxTask process run debug * add protobuf dependency，MR、Spark task etc need this * TaskUpdateQueueConsumer modify * TaskExecutionContextBuilder set TaskInstance workgroup * WorkerGroupService queryAllGroup modify query available work group * 1，get workergroup from zk modify 2，SpringConnectionFactory repeat load modify * master and worker register ip use OSUtils.getHost() * ProcessInstance host set ip:port format * worker fault tolerance modify * Constants and .env modify * master fault tolerant bug modify * UT add pom.xml * timing online modify * when taskResponse is faster than taskAck to db，task state will error add async queue and new a thread reslove this problem * TaskExecutionContext set host Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>	5 years ago
dailidong	7da35238e1	[Refactor worker] fix cpu pressure may be too high bug (#2290 ) * update logback * update log * refactor worker registry (#2107) * Refactor worker (#2115) * refactor worker registry * refactor master server * Modify workgroupid parameter name (#2105) * Delete worker group management page * Modify workgroupid parameter name * Refactor worker (#2121) * refactor worker registry * refactor master server * refactor MasterSchedulerService * cancelTaskInstance set TaskExecutionContext host,logPath,executePath (#2126) * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * add license * modify javadoc error * TaskExecutionContext create modify * buildAckCommand taskInstanceId not set modify * java doc error modify * add comment * ExecutorManager interface add generic type * add TaskInstanceCacheManager receive Worker report result * TaskInstance setExecutePath * add TaskInstanceCacheManager to receive Worker Task result report * TaskInstanceCacheManager add remove method * add license * add dispatcht task method * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * 1，worker TaskPros use TaskExecutionContext replase 2，Master kill Task , KillTaskProcessor modify * worker remove db * ShellTask modify * master persistence processId and appIds * master persistence processId and appIds * master add kill task logic * master add kill task logic * master add kill task logic * javadoc error modify * remove chinese log * executeDirectly method add Override * remote module modify * TaskKillResponseProcessor command type modify * create buildKillCommand * host add host:port format * host add host:port format * TaskAckProcessor modify * TaskAckProcessor modify * task prioriry refator * remove ITaskQueue * task prioriry refator * remove ITaskQueue * TaskPriority refactor * remove logs * WorkerServer refactor * MasterSchedulerService modify * WorkerConfig listen port modify * modify master and worker listen port * cancelTaskInstance set TaskExecutionContext host,logPath,executePath * cancelTaskInstance set TaskExecutionContext host,logPath,executePath Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn> * not exist in openjdk,just delete * add master and worker properties * add master and worker properties * add master and worker properties * fix cpu 100% bug Co-authored-by: Tboy <guo.jiwei@immomo.com> Co-authored-by: break60 <790061044@qq.com> Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com> Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>	5 years ago
qiaozhanwei	11e3bff63e	timing online modify (#2296 ) * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * add license * modify javadoc error * TaskExecutionContext create modify * buildAckCommand taskInstanceId not set modify * java doc error modify * add comment * ExecutorManager interface add generic type * add TaskInstanceCacheManager receive Worker report result * TaskInstance setExecutePath * add TaskInstanceCacheManager to receive Worker Task result report * TaskInstanceCacheManager add remove method * add license * add dispatcht task method * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * 1，worker TaskPros use TaskExecutionContext replase 2，Master kill Task , KillTaskProcessor modify * worker remove db * ShellTask modify * master persistence processId and appIds * master persistence processId and appIds * master add kill task logic * master add kill task logic * master add kill task logic * javadoc error modify * remove chinese log * executeDirectly method add Override * remote module modify * TaskKillResponseProcessor command type modify * create buildKillCommand * host add host:port format * host add host:port format * TaskAckProcessor modify * TaskAckProcessor modify * task prioriry refator * remove ITaskQueue * task prioriry refator * remove ITaskQueue * TaskPriority refactor * remove logs * WorkerServer refactor * MasterSchedulerService modify * WorkerConfig listen port modify * modify master and worker listen port * cancelTaskInstance set TaskExecutionContext host,logPath,executePath * cancelTaskInstance set TaskExecutionContext host,logPath,executePath * Encapsulate the parameters required by sqltask * 1，Encapsulate the parameters required by sqltask 2，SQLTask optimization * AbstractTask modify * ProcedureTask optimization * MasterSchedulerService modify * TaskUpdateQueueConsumer modify * test * DataxTask process run debug * DataxTask process run debug * add protobuf dependency，MR、Spark task etc need this * TaskUpdateQueueConsumer modify * TaskExecutionContextBuilder set TaskInstance workgroup * WorkerGroupService queryAllGroup modify query available work group * 1，get workergroup from zk modify 2，SpringConnectionFactory repeat load modify * master and worker register ip use OSUtils.getHost() * ProcessInstance host set ip:port format * worker fault tolerance modify * Constants and .env modify * master fault tolerant bug modify * UT add pom.xml * timing online modify Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>	5 years ago
Tboy	1e5a1140ab	add RandomSelectorTest, RoundRobinSelectorTest, TaskCallbackServiceTest (#2291 ) * let quartz use the same datasource * move master/worker config from dao.properties to each config add master/worker registry test * move mybatis config from application.properties to SpringConnectionFactory * move mybatis-plus config from application.properties to SpringConnectionFactory * refactor TaskCallbackService * add ZookeeperNodeManagerTest * add NettyExecutorManagerTest * refactor TaskKillProcessor * add RandomSelectorTest, RoundRobinSelectorTest, TaskCallbackServiceTest	5 years ago
break60	5c2e94e99e	Change normalize.scss import method and animation.scss license modification	5 years ago
break60	1964d47bdb	Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev	5 years ago
Tboy	92fd6479a9	Refactor worker (#2279 ) * let quartz use the same datasource * move master/worker config from dao.properties to each config add master/worker registry test * move mybatis config from application.properties to SpringConnectionFactory * move mybatis-plus config from application.properties to SpringConnectionFactory * refactor TaskCallbackService * add ZookeeperNodeManagerTest * add NettyExecutorManagerTest * refactor TaskKillProcessor	5 years ago
qiaozhanwei	052f9d10bf	UT Coverage rate test (#2276 ) * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * 1, master persistent task 2. extract master and worker communication model * add license * modify javadoc error * TaskExecutionContext create modify * buildAckCommand taskInstanceId not set modify * java doc error modify * add comment * ExecutorManager interface add generic type * add TaskInstanceCacheManager receive Worker report result * TaskInstance setExecutePath * add TaskInstanceCacheManager to receive Worker Task result report * TaskInstanceCacheManager add remove method * add license * add dispatcht task method * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * AbstractCommandExecutor remove db access * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * taskInstanceCache is null ,need load from db * 1，worker TaskPros use TaskExecutionContext replase 2，Master kill Task , KillTaskProcessor modify * worker remove db * ShellTask modify * master persistence processId and appIds * master persistence processId and appIds * master add kill task logic * master add kill task logic * master add kill task logic * javadoc error modify * remove chinese log * executeDirectly method add Override * remote module modify * TaskKillResponseProcessor command type modify * create buildKillCommand * host add host:port format * host add host:port format * TaskAckProcessor modify * TaskAckProcessor modify * task prioriry refator * remove ITaskQueue * task prioriry refator * remove ITaskQueue * TaskPriority refactor * remove logs * WorkerServer refactor * MasterSchedulerService modify * WorkerConfig listen port modify * modify master and worker listen port * cancelTaskInstance set TaskExecutionContext host,logPath,executePath * cancelTaskInstance set TaskExecutionContext host,logPath,executePath * Encapsulate the parameters required by sqltask * 1，Encapsulate the parameters required by sqltask 2，SQLTask optimization * AbstractTask modify * ProcedureTask optimization * MasterSchedulerService modify * TaskUpdateQueueConsumer modify * test * DataxTask process run debug * DataxTask process run debug * add protobuf dependency，MR、Spark task etc need this * TaskUpdateQueueConsumer modify * TaskExecutionContextBuilder set TaskInstance workgroup * WorkerGroupService queryAllGroup modify query available work group * 1，get workergroup from zk modify 2，SpringConnectionFactory repeat load modify * master and worker register ip use OSUtils.getHost() * ProcessInstance host set ip:port format * worker fault tolerance modify * Constants and .env modify * master fault tolerant bug modify * UT add pom.xml Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>	5 years ago
khadgarmage	53b62d403f	add ds logs and server status check with e2e test (#2277 )	5 years ago

1 2 3 4 5 ...

2460 Commits (28291219366c00e1fb9bb1eebb586bf2a91c8400) All Branches Search

2460 Commits (28291219366c00e1fb9bb1eebb586bf2a91c8400)

All Branches