break60
520c40b626
Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev
4 years ago
Rubik-W
4440b35c4e
fix #2334 ( #2335 )
...
Co-authored-by: dailidong <dailidong66@gmail.com>
4 years ago
songgg
f7e89e4561
some annotation optimizations ( #2333 )
...
Co-authored-by: songqh <songquanhe@foxmail.com>
4 years ago
qiaozhanwei
c2cf8fb7b1
test
4 years ago
dailidong
195dfb508c
remove zookeeper as task queue relevant config ( #2330 )
...
* update logback
* update log
* refactor worker registry (#2107 )
* Refactor worker (#2115 )
* refactor worker registry
* refactor master server
* Modify workgroupid parameter name (#2105 )
* Delete worker group management page
* Modify workgroupid parameter name
* Refactor worker (#2121 )
* refactor worker registry
* refactor master server
* refactor MasterSchedulerService
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath (#2126 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
* not exist in openjdk,just delete
* add master and worker properties
* add master and worker properties
* add master and worker properties
* fix cpu 100% bug
* simplify master、 worker、alert、dao properties
* add master and worker properties
* add master and worker properties
* add master and worker properties
* Optimize code , reduce cost time from 60ms to 0.5ms
* merge code form remote
* remove zookeeper as task queue relevant config
Co-authored-by: Tboy <guo.jiwei@immomo.com>
Co-authored-by: break60 <790061044@qq.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
4 years ago
Tboy
3bafa97f96
Refactor worker ( #2331 )
...
* let quartz use the same datasource
* move master/worker config from dao.properties to each config
add master/worker registry test
* move mybatis config from application.properties to SpringConnectionFactory
* move mybatis-plus config from application.properties to SpringConnectionFactory
* refactor TaskCallbackService
* add ZookeeperNodeManagerTest
* add NettyExecutorManagerTest
* refactor TaskKillProcessor
* add RandomSelectorTest, RoundRobinSelectorTest, TaskCallbackServiceTest
* add RoundRobinHostManagerTest, ExecutorDispatcherTest
* refactor task response service
* add TaskResponseServiceTest
* modify close method for MasterSchedulerService
4 years ago
break60
3edca1465d
Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev
4 years ago
oceanos
48c2cb7934
Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev
4 years ago
liwenhe1993
526e5c91f1
Connection mode of adding Sid to Oracle ( #2254 )
...
* Connection mode of adding Sid to Oracle
* Remove code
* Add asf
* Add unit test
* Add unit test
* Add unit test
* solve the conflict
4 years ago
qiaozhanwei
68cb81fdf5
1,add UT in pom 2,refactor TaskUpdateQueue ( #2326 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* Encapsulate the parameters required by sqltask
* 1,Encapsulate the parameters required by sqltask
2,SQLTask optimization
* AbstractTask modify
* ProcedureTask optimization
* MasterSchedulerService modify
* TaskUpdateQueueConsumer modify
* test
* DataxTask process run debug
* DataxTask process run debug
* add protobuf dependency,MR、Spark task etc need this
* TaskUpdateQueueConsumer modify
* TaskExecutionContextBuilder set TaskInstance workgroup
* WorkerGroupService queryAllGroup modify
query available work group
* 1,get workergroup from zk modify
2,SpringConnectionFactory repeat load modify
* master and worker register ip use OSUtils.getHost()
* ProcessInstance host set ip:port format
* worker fault tolerance modify
* Constants and .env modify
* master fault tolerant bug modify
* UT add pom.xml
* timing online modify
* when taskResponse is faster than taskAck to db,task state will error
add async queue and new a thread reslove this problem
* TaskExecutionContext set host
* 1,TaskManager refactor
2, api start load server dolphinschedule-daemon.sh modify
* 1,TaskManager refactor
2, api start load server dolphinschedule-daemon.sh modify
* add UT in pom.xml
* revert dolphinscheduler-daemon.sh
* ZookeeperRegister use common.properties zookeeperRoot path
* api start exclude org.apache.dolphinscheduler.server.*
* ZookeeperRegister use common.properties zookeeperRoot path
* 1,api start load server filter
2,SHELL task exitStatusCode modify
* java doc error modify
* java doc error modify
* remove todo
* add UT in pom
* 1,add UT in pom
2,refactor TaskUpdateQueue
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
4 years ago
liwenhe1993
6cf7e6c115
Remove .helmignore file ( #2328 )
4 years ago
liwenhe1993
da459e4fc0
Remove .helmignore file
4 years ago
liwenhe1993
f6ca5480dd
Support kubernetes deployment ( #2153 )
...
* Support kubernetes deployment
* Support kubernetes deployment
4 years ago
t1mon
d4735334a1
Fix SqlTask kerberos load scope unreasonable. #2178 ( #2321 )
...
* Optimize PropertyUtils instantiation.
* Fix info error.
* Fix SqlTask kerberos load scope unreasonable. #2178
4 years ago
break60
570f8f523d
Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev
4 years ago
Rubik-W
a851168a35
fix:Optimize content returned by interface ( #2294 )
4 years ago
break60
2aa4ac8884
Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev
4 years ago
lgcareer
38c965ddc9
new feature for #404 add resource tree function ( #2323 )
...
* add create resource directory
* add create resource directory
* update the resource test
* add upgrade sql in version 1.2.2
* Adding request parameter id to update queryResourceListPaging
* set isDirectory value is false default
* add full name to update updateResource
* remove request parameter isDirectory to update createResource method
* update queryResourceListPaging with change get to post
* update updateResource method with remove fullName
* File management list modification (#1976 )
* add resource component
* add resource tree visitor
* return json string
* update queryResourceList
* upload file need fullName
* add method rootNode
* Shell task resources and authorization resources (#1989 )
* File management list modification
* Shell task resources and authorization resources
* download resource when execute task
* download resource when execute task
* update authorization type
* download resource when execute task
* Spark task resource changes (#1990 )
* File management list modification
* Shell task resources and authorization resources
* Spark task resource changes
* download resource when execute task
* update udf function service
* add resource type in ResourceComponent
* UDF resource tree and change DAG style (#2019 )
* File management list modification
* Shell task resources and authorization resources
* Spark task resource changes
* UDF resource tree and change DAG style
* add deleteIds method in ResourceMapper and ResourceMapperTest
* Add comments on class and method
* add queryResourceByName method in controller
* update verify-name with change name to full name
* update queryResource with add parameter pid
* update queryResource with add parameter pid
* add resource ids in process definition and delete resource need judge whether it is used by any process definition
* Breadcrumb development (#2033 )
* File management list modification
* Shell task resources and authorization resources
* Spark task resource changes
* UDF resource tree and change DAG style
* Breadcrumb development
* Breadcrumb development
* Resource tree bug fix (#2040 )
* File management list modification
* Shell task resources and authorization resources
* Spark task resource changes
* UDF resource tree and change DAG style
* Breadcrumb development
* Breadcrumb development
* Resource tree bug fix
* update resource service test
* Fix github action rerun failed
* add status of PARENT_RESOURCE_NOT_EXIST
* Fix github action rerun failed (#2067 )
* update resource service test
* Fix github action rerun failed
* add status of PARENT_RESOURCE_NOT_EXIST
* Change crumb position
* Change crumb position (#2068 )
* build resource process definition map
* UDF changed to multiple choice
* UDF changed to multiple choice (#2077 )
* Change crumb position
* UDF changed to multiple choice
* build resource process definition map (#2076 )
* update resource service test
* Fix github action rerun failed
* add status of PARENT_RESOURCE_NOT_EXIST
* build resource process definition map
* update resource name also need update all the children full name
* need add queryResource
* update resource name also need update all the children full name (#2096 )
* update resource service test
* Fix github action rerun failed
* add status of PARENT_RESOURCE_NOT_EXIST
* build resource process definition map
* update resource name also need update all the children full name
* need add queryResource
* Limit customization file content to no more than 3000 lines
* Limit customization file content to no more than 3000 lines(#2128 )
* Limit customization file content to no more than 3000 lines(#2128 ) (#2140 )
* Change crumb position
* UDF changed to multiple choice
* Limit customization file content to no more than 3000 lines
* Limit customization file content to no more than 3000 lines(#2128 )
* add queryResourceJarList
* add queryResourceJarList
* add queryResourceJarList
* add queryResourceJarList (#2192 )
* update resource service test
* Fix github action rerun failed
* add status of PARENT_RESOURCE_NOT_EXIST
* build resource process definition map
* update resource name also need update all the children full name
* need add queryResource
* add queryResourceJarList
* add queryResourceJarList
* add queryResourceJarList
* Modify the main jar package
* Modify the main jar package (#2200 )
* Change crumb position
* UDF changed to multiple choice
* Limit customization file content to no more than 3000 lines
* Limit customization file content to no more than 3000 lines(#2128 )
* Modify the main jar package
* add resource filter in order to get filtered resource
* add comments of resource filter
* update list children by resource
* choose main jar with resource tree (#2220 )
* update resource service test
* Fix github action rerun failed
* add status of PARENT_RESOURCE_NOT_EXIST
* build resource process definition map
* update resource name also need update all the children full name
* need add queryResource
* add queryResourceJarList
* add queryResourceJarList
* add queryResourceJarList
* add resource filter in order to get filtered resource
* add comments of resource filter
* update list children by resource
* Return null if query resource list is empty
* update queryResource method change parameter pid to id
* getResouDelete checksum and modify parameter namerceId
* revert .env
* remove parameter pid
* Delete request interface
* go back to the last page
* jar interface call
* Fix issue #2234 and #2228
* change resource name with full name
* Fix issue #2234 and #2228 (#2246 )
* update resource service test
* Fix github action rerun failed
* add status of PARENT_RESOURCE_NOT_EXIST
* build resource process definition map
* update resource name also need update all the children full name
* need add queryResource
* add queryResourceJarList
* add queryResourceJarList
* add queryResourceJarList
* add resource filter in order to get filtered resource
* add comments of resource filter
* update list children by resource
* Return null if query resource list is empty
* update queryResource method change parameter pid to id
* revert .env
* remove parameter pid
* Fix issue #2234 and #2228
* change resource name with full name
* Fix list query value error
* remove unauth-file with authorize-resource-tree
* Repair data cannot be echoed
* Repair data cannot be echoed
* execute mr and spark task need query resource name before
* Authorized resource interface replacement
* Authorized resource interface replacement
* Filter UDF resources
* Change parameters
* need query all authorized directory children when create task
* Change normalize.scss import method and animation.scss license modification
* Delete file list update processing
* It's fixed that resource not deleted in hdfs when delete it.
* add tooltips
* add tooltips (#2310 )
* Echo workflow name and modify udf management name
* [new feature]add resource tree function
* revert front code in order to be same as dev branch
* revert front code in order to be same as dev branch
* revert common.properties and application.properties
* add super method
* update flink parameter test
* update flink parameter and unit test
* update resource service test
* If resource list is empty,need init it
* update flink parameter test
Co-authored-by: break60 <790061044@qq.com>
Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
4 years ago
gabry.wu
48d7612cd5
Adapting partial code(file name start with S #2 ) to the sonar cloud rule ( #2270 )
4 years ago
liwenhe1993
c706b21c62
taskProps.getScheduleTime() may be null, but there is no check if it … ( #2256 )
...
* taskProps.getScheduleTime() may be null, but there is no check if it is null or not
* Add unit test
4 years ago
break60
7780153bdd
Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev
4 years ago
break60
c52c92e015
Resource tree code merge ( #2322 )
...
* Change DOM label
* Change name to lowercase
* Limit customization file content to no more than 3000 lines
* dd branch flow node verification
* datax
* datax add custom
* Change normalize.scss import method and animation.scss license modification
* Resource tree code merge
4 years ago
Tboy
23ba035182
add TaskResponseServiceTest ( #2325 )
...
* let quartz use the same datasource
* move master/worker config from dao.properties to each config
add master/worker registry test
* move mybatis config from application.properties to SpringConnectionFactory
* move mybatis-plus config from application.properties to SpringConnectionFactory
* refactor TaskCallbackService
* add ZookeeperNodeManagerTest
* add NettyExecutorManagerTest
* refactor TaskKillProcessor
* add RandomSelectorTest, RoundRobinSelectorTest, TaskCallbackServiceTest
* add RoundRobinHostManagerTest, ExecutorDispatcherTest
* refactor task response service
* add TaskResponseServiceTest
4 years ago
break60
64b28a5171
Resource tree code merge
4 years ago
oceanos
75712a515f
Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev
4 years ago
break60
fd250ff3d3
Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev
4 years ago
lgcareer
af8efe5184
add worker group and remove worker group id in dll ( #2313 )
4 years ago
dailidong
7e9f45e514
[Refactor worker] Optimize code , reduce cost time from 60ms to 0.5ms ( #2318 )
...
* update logback
* update log
* refactor worker registry (#2107 )
* Refactor worker (#2115 )
* refactor worker registry
* refactor master server
* Modify workgroupid parameter name (#2105 )
* Delete worker group management page
* Modify workgroupid parameter name
* Refactor worker (#2121 )
* refactor worker registry
* refactor master server
* refactor MasterSchedulerService
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath (#2126 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
* not exist in openjdk,just delete
* add master and worker properties
* add master and worker properties
* add master and worker properties
* fix cpu 100% bug
* simplify master、 worker、alert、dao properties
* add master and worker properties
* add master and worker properties
* add master and worker properties
* Optimize code , reduce cost time from 60ms to 0.5ms
* merge code form remote
Co-authored-by: Tboy <guo.jiwei@immomo.com>
Co-authored-by: break60 <790061044@qq.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
4 years ago
Tboy
5730bfe2be
Refactor worker ( #2319 )
...
* let quartz use the same datasource
* move master/worker config from dao.properties to each config
add master/worker registry test
* move mybatis config from application.properties to SpringConnectionFactory
* move mybatis-plus config from application.properties to SpringConnectionFactory
* refactor TaskCallbackService
* add ZookeeperNodeManagerTest
* add NettyExecutorManagerTest
* refactor TaskKillProcessor
* add RandomSelectorTest, RoundRobinSelectorTest, TaskCallbackServiceTest
* add RoundRobinHostManagerTest, ExecutorDispatcherTest
* refactor task response service
4 years ago
gabry.wu
eef8bf4e6d
Adapting partial code(file name start with P #2 ) to the sonar cloud rule ( #2260 )
4 years ago
孙朝和
d8cba5ff1b
resolve shell task build command nullpointexception ( #2267 )
...
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
4 years ago
t1mon
efde8fd4e7
Refactor BaseDataSource and unified handling of JDBC resource releases. ( #2184 )
...
* Refactor BaseDataSource.isConnectable().
* Add BaseDataSource test.
* Unified handling of JDBC resource releases.
* Fix sonar check error.
* Remove useless logger.
* Refactor BaseDataSource.getJdbcUrl().
* Modify ConnectionUtils.releaseResource().
* Add javadoc
* Remove useless method.
* Add BaseDataSourceTest UT
4 years ago
gabry.wu
d83a94a305
Adapting partial code(file name start with T) to the sonar cloud rule ( #2271 )
4 years ago
tswstarplanet
74525c28b5
Decouple DruidDataSource in ConnectionFactory and cache DataSource instance ( #2232 )
...
* Decouple DruidDataSource in ConnectionFactory and cache DataSource
instance
* init dataSource first
* fix code smell
* add unit test
* add unit test
* add unit test
* add unit test
* fix unit test
* delete useless logger
4 years ago
gabry.wu
42f9385ee4
Adapting partial code(file name start with W) to the sonar cloud rule ( #2287 )
4 years ago
WeiYe
a1b8386c8c
Update test code logger class name ( #2303 )
4 years ago
Tboy
e720ca5463
Refactor worker ( #2312 )
...
* let quartz use the same datasource
* move master/worker config from dao.properties to each config
add master/worker registry test
* move mybatis config from application.properties to SpringConnectionFactory
* move mybatis-plus config from application.properties to SpringConnectionFactory
* refactor TaskCallbackService
* add ZookeeperNodeManagerTest
* add NettyExecutorManagerTest
* refactor TaskKillProcessor
* add RandomSelectorTest, RoundRobinSelectorTest, TaskCallbackServiceTest
* add RoundRobinHostManagerTest, ExecutorDispatcherTest
4 years ago
xingchun-chen
5edc2af311
e2e add project and workflow case ( #2309 )
...
* add LoginTest license
* Delete useless packages
* e2e add project and workflow case
* e2e add project and workflow case
* e2e add project and workflow case
* e2e add project and workflow case
* e2e add project and workflow case
* e2e add project and workflow case
* e2e add project and workflow case
* e2e add project and workflow case
* e2e add project and workflow case
* e2e add project and workflow case
* e2e add project and workflow case
* e2e add project and workflow case
Co-authored-by: chenxingchun <438044805@qq.com>
4 years ago
qiaozhanwei
03e29b9311
1,ZookeeperRegister use common.properties zookeeperRoot path 2,api start exclude org.apache.dolphinscheduler.server.* ( #2307 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* Encapsulate the parameters required by sqltask
* 1,Encapsulate the parameters required by sqltask
2,SQLTask optimization
* AbstractTask modify
* ProcedureTask optimization
* MasterSchedulerService modify
* TaskUpdateQueueConsumer modify
* test
* DataxTask process run debug
* DataxTask process run debug
* add protobuf dependency,MR、Spark task etc need this
* TaskUpdateQueueConsumer modify
* TaskExecutionContextBuilder set TaskInstance workgroup
* WorkerGroupService queryAllGroup modify
query available work group
* 1,get workergroup from zk modify
2,SpringConnectionFactory repeat load modify
* master and worker register ip use OSUtils.getHost()
* ProcessInstance host set ip:port format
* worker fault tolerance modify
* Constants and .env modify
* master fault tolerant bug modify
* UT add pom.xml
* timing online modify
* when taskResponse is faster than taskAck to db,task state will error
add async queue and new a thread reslove this problem
* TaskExecutionContext set host
* 1,TaskManager refactor
2, api start load server dolphinschedule-daemon.sh modify
* 1,TaskManager refactor
2, api start load server dolphinschedule-daemon.sh modify
* add UT in pom.xml
* revert dolphinscheduler-daemon.sh
* ZookeeperRegister use common.properties zookeeperRoot path
* api start exclude org.apache.dolphinscheduler.server.*
* ZookeeperRegister use common.properties zookeeperRoot path
* 1,api start load server filter
2,SHELL task exitStatusCode modify
* java doc error modify
* java doc error modify
* remove todo
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
4 years ago
dailidong
ace907e7dd
[refactor-worker] simplify master、 worker、alert、dao properties ( #2304 )
...
* update logback
* update log
* refactor worker registry (#2107 )
* Refactor worker (#2115 )
* refactor worker registry
* refactor master server
* Modify workgroupid parameter name (#2105 )
* Delete worker group management page
* Modify workgroupid parameter name
* Refactor worker (#2121 )
* refactor worker registry
* refactor master server
* refactor MasterSchedulerService
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath (#2126 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
* not exist in openjdk,just delete
* add master and worker properties
* add master and worker properties
* add master and worker properties
* fix cpu 100% bug
* simplify master、 worker、alert、dao properties
* add master and worker properties
* add master and worker properties
* add master and worker properties
Co-authored-by: Tboy <guo.jiwei@immomo.com>
Co-authored-by: break60 <790061044@qq.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
4 years ago
qiaozhanwei
f112415b13
TaskManager refactor ( #2302 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* Encapsulate the parameters required by sqltask
* 1,Encapsulate the parameters required by sqltask
2,SQLTask optimization
* AbstractTask modify
* ProcedureTask optimization
* MasterSchedulerService modify
* TaskUpdateQueueConsumer modify
* test
* DataxTask process run debug
* DataxTask process run debug
* add protobuf dependency,MR、Spark task etc need this
* TaskUpdateQueueConsumer modify
* TaskExecutionContextBuilder set TaskInstance workgroup
* WorkerGroupService queryAllGroup modify
query available work group
* 1,get workergroup from zk modify
2,SpringConnectionFactory repeat load modify
* master and worker register ip use OSUtils.getHost()
* ProcessInstance host set ip:port format
* worker fault tolerance modify
* Constants and .env modify
* master fault tolerant bug modify
* UT add pom.xml
* timing online modify
* when taskResponse is faster than taskAck to db,task state will error
add async queue and new a thread reslove this problem
* TaskExecutionContext set host
* 1,TaskManager refactor
2, api start load server dolphinschedule-daemon.sh modify
* 1,TaskManager refactor
2, api start load server dolphinschedule-daemon.sh modify
* add UT in pom.xml
* revert dolphinscheduler-daemon.sh
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
4 years ago
break60
0d70e065ba
Change normalize.scss import method and animation.scss license modification ( #2295 )
...
* Change DOM label
* Change name to lowercase
* Limit customization file content to no more than 3000 lines
* dd branch flow node verification
* datax
* datax add custom
* Change normalize.scss import method and animation.scss license modification
4 years ago
qiaozhanwei
62f7d21bda
add async queue and new a thread reslove taskResponse is faster than taskAck to db ( #2297 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* Encapsulate the parameters required by sqltask
* 1,Encapsulate the parameters required by sqltask
2,SQLTask optimization
* AbstractTask modify
* ProcedureTask optimization
* MasterSchedulerService modify
* TaskUpdateQueueConsumer modify
* test
* DataxTask process run debug
* DataxTask process run debug
* add protobuf dependency,MR、Spark task etc need this
* TaskUpdateQueueConsumer modify
* TaskExecutionContextBuilder set TaskInstance workgroup
* WorkerGroupService queryAllGroup modify
query available work group
* 1,get workergroup from zk modify
2,SpringConnectionFactory repeat load modify
* master and worker register ip use OSUtils.getHost()
* ProcessInstance host set ip:port format
* worker fault tolerance modify
* Constants and .env modify
* master fault tolerant bug modify
* UT add pom.xml
* timing online modify
* when taskResponse is faster than taskAck to db,task state will error
add async queue and new a thread reslove this problem
* TaskExecutionContext set host
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
4 years ago
dailidong
7da35238e1
[Refactor worker] fix cpu pressure may be too high bug ( #2290 )
...
* update logback
* update log
* refactor worker registry (#2107 )
* Refactor worker (#2115 )
* refactor worker registry
* refactor master server
* Modify workgroupid parameter name (#2105 )
* Delete worker group management page
* Modify workgroupid parameter name
* Refactor worker (#2121 )
* refactor worker registry
* refactor master server
* refactor MasterSchedulerService
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath (#2126 )
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
* not exist in openjdk,just delete
* add master and worker properties
* add master and worker properties
* add master and worker properties
* fix cpu 100% bug
Co-authored-by: Tboy <guo.jiwei@immomo.com>
Co-authored-by: break60 <790061044@qq.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
4 years ago
qiaozhanwei
11e3bff63e
timing online modify ( #2296 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* Encapsulate the parameters required by sqltask
* 1,Encapsulate the parameters required by sqltask
2,SQLTask optimization
* AbstractTask modify
* ProcedureTask optimization
* MasterSchedulerService modify
* TaskUpdateQueueConsumer modify
* test
* DataxTask process run debug
* DataxTask process run debug
* add protobuf dependency,MR、Spark task etc need this
* TaskUpdateQueueConsumer modify
* TaskExecutionContextBuilder set TaskInstance workgroup
* WorkerGroupService queryAllGroup modify
query available work group
* 1,get workergroup from zk modify
2,SpringConnectionFactory repeat load modify
* master and worker register ip use OSUtils.getHost()
* ProcessInstance host set ip:port format
* worker fault tolerance modify
* Constants and .env modify
* master fault tolerant bug modify
* UT add pom.xml
* timing online modify
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
4 years ago
Tboy
1e5a1140ab
add RandomSelectorTest, RoundRobinSelectorTest, TaskCallbackServiceTest ( #2291 )
...
* let quartz use the same datasource
* move master/worker config from dao.properties to each config
add master/worker registry test
* move mybatis config from application.properties to SpringConnectionFactory
* move mybatis-plus config from application.properties to SpringConnectionFactory
* refactor TaskCallbackService
* add ZookeeperNodeManagerTest
* add NettyExecutorManagerTest
* refactor TaskKillProcessor
* add RandomSelectorTest, RoundRobinSelectorTest, TaskCallbackServiceTest
4 years ago
break60
5c2e94e99e
Change normalize.scss import method and animation.scss license modification
4 years ago
break60
1964d47bdb
Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev
4 years ago
Tboy
92fd6479a9
Refactor worker ( #2279 )
...
* let quartz use the same datasource
* move master/worker config from dao.properties to each config
add master/worker registry test
* move mybatis config from application.properties to SpringConnectionFactory
* move mybatis-plus config from application.properties to SpringConnectionFactory
* refactor TaskCallbackService
* add ZookeeperNodeManagerTest
* add NettyExecutorManagerTest
* refactor TaskKillProcessor
4 years ago
qiaozhanwei
052f9d10bf
UT Coverage rate test ( #2276 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* Encapsulate the parameters required by sqltask
* 1,Encapsulate the parameters required by sqltask
2,SQLTask optimization
* AbstractTask modify
* ProcedureTask optimization
* MasterSchedulerService modify
* TaskUpdateQueueConsumer modify
* test
* DataxTask process run debug
* DataxTask process run debug
* add protobuf dependency,MR、Spark task etc need this
* TaskUpdateQueueConsumer modify
* TaskExecutionContextBuilder set TaskInstance workgroup
* WorkerGroupService queryAllGroup modify
query available work group
* 1,get workergroup from zk modify
2,SpringConnectionFactory repeat load modify
* master and worker register ip use OSUtils.getHost()
* ProcessInstance host set ip:port format
* worker fault tolerance modify
* Constants and .env modify
* master fault tolerant bug modify
* UT add pom.xml
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
4 years ago