e31b48fdb2
1，get workergroup from zk modify 2，SpringConnectionFactory repeat load modify ( #2201 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* Encapsulate the parameters required by sqltask
* 1，Encapsulate the parameters required by sqltask
2，SQLTask optimization
* AbstractTask modify
* ProcedureTask optimization
* MasterSchedulerService modify
* TaskUpdateQueueConsumer modify
* test
* DataxTask process run debug
* DataxTask process run debug
* add protobuf dependency，MR、Spark task etc need this
* TaskUpdateQueueConsumer modify
* TaskExecutionContextBuilder set TaskInstance workgroup
* WorkerGroupService queryAllGroup modify
query available work group
* 1，get workergroup from zk modify
2，SpringConnectionFactory repeat load modify
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
4 years ago
69633a29c6
fix: Api server startup fail ( #2198 )
4 years ago
f28ba55014
fix ResultSet not close and reformat code ( #2183 )
...
* fix ResultSet not close
* fix ResultSet not close
* extract code of closing resource to a method
* remove redundant if condition
* modify e2e timeout
4 years ago
a3767642a5
change static field to instance field ( #2190 )
4 years ago
f6f29f4abb
Replace Charset.forName("UTF-8") with StandardCharsets.UTF_8. ( #2193 )
4 years ago
11dde20311
Refactor worker ( #2189 )
...
* let quartz use the same datasource
* move master/worker config from dao.properties to each config
add master/worker registry test
* move mybatis config from application.properties to SpringConnectionFactory
* move mybatis-plus config from application.properties to SpringConnectionFactory
4 years ago
aa650b4426
Worker group parameter modification ( #2188 )
...
* Delete worker group management page
* Modify workgroupid parameter name
* Worker group parameter modification
4 years ago
1658176ef5
Adapting partial code(file name start with P #1 ) to the sonar cloud rule ( #2171 ) ( #2187 )
4 years ago
e0fc3171da
Cache HadoopUtils instance with specific days expire time ( #2181 )
...
* Cache HadoopUtils instance with 7 days expire time
* solve sonar issue
* add kerberos expire time config
* move KERBEROS_EXPIRE_TIME to Constants.java
4 years ago
fcc57320d6
Adapting partial code(file name start with M #1 ) to the sonar cloud rule ( #2173 )
...
* Adapting partial code(file name start with M #1 ) to the sonar cloud rule
* remove unused import
4 years ago
75efbb5402
Refactor worker ( #2186 )
...
* let quartz use the same datasource
* move master/worker config from dao.properties to each config
add master/worker registry test
4 years ago
61ebbf2946
Adapting partial code(file name start with L) to the sonar cloud rule ( #2171 )
4 years ago
e0389aebf6
fix: NullPointException in the case of task queuing ( #2175 )
4 years ago
a076ae4c36
Adapting partial code(file name start with J) to the sonar cloud rule ( #2165 )
...
* Adapting partial code(file name start with J) to the sonar cloud rule
* fix unit failure
4 years ago
80c6ce5711
The master and worker server exit exception #2163 ( #2176 )
...
* fix : #2163
* fix: format
4 years ago
2e63dc9c96
Delete .asf.yaml ( #2168 )
4 years ago
e98c6f3008
Update .asf.yaml ( #2167 )
4 years ago
205af16d55
Create .asf.yaml ( #2166 )
4 years ago
daad5efbab
use static method in static class JSON ( #2156 )
4 years ago
dd2c50a10d
Toolbar of the DAG has not prompt ( #2161 )
...
* Toolbar of the DAG has not prompt
* Remove duplication code
4 years ago
71b11e2c08
DataxTask process test modify ( #2162 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* Encapsulate the parameters required by sqltask
* 1，Encapsulate the parameters required by sqltask
2，SQLTask optimization
* AbstractTask modify
* ProcedureTask optimization
* MasterSchedulerService modify
* TaskUpdateQueueConsumer modify
* test
* DataxTask process run debug
* DataxTask process run debug
* add protobuf dependency，MR、Spark task etc need this
* TaskUpdateQueueConsumer modify
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
4 years ago
a0088c1137
fix rpmbuild and DS Ambari plugin bug ( #2144 )
...
* Update quick-start.md
* 更新demo的地址
* add rpm plugin
* delete the git ignore files
* 测试git ignore
* 测试ignore
* optimize the .gitignore file
* add rpm package, delete unuse copy jar action in assembly
* delete docs dir
* add rpm plugin support
* add rpm plugin support
* add rpm plugin support
* dolphinscheduler-common have not bin dir
* dolphinscheduler-common have not bin dir
* delete unuse config about assembly
* add defineSatement
* add rpm plugin support
* add rpm plugin support
* add rpm plugin support
* add rpm support
* update install dirname , make front and backend together
* update rpm name
* update rpm name
* update rpm config
* add jars excludes
* add jars excludes
* add rpm plugion
* add rpm plugion
* add rpm plugion
* add auto create /opt/soft
* The install dir can not be fixed to apache-dolphinscheduler-incubating-1.2.1 , its version is 1.2.2
* install from rpm will create a link
* the ui install thrown rpm only support jetty not nginx
* the ui install thrown rpm only support jetty not nginx
* sql目录里加上soft_version文件
* env dir need in conf dir
* remove no use module
* add module
* updaate
* rpmbuild add dolphinscheduler-service module
4 years ago
93aa160698
Adapting partial code(file name start with I) to the sonar cloud rule ( #2155 )
4 years ago
607ec3d174
ProcedureTask process test modify ( #2159 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* Encapsulate the parameters required by sqltask
* 1，Encapsulate the parameters required by sqltask
2，SQLTask optimization
* AbstractTask modify
* ProcedureTask optimization
* MasterSchedulerService modify
* TaskUpdateQueueConsumer modify
* test
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
4 years ago
455d8c7a7c
Delete .asf.yaml ( #2158 )
4 years ago
91ecc549a1
fix bug: data too long for column 'app_link' ( #2146 )
...
* fix bug: data to long for app_link field
* change app_link field length in 1.2.2 ddl
* change app_link field type to text
4 years ago
68e4e5c4e0
add input cache for sqoop and conditions tasks ( #2118 )
...
Co-authored-by: zhukai <zhukai@ygsoft.com>
4 years ago
7473d5e928
Adapting partial code(file name start with H) to the sonar cloud rule ( #2145 )
4 years ago
9fda4da70e
fix: The toolbar buttons of Dag disable bug ( #2124 )
4 years ago
712a1bc126
Limit customization file content to no more than 3000 lines( #2128 ) ( #2141 )
...
* Change DOM label
* Change name to lowercase
* Limit customization file content to no more than 3000 lines
4 years ago
46e30b1e50
Add mail server configuration description information ( #2143 )
...
* Add mail server configuration description information
* Remove mail server configuration note in alert.properties
4 years ago
16eeaf50b4
Fix javadoc and DataxTaskTest UT error ( #2147 )
...
* Fix javadoc and DataxTaskTest UT error
* Fix UT error
* LoggerServiceTest UT error
* WorkerGroupServiceTest UT error
Modify Host.of()
* Add address NPE check
* Trigger actions rerun
* Try to solve maven VM crash
* Try to solve maven VM crash 1
* Try to solve maven VM crash 2
* Try to solve maven VM crash 3
4 years ago
afb907b4b0
fix unequal number of arguments and placeholder ( #2149 )
4 years ago
e3065dbfcc
fix field access modifier ( #2150 )
4 years ago
e27ba3fd97
use logger to print exception ( #2151 )
4 years ago
ebfacbfcfa
docker run failed when docker command was `docker run -d` ( #2125 )
...
* docker run failed when docker command was `docker run -d`
1. add tini
2. replace `tee` command by `tail` command in startup.sh
3. api-server need to support zookeeper
* modify `check.sh` in dockerfile
* add init directories and log files
4 years ago
b03e0a02c5
fix wrong sonar analysis with checkout V2. ( #2148 )
4 years ago
84bcdc7896
Limit customization file content to no more than 3000 lines
4 years ago
1db4ac6b63
edit README.md modify word from expand to extend ( #2138 )
...
Co-authored-by: guirong.ggr <guirong.ggr@dtwave-inc.com>
4 years ago
7f72df4ca2
let quartz use the same datasource ( #2137 )
4 years ago
ce17531024
add .asf.yaml file ( #2136 )
...
Co-authored-by: guirong.ggr <guirong.ggr@dtwave-inc.com>
4 years ago
ebf7a980d2
Adapting partial code(file name start with F) to the sonar cloud rule ( #2045 )
...
* Adapting partial code(file name start with F) to the sonar cloud rule
* add more unit test
* add License
* add includes configuration to maven-surefire-plugin
* fix getResourceFilesList incorrect logic
4 years ago
fd2c2eeb32
1，encapsulate the parameters required by sqltask 2，SQLTask optimization ( #2135 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* Encapsulate the parameters required by sqltask
* 1，Encapsulate the parameters required by sqltask
2，SQLTask optimization
* AbstractTask modify
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
4 years ago
bbe2cd4864
refactor worker group ( #2132 )
...
* replace worker group id with worker group
* add worker group field in ddl
* remove worker group id
4 years ago
621ebf6090
Merge branch 'refactor-worker' into refactor-worker
4 years ago
9224b49b58
access field handle of FileDescriptor in ProcessImplForWin32 by reflection for portability ( #2113 )
...
* access field handle of FileDescriptor in ProcessImplForWin32 by reflection for portability
Current implementation relies on `sun.misc.JavaIOFileDescriptorAccess`
which is only accessible on oraclejdk8.
Basically the demand is getting & setting `handle` field of
`FileDescriptor`, so we can directly do that with reflection.
Though, I suspect the necessity we introduce ProcessImplForWin32. Maybe
we could have a better way to support worker server to run bat script.
* harden initialization of ProcessImplForWin32
* ignore ShellTaskTest#testHandleForWindows outside Windows
4 years ago
f551913249
split logback config into seperate child module logback ( #2120 )
...
* update logback
* update log
4 years ago
20af8ff93e
cancelTaskInstance set TaskExecutionContext host,logPath,executePath ( #2126 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
4 years ago
d7ad1d8b05
Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev
4 years ago
fd6f13fff7
Refactor worker ( #2121 )
...
* refactor worker registry
* refactor master server
* refactor MasterSchedulerService
4 years ago
qiaozhanwei
Rubik-W
tswstarplanet
t1mon
Tboy
break60
gabry.wu
DS
liwenhe1993
gaojun2048
dailidong
Yelli
zhukai
Jave-Chen
AhahaGe
lgcareer
Tboy
tison