e0389aebf6
fix: NullPointException in the case of task queuing ( #2175 )
4 years ago
a076ae4c36
Adapting partial code(file name start with J) to the sonar cloud rule ( #2165 )
...
* Adapting partial code(file name start with J) to the sonar cloud rule
* fix unit failure
4 years ago
80c6ce5711
The master and worker server exit exception #2163 ( #2176 )
...
* fix : #2163
* fix: format
4 years ago
2e63dc9c96
Delete .asf.yaml ( #2168 )
4 years ago
e98c6f3008
Update .asf.yaml ( #2167 )
4 years ago
205af16d55
Create .asf.yaml ( #2166 )
4 years ago
daad5efbab
use static method in static class JSON ( #2156 )
4 years ago
dd2c50a10d
Toolbar of the DAG has not prompt ( #2161 )
...
* Toolbar of the DAG has not prompt
* Remove duplication code
4 years ago
71b11e2c08
DataxTask process test modify ( #2162 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* Encapsulate the parameters required by sqltask
* 1，Encapsulate the parameters required by sqltask
2，SQLTask optimization
* AbstractTask modify
* ProcedureTask optimization
* MasterSchedulerService modify
* TaskUpdateQueueConsumer modify
* test
* DataxTask process run debug
* DataxTask process run debug
* add protobuf dependency，MR、Spark task etc need this
* TaskUpdateQueueConsumer modify
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
4 years ago
a0088c1137
fix rpmbuild and DS Ambari plugin bug ( #2144 )
...
* Update quick-start.md
* 更新demo的地址
* add rpm plugin
* delete the git ignore files
* 测试git ignore
* 测试ignore
* optimize the .gitignore file
* add rpm package, delete unuse copy jar action in assembly
* delete docs dir
* add rpm plugin support
* add rpm plugin support
* add rpm plugin support
* dolphinscheduler-common have not bin dir
* dolphinscheduler-common have not bin dir
* delete unuse config about assembly
* add defineSatement
* add rpm plugin support
* add rpm plugin support
* add rpm plugin support
* add rpm support
* update install dirname , make front and backend together
* update rpm name
* update rpm name
* update rpm config
* add jars excludes
* add jars excludes
* add rpm plugion
* add rpm plugion
* add rpm plugion
* add auto create /opt/soft
* The install dir can not be fixed to apache-dolphinscheduler-incubating-1.2.1 , its version is 1.2.2
* install from rpm will create a link
* the ui install thrown rpm only support jetty not nginx
* the ui install thrown rpm only support jetty not nginx
* sql目录里加上soft_version文件
* env dir need in conf dir
* remove no use module
* add module
* updaate
* rpmbuild add dolphinscheduler-service module
4 years ago
93aa160698
Adapting partial code(file name start with I) to the sonar cloud rule ( #2155 )
4 years ago
607ec3d174
ProcedureTask process test modify ( #2159 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* Encapsulate the parameters required by sqltask
* 1，Encapsulate the parameters required by sqltask
2，SQLTask optimization
* AbstractTask modify
* ProcedureTask optimization
* MasterSchedulerService modify
* TaskUpdateQueueConsumer modify
* test
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
4 years ago
455d8c7a7c
Delete .asf.yaml ( #2158 )
4 years ago
91ecc549a1
fix bug: data too long for column 'app_link' ( #2146 )
...
* fix bug: data to long for app_link field
* change app_link field length in 1.2.2 ddl
* change app_link field type to text
4 years ago
68e4e5c4e0
add input cache for sqoop and conditions tasks ( #2118 )
...
Co-authored-by: zhukai <zhukai@ygsoft.com>
4 years ago
7473d5e928
Adapting partial code(file name start with H) to the sonar cloud rule ( #2145 )
4 years ago
9fda4da70e
fix: The toolbar buttons of Dag disable bug ( #2124 )
4 years ago
712a1bc126
Limit customization file content to no more than 3000 lines( #2128 ) ( #2141 )
...
* Change DOM label
* Change name to lowercase
* Limit customization file content to no more than 3000 lines
4 years ago
46e30b1e50
Add mail server configuration description information ( #2143 )
...
* Add mail server configuration description information
* Remove mail server configuration note in alert.properties
4 years ago
16eeaf50b4
Fix javadoc and DataxTaskTest UT error ( #2147 )
...
* Fix javadoc and DataxTaskTest UT error
* Fix UT error
* LoggerServiceTest UT error
* WorkerGroupServiceTest UT error
Modify Host.of()
* Add address NPE check
* Trigger actions rerun
* Try to solve maven VM crash
* Try to solve maven VM crash 1
* Try to solve maven VM crash 2
* Try to solve maven VM crash 3
4 years ago
afb907b4b0
fix unequal number of arguments and placeholder ( #2149 )
4 years ago
e3065dbfcc
fix field access modifier ( #2150 )
4 years ago
e27ba3fd97
use logger to print exception ( #2151 )
4 years ago
ebfacbfcfa
docker run failed when docker command was `docker run -d` ( #2125 )
...
* docker run failed when docker command was `docker run -d`
1. add tini
2. replace `tee` command by `tail` command in startup.sh
3. api-server need to support zookeeper
* modify `check.sh` in dockerfile
* add init directories and log files
4 years ago
b03e0a02c5
fix wrong sonar analysis with checkout V2. ( #2148 )
4 years ago
84bcdc7896
Limit customization file content to no more than 3000 lines
4 years ago
1db4ac6b63
edit README.md modify word from expand to extend ( #2138 )
...
Co-authored-by: guirong.ggr <guirong.ggr@dtwave-inc.com>
4 years ago
7f72df4ca2
let quartz use the same datasource ( #2137 )
4 years ago
ce17531024
add .asf.yaml file ( #2136 )
...
Co-authored-by: guirong.ggr <guirong.ggr@dtwave-inc.com>
4 years ago
ebf7a980d2
Adapting partial code(file name start with F) to the sonar cloud rule ( #2045 )
...
* Adapting partial code(file name start with F) to the sonar cloud rule
* add more unit test
* add License
* add includes configuration to maven-surefire-plugin
* fix getResourceFilesList incorrect logic
4 years ago
fd2c2eeb32
1，encapsulate the parameters required by sqltask 2，SQLTask optimization ( #2135 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* Encapsulate the parameters required by sqltask
* 1，Encapsulate the parameters required by sqltask
2，SQLTask optimization
* AbstractTask modify
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
4 years ago
bbe2cd4864
refactor worker group ( #2132 )
...
* replace worker group id with worker group
* add worker group field in ddl
* remove worker group id
4 years ago
621ebf6090
Merge branch 'refactor-worker' into refactor-worker
4 years ago
9224b49b58
access field handle of FileDescriptor in ProcessImplForWin32 by reflection for portability ( #2113 )
...
* access field handle of FileDescriptor in ProcessImplForWin32 by reflection for portability
Current implementation relies on `sun.misc.JavaIOFileDescriptorAccess`
which is only accessible on oraclejdk8.
Basically the demand is getting & setting `handle` field of
`FileDescriptor`, so we can directly do that with reflection.
Though, I suspect the necessity we introduce ProcessImplForWin32. Maybe
we could have a better way to support worker server to run bat script.
* harden initialization of ProcessImplForWin32
* ignore ShellTaskTest#testHandleForWindows outside Windows
4 years ago
f551913249
split logback config into seperate child module logback ( #2120 )
...
* update logback
* update log
4 years ago
20af8ff93e
cancelTaskInstance set TaskExecutionContext host,logPath,executePath ( #2126 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
4 years ago
d7ad1d8b05
Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev
4 years ago
fd6f13fff7
Refactor worker ( #2121 )
...
* refactor worker registry
* refactor master server
* refactor MasterSchedulerService
4 years ago
932fc72794
Merge branch 'refactor-worker' of github.com:Technoboy-/incubator-dolphinscheduler into refactor-worker
4 years ago
cc97052cc0
refactor MasterSchedulerService
4 years ago
645529e4cc
Merge pull request #24 from apache/refactor-worker
...
Refactor worker
4 years ago
450a1f56fc
Adapting partial code(file name start with E) to the sonar cloud rule ( #1999 )
...
* Adapting partial code(file name start with E) to the sonar cloud rule
* remove isEmpty invoke
* resolve conflicts
4 years ago
ad381fcb71
move UT class to reasonable package ( #2116 )
4 years ago
275df85250
fix: remove redundant import class ( #2110 )
4 years ago
290772dfaa
Modify workgroupid parameter name ( #2105 )
...
* Delete worker group management page
* Modify workgroupid parameter name
4 years ago
b2268b1af9
use stringbuilder to append string ( #2108 )
4 years ago
89b42db3b4
Add docker internal server check script ( #2092 )
...
* add environment variables add checkpoint.sh file to docker image
1. add `POSTGRESQL_DATABASE` environment variable
2. add `DOLPHINSCHEDULER_DATA_BASEDIR_PATH` environment variable
3. add `DOLPHINSCHEDULER_DATA_DOWNLOAD_BASEDIR_PATH` environment variable
4. add `DOLPHINSCHEDULER_PROCESS_EXEC_BASEPATH` environment variable
5. add checkpoint.sh file to check server process
6. modify `README.md` and `README_zh_CN.md`
* add environment variables add checkpoint.sh file to docker image
1. add `POSTGRESQL_DATABASE` environment variable
2. add `DOLPHINSCHEDULER_DATA_BASEDIR_PATH` environment variable
3. add `DOLPHINSCHEDULER_DATA_DOWNLOAD_BASEDIR_PATH` environment variable
4. add `DOLPHINSCHEDULER_PROCESS_EXEC_BASEPATH` environment variable
5. add checkpoint.sh file to check server process
6. modify `README.md` and `README_zh_CN.md`
* add logger port and modify dolphinscheduler_env to dolphinscheduler_env.sh
4 years ago
0febd9530c
Refactor worker ( #2115 )
...
* refactor worker registry
* refactor master server
4 years ago
a4c2dfa311
refactor master server
4 years ago
c3293887fa
Merge pull request #23 from apache/refactor-worker
...
refactor worker registry (#2107 )
4 years ago
Rubik-W
gabry.wu
DS
liwenhe1993
qiaozhanwei
gaojun2048
dailidong
Yelli
zhukai
break60
t1mon
tswstarplanet
Jave-Chen
AhahaGe
Tboy
lgcareer
Tboy
tison