AhahaGe
ce17531024
add .asf.yaml file ( #2136 )
...
Co-authored-by: guirong.ggr <guirong.ggr@dtwave-inc.com>
5 years ago
gabry.wu
ebf7a980d2
Adapting partial code(file name start with F) to the sonar cloud rule ( #2045 )
...
* Adapting partial code(file name start with F) to the sonar cloud rule
* add more unit test
* add License
* add includes configuration to maven-surefire-plugin
* fix getResourceFilesList incorrect logic
5 years ago
qiaozhanwei
fd2c2eeb32
1,encapsulate the parameters required by sqltask 2,SQLTask optimization ( #2135 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* Encapsulate the parameters required by sqltask
* 1,Encapsulate the parameters required by sqltask
2,SQLTask optimization
* AbstractTask modify
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
5 years ago
lgcareer
bbe2cd4864
refactor worker group ( #2132 )
...
* replace worker group id with worker group
* add worker group field in ddl
* remove worker group id
5 years ago
Tboy
621ebf6090
Merge branch 'refactor-worker' into refactor-worker
5 years ago
tison
9224b49b58
access field handle of FileDescriptor in ProcessImplForWin32 by reflection for portability ( #2113 )
...
* access field handle of FileDescriptor in ProcessImplForWin32 by reflection for portability
Current implementation relies on `sun.misc.JavaIOFileDescriptorAccess`
which is only accessible on oraclejdk8.
Basically the demand is getting & setting `handle` field of
`FileDescriptor`, so we can directly do that with reflection.
Though, I suspect the necessity we introduce ProcessImplForWin32. Maybe
we could have a better way to support worker server to run bat script.
* harden initialization of ProcessImplForWin32
* ignore ShellTaskTest#testHandleForWindows outside Windows
5 years ago
dailidong
f551913249
split logback config into seperate child module logback ( #2120 )
...
* update logback
* update log
5 years ago
qiaozhanwei
20af8ff93e
cancelTaskInstance set TaskExecutionContext host,logPath,executePath ( #2126 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
5 years ago
break60
d7ad1d8b05
Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev
5 years ago
Tboy
fd6f13fff7
Refactor worker ( #2121 )
...
* refactor worker registry
* refactor master server
* refactor MasterSchedulerService
5 years ago
Technoboy-
932fc72794
Merge branch 'refactor-worker' of github.com:Technoboy-/incubator-dolphinscheduler into refactor-worker
5 years ago
Technoboy-
cc97052cc0
refactor MasterSchedulerService
5 years ago
Tboy
645529e4cc
Merge pull request #24 from apache/refactor-worker
...
Refactor worker
5 years ago
gabry.wu
450a1f56fc
Adapting partial code(file name start with E) to the sonar cloud rule ( #1999 )
...
* Adapting partial code(file name start with E) to the sonar cloud rule
* remove isEmpty invoke
* resolve conflicts
5 years ago
gabry.wu
ad381fcb71
move UT class to reasonable package ( #2116 )
5 years ago
Rubik-W
275df85250
fix: remove redundant import class ( #2110 )
5 years ago
break60
290772dfaa
Modify workgroupid parameter name ( #2105 )
...
* Delete worker group management page
* Modify workgroupid parameter name
5 years ago
tswstarplanet
b2268b1af9
use stringbuilder to append string ( #2108 )
5 years ago
liwenhe1993
89b42db3b4
Add docker internal server check script ( #2092 )
...
* add environment variables add checkpoint.sh file to docker image
1. add `POSTGRESQL_DATABASE` environment variable
2. add `DOLPHINSCHEDULER_DATA_BASEDIR_PATH` environment variable
3. add `DOLPHINSCHEDULER_DATA_DOWNLOAD_BASEDIR_PATH` environment variable
4. add `DOLPHINSCHEDULER_PROCESS_EXEC_BASEPATH` environment variable
5. add checkpoint.sh file to check server process
6. modify `README.md` and `README_zh_CN.md`
* add environment variables add checkpoint.sh file to docker image
1. add `POSTGRESQL_DATABASE` environment variable
2. add `DOLPHINSCHEDULER_DATA_BASEDIR_PATH` environment variable
3. add `DOLPHINSCHEDULER_DATA_DOWNLOAD_BASEDIR_PATH` environment variable
4. add `DOLPHINSCHEDULER_PROCESS_EXEC_BASEPATH` environment variable
5. add checkpoint.sh file to check server process
6. modify `README.md` and `README_zh_CN.md`
* add logger port and modify dolphinscheduler_env to dolphinscheduler_env.sh
5 years ago
Tboy
0febd9530c
Refactor worker ( #2115 )
...
* refactor worker registry
* refactor master server
5 years ago
Technoboy-
a4c2dfa311
refactor master server
5 years ago
Tboy
c3293887fa
Merge pull request #23 from apache/refactor-worker
...
refactor worker registry (#2107 )
5 years ago
Tboy
d617f9df32
refactor worker registry ( #2107 )
5 years ago
gabry.wu
0e1dd8d7d9
remove incorrect or unnecessary URL tag ( #2099 )
5 years ago
Technoboy-
76fd384a2e
refactor worker registry
5 years ago
Tboy
a51f3ea7c7
Merge pull request #22 from apache/refactor-worker
...
Refactor worker
5 years ago
Tboy
5c621009b5
Merge branch 'refactor-worker' into refactor-worker
5 years ago
qiaozhanwei
043df7e189
WorkerServer refactor ( #2106 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
5 years ago
Tboy
4eebde8355
Refactor worker ( #2103 )
...
* refactor kill logic
* refactor ExecutionContext
* refactor worker
5 years ago
Technoboy-
aeb8b852e7
refactor worker
5 years ago
Tboy
b42357a16d
Merge pull request #21 from apache/refactor-worker
...
TaskPriority refactor (#2097 )
5 years ago
qiaozhanwei
eb45ff9e83
TaskPriority refactor ( #2097 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
5 years ago
dailidong
4bb97dfb8b
fix actions/checkout@v2 problem ( #2095 )
5 years ago
dailidong
6070738acb
Support worker server to run bat script ( #2023 )
...
* Support worker server to run bat script
1. Reimplement ProcessImpl.java, ProcessEnvironment.java and ProcessBuilder.java for Windows
2. Modify shell task code for windows
3. Add ASF License
* Add Unit Test
5 years ago
dailidong
1f92b4c4db
Optimize ParameterUtils.curingGlobalParams() execution efficiency ( #2090 )
...
* Optimize ParameterUtils.curingGlobalParams() execution efficiency
* Remove excess null check
5 years ago
Tboy
9964cf9057
Merge pull request #20 from apache/refactor-worker
...
task prioriry refator (#2094 )
5 years ago
Tboy
7ed28af4fe
Upgrade curator version ( #2083 )
...
* upgrade curator version
issue: https://github.com/apache/incubator-dolphinscheduler/issues/2020
curator issue: https://github.com/apache/curator/pull/297
* add DefaultEnsembleProvider override
* add some unit test
* add License
5 years ago
qiaozhanwei
8aaee39aa2
task prioriry refator ( #2094 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
5 years ago
dailidong
4230d5a737
add embedded database h2 and update embedded zookeeper ( #2082 )
...
add embedded database h2 and update embedded zookeeper
5 years ago
dailidong
6ec85540bf
fix typo ( #2093 )
5 years ago
Jave-Chen
d33b883420
Merge pull request #2085 from khadgarmage/ci-action-alter
...
action push delete
5 years ago
xiaochun.liu
18d7ff274b
action push delete
5 years ago
qiaozhanwei
d5c795bd45
fix ugly logger usage ( #2063 )
...
* fix ugly logger usage
* remove space
5 years ago
qiaozhanwei
3e5a939913
delete unused import ( #2066 )
5 years ago
qiaozhanwei
b8014d91cf
add workflow id, name for email content ( #2071 )
...
* add tenantService UT
* fix spell
* #464 email content
* modify mysql1.2.2 ddl
5 years ago
Tboy
1d4bf420b0
Merge pull request #19 from apache/refactor-worker
...
Refactor worker (#2081 )
5 years ago
Tboy
15bf740221
Refactor worker ( #2081 )
...
* refactor kill logic
* refactor ExecutionContext
5 years ago
Technoboy-
09173e8d23
refactor ExecutionContext
5 years ago
Tboy
b0f9cd720b
Merge pull request #18 from apache/refactor-worker
...
Refactor worker
5 years ago
zhukai
3186545da1
1. add input cache for datax task ( #2034 )
...
2. add cache for codemirror component, include python, shell and sql task.
5 years ago