7f725c32b5
Update pom.xml
5 years ago
f56ae9e64a
add http task timeout & test unit ( #2955 )
...
* add http task timeout & test unit
* add http task timeout & test unit
Co-authored-by: zhanglong <zhanglong@ysstech.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
5 years ago
65841a5709
1. change windows new line 'CR' to linux new line 'LF'
...
2. solve conflict
5 years ago
71619ed29b
Update pom.xml
5 years ago
b5a296f4cd
Update pom.xml
5 years ago
094d09ba74
Update pom.xml
5 years ago
822addac30
add sqoop task UT to pom
5 years ago
fad09075f9
remove fastjson maven dependence
5 years ago
6e907f150c
1. specified jackson Date format.
...
2. remove getMapper method,replace with toList()
3. rewrite toList method,support constructCollectionType.
4. parseObject(resp, Map.class) replace to toMap() method
5. rename JacksonSerialize to JsonSerializer
rewrite createArrayNode,createObjectNode method will Affect too many lines ，i will fix it in the next pr.
5 years ago
422fe5136d
[license] add license and notice ( #2916 )
...
* fix bug: send task result to master until success.
* fix bug: send task result to master until success.
* add sleep
* set null if send result failed
* set null if send result failed
* change max length of username to 39
* test for gc
* change version to 1.3.0
* check and add some license files
* check and add some license files
* revert
Co-authored-by: baoliang <baoliang@analysys.com.cn>
5 years ago
b21259d600
add MicroBenchMark model ( #2851 )
...
* add MicroBenchMark model
* add log
* fix propertyName
* small change
* add ut
* format
* add @Test
* fix maven compile error
* remove enum set method
* exclusion microbench ut
* small change
* Use assembly instead of shade
Co-authored-by: dailidong <dailidong66@gmail.com>
5 years ago
4a10157506
ut fix
5 years ago
ef4c1e7930
add source license full text into source license ( #2840 )
...
* add source license full text into source license
* exclude ScriptRunner.java on rat check
5 years ago
0130da2bdf
Fix [Bug] process definition json worker group convert #2794 ( #2795 )
...
* add LoggerServerTest UT
* add LoggerServerTest UT
* add LoggerServerTest UT
add RemoveTaskLogRequestCommandTest UT
add RemoveTaskLogResponseCommandTest
* master select worker filter high load worker #2704
* master select worker filter high load worker #2704
* master select worker filter high load worker #2704
* master select worker filter high load worker #2704
* master select worker filter high load worker #2704
* master select worker filter high load worker #2704
* add not worker log and remove worker invalid property
* process definition json worker group convert #2794
* process definition json worker group convert #2794
* process definition json worker group convert #2794
* process definition json worker group convert #2794
* process definition json worker group convert #2794
* process definition json worker group convert #2794
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
5 years ago
c28a697ee8
add RetryerUtils to retry async & sync function gracefully ( #2697 )
...
* add RetryerUtils tool class for async & sync retry invoking
* add RetryerUtils tool class for async & sync retry invoking
* add guava-retrying-2.0.0.jar to known-dependencies.txt
* fix javadoc
* fix code smell
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
5 years ago
f4583d78e9
add UT ( #2717 )
...
* add LoggerServerTest UT
* add LoggerServerTest UT
* add LoggerServerTest UT
add RemoveTaskLogRequestCommandTest UT
add RemoveTaskLogResponseCommandTest
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
5 years ago
689c216050
add LoggerServerTest UT ( #2713 )
...
* add LoggerServerTest UT
* add LoggerServerTest UT
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
5 years ago
de3a81cab5
support ali nail(ding talk) , issue:2458 ( #2594 )
...
* Create TextMessage.java
* Create DingTalkUtils.java
* Update alert.properties
* Update Constants.java
* Update Constants.java
* Create DingTalkMsgFormatter.java
* Update alert.properties
* Create DingTalkMsgFormatterTest.java
* Create DingTalkUtilsTest.java
* Update DingTalkUtils.java
* Create DingTalkManager.java
* Update EmailAlertPlugin.java
* feature: test send msg with proxy
* fix:delete comment and add exception info
* Update DingTalkUtilsTest.java
* Update DingTalkUtils.java
* Update DingTalkManager.java
* Update EmailAlertPlugin.java
* Update DingTalkUtilsTest.java
* Update DingTalkMsgFormatter.java
* Update DingTalkUtils.java
* Update DingTalkMsgFormatterTest.java
* Update DingTalkUtilsTest.java
* Update TextMessage.java
* Update DingTalkManager.java
* Delete TextMessage.java
* fix: add test case
* fix:add http client test case
* fix:delete ignore test case
* Update pom.xml
Co-authored-by: dailidong <dailidong66@gmail.com>
5 years ago
6e08b29254
Add a script to check the license ( #2552 )
...
* Add a script to check the license
### Motivation
Check licenses in the distribution package
### Modification
1. Add a script to check licenses, and list known licenses.
1. Remove unused dep license, lombok.
1. Remove unnecessary ignored patterns in .gitignore.
1. Add missing config items to apache-rat plugin, which just checked a small part of files before.
1. Add check to GitHub Actions process
### Result
1. Newly-added dependencies should be checked and confirmed.
1. Closes #1578
* Separate rat and dependency check
* Tee dependencies to stdout
* Add more debug messages
* Check light-weight task first
5 years ago
906ffa09a9
resolve code checks
5 years ago
85f3ac10a0
feat: add plugin management for alert service
5 years ago
a12103b08c
no valid worker group，master can kill task directly ( #2541 )
...
* dispatch task fail will set task status failed
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，no worker condition , master will while ture wait for worker startup
2，worker response task status sync wait for result
* 1，task status statistics and process status statistics bug fix (#2357 )
2，worker group bug fix
* 1，task status statistics and process status statistics bug fix (#2357 )
2，worker group bug fix
* 1，task status statistics and process status statistics bug fix (#2357 )
2，worker group bug fix
* 1，task status statistics and process status statistics bug fix (#2357 )
2，worker group bug fix
* send mail error, #2466 bug fix
* send mail error, #2466 bug fix
* send mail error, #2466 bug fix
* send mail error, #2466 bug fix
* #2486 bug fix
* host and workergroup compatible
* EnterpriseWeChatUtils modify
* EnterpriseWeChatUtils modify
* EnterpriseWeChatUtils modify
* #2499 bug fix
* add comment
* revert comment
* revert comment
* #2499 buf fix
* #2499 bug fix
* #2499 bug fix
* #2499 bug fix
* #2499 bug fix
* #2499 bug fix
* #2499 bug fix
* no valid worker group，master can kill task directly
* no valid worker group，master can kill task directly
* no valid worker group，master can kill task directly
* no valid worker group，master can kill task directly
* no valid worker group，master can kill task directly
* no valid worker group，master can kill task directly
* no valid worker group，master can kill task directly
* no valid worker group，master can kill task directly
* no valid worker group，master can kill task directly
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
5 years ago
64d335d9f2
copy process feature #2412 ( #2471 )
...
* Update pom.xml (#2467 )
* add copy process
* add copy process 2
* add copy process 3
* add copy process 4
* add copy process 5
Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
5 years ago
44770a8767
[ fix #2442 ] It will give a error tip that resource not exist exwhen update resources. ( #2483 )
...
* fix #2442 and remove unavailable code
* revert verifyResourceName method
* Add ServiceException
* add ServiceExceptionTest
* update ServiceExceptionTest
* add ServiceExceptionTest in pom
5 years ago
eec92ed1fc
simplify and optimize config ( #2469 )
...
* [Refactor worker] simplify and optimize config (#2386 )
* simplify config
* simplify config
* simplify and optimize config
* Update HadoopUtils.java
optimize HadoopUtils
* Update HadoopUtilsTest.java
* Update HadoopUtilsTest.java
* Update HadoopUtilsTest.java
* Update HttpUtils.java
* Update pom.xml
* Update HadoopUtilsTest.java
* Update HadoopUtilsTest.java
* Update HadoopUtilsTest.java
* Update HadoopUtilsTest.java
* Update HadoopUtilsTest.java
* Update HadoopUtilsTest.java
5 years ago
0b405b6fc8
Update pom.xml
5 years ago
dc36c77ba7
Update pom.xml ( #2467 )
5 years ago
0f54149ae3
exclude HadoopUtils Test ( #2465 )
...
* Update pom.xml
* Update pom.xml
5 years ago
4946e88872
move dependent and conditions to master server ( #2423 )
...
* move condition and dependent to master
* move conditions to master
* move conditions and dependent task to master
* move conditions to master
* update test
* add log
* add test for dependent task
* add test for dependent task
* update
* update
* refactor complexity code
* refactor complexity code
* add conditions task test
* add conditions task test
* update
* update host to host:port
* update logback.xml
Co-authored-by: baoliang <baoliang@analysys.com.cn>
5 years ago
ca66cfec17
solve code spell problem ( #2419 )
...
* solve code spell problem
* remove controller test
* add controller test
* sovle spell problem and sovle conflict
* sovle spell problem and sovle conflict
* sovle spell problem and add test
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
5 years ago
3c9ba0a3f3
api server exception management and code optimization ( #397 ) ( #2397 )
...
* api server exception management and code optimization (#397 )
* add unit test for exception handler.
* fix some style and naming issues
Co-authored-by: dailidong <dailidong66@gmail.com>
5 years ago
ff535e766d
update UT in pom.xml
5 years ago
9d06fe7755
update UT in pom.xml
5 years ago
b2c92deace
update UT in pom.xml
5 years ago
66d587c40d
update UT in pom.xml
5 years ago
b1a773f341
update UT in pom.xml
5 years ago
9de9f18c08
update UT in pom.xml
5 years ago
4160c21d8f
update UT in pom.xml
5 years ago
b581f02a21
add dao other mapper add in pom.xml
5 years ago
f02e7e8b37
pom modify
5 years ago
f65ab56d2d
UT modify
5 years ago
9dfc93e0a2
Exclude .mvn/jvm.config and .mvn/wrapper/maven-wrapper.properties ( #2387 )
...
* Remove .helmignore file
* Exclude .mvn/jvm.config and .mvn/wrapper/maven-wrapper.properties
When running rat, exclude .mvn/jvm.config and .mvn/wrapper/maven-wrapper.properties
5 years ago
af2b3187b9
update pom.xml
5 years ago
433b41dd7f
refactor-worker merge to dev bug fix
5 years ago
94713cef79
refactor-worker merge to dev bug fix
5 years ago
69e000b542
#2282 fix workflow dependent bug ( #2329 )
...
* fix workflow dependent bug
* fix workflow dependent bug 2
* fix workflow dependent bug 2
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
5 years ago
526e5c91f1
Connection mode of adding Sid to Oracle ( #2254 )
...
* Connection mode of adding Sid to Oracle
* Remove code
* Add asf
* Add unit test
* Add unit test
* Add unit test
* solve the conflict
5 years ago
68cb81fdf5
1，add UT in pom 2，refactor TaskUpdateQueue ( #2326 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* Encapsulate the parameters required by sqltask
* 1，Encapsulate the parameters required by sqltask
2，SQLTask optimization
* AbstractTask modify
* ProcedureTask optimization
* MasterSchedulerService modify
* TaskUpdateQueueConsumer modify
* test
* DataxTask process run debug
* DataxTask process run debug
* add protobuf dependency，MR、Spark task etc need this
* TaskUpdateQueueConsumer modify
* TaskExecutionContextBuilder set TaskInstance workgroup
* WorkerGroupService queryAllGroup modify
query available work group
* 1，get workergroup from zk modify
2，SpringConnectionFactory repeat load modify
* master and worker register ip use OSUtils.getHost()
* ProcessInstance host set ip:port format
* worker fault tolerance modify
* Constants and .env modify
* master fault tolerant bug modify
* UT add pom.xml
* timing online modify
* when taskResponse is faster than taskAck to db，task state will error
add async queue and new a thread reslove this problem
* TaskExecutionContext set host
* 1，TaskManager refactor
2， api start load server dolphinschedule-daemon.sh modify
* 1，TaskManager refactor
2， api start load server dolphinschedule-daemon.sh modify
* add UT in pom.xml
* revert dolphinscheduler-daemon.sh
* ZookeeperRegister use common.properties zookeeperRoot path
* api start exclude org.apache.dolphinscheduler.server.*
* ZookeeperRegister use common.properties zookeeperRoot path
* 1，api start load server filter
2，SHELL task exitStatusCode modify
* java doc error modify
* java doc error modify
* remove todo
* add UT in pom
* 1，add UT in pom
2，refactor TaskUpdateQueue
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
5 years ago
efde8fd4e7
Refactor BaseDataSource and unified handling of JDBC resource releases. ( #2184 )
...
* Refactor BaseDataSource.isConnectable().
* Add BaseDataSource test.
* Unified handling of JDBC resource releases.
* Fix sonar check error.
* Remove useless logger.
* Refactor BaseDataSource.getJdbcUrl().
* Modify ConnectionUtils.releaseResource().
* Add javadoc
* Remove useless method.
* Add BaseDataSourceTest UT
5 years ago
f112415b13
TaskManager refactor ( #2302 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* Encapsulate the parameters required by sqltask
* 1，Encapsulate the parameters required by sqltask
2，SQLTask optimization
* AbstractTask modify
* ProcedureTask optimization
* MasterSchedulerService modify
* TaskUpdateQueueConsumer modify
* test
* DataxTask process run debug
* DataxTask process run debug
* add protobuf dependency，MR、Spark task etc need this
* TaskUpdateQueueConsumer modify
* TaskExecutionContextBuilder set TaskInstance workgroup
* WorkerGroupService queryAllGroup modify
query available work group
* 1，get workergroup from zk modify
2，SpringConnectionFactory repeat load modify
* master and worker register ip use OSUtils.getHost()
* ProcessInstance host set ip:port format
* worker fault tolerance modify
* Constants and .env modify
* master fault tolerant bug modify
* UT add pom.xml
* timing online modify
* when taskResponse is faster than taskAck to db，task state will error
add async queue and new a thread reslove this problem
* TaskExecutionContext set host
* 1，TaskManager refactor
2， api start load server dolphinschedule-daemon.sh modify
* 1，TaskManager refactor
2， api start load server dolphinschedule-daemon.sh modify
* add UT in pom.xml
* revert dolphinscheduler-daemon.sh
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
5 years ago
Simon
BoYiZhang
simon824
eights
bao liang
CalvinKirs
lgcareer
qiaozhanwei
gabry.wu
xiaozhaoying
kezhenxu94
hgaol
zixi0825
dailidong
qiaozhanwei
liwenhe1993
t1mon