743294668
a58d1b3b45
Modify OracleDB connection string format. ( #2970 )
...
Co-authored-by: taoyuxin <taoyuxin@inspur.com>
4 years ago
Simon
2e4adbe03f
Update pom.xml
4 years ago
Simon
6c5b0706f5
Update pom.xml
4 years ago
Simon
7f725c32b5
Update pom.xml
4 years ago
BoYiZhang
f56ae9e64a
add http task timeout & test unit ( #2955 )
...
* add http task timeout & test unit
* add http task timeout & test unit
Co-authored-by: zhanglong <zhanglong@ysstech.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
4 years ago
simon824
65841a5709
1. change windows new line 'CR' to linux new line 'LF'
...
2. solve conflict
4 years ago
Simon
71619ed29b
Update pom.xml
4 years ago
Simon
b5a296f4cd
Update pom.xml
4 years ago
Simon
094d09ba74
Update pom.xml
4 years ago
eights
822addac30
add sqoop task UT to pom
4 years ago
simon824
fad09075f9
remove fastjson maven dependence
4 years ago
simon824
6e907f150c
1. specified jackson Date format.
...
2. remove getMapper method,replace with toList()
3. rewrite toList method,support constructCollectionType.
4. parseObject(resp, Map.class) replace to toMap() method
5. rename JacksonSerialize to JsonSerializer
rewrite createArrayNode,createObjectNode method will Affect too many lines ,i will fix it in the next pr.
4 years ago
CalvinKirs
b21259d600
add MicroBenchMark model ( #2851 )
...
* add MicroBenchMark model
* add log
* fix propertyName
* small change
* add ut
* format
* add @Test
* fix maven compile error
* remove enum set method
* exclusion microbench ut
* small change
* Use assembly instead of shade
Co-authored-by: dailidong <dailidong66@gmail.com>
4 years ago
simon
4a10157506
ut fix
4 years ago
gabry.wu
c28a697ee8
add RetryerUtils to retry async & sync function gracefully ( #2697 )
...
* add RetryerUtils tool class for async & sync retry invoking
* add RetryerUtils tool class for async & sync retry invoking
* add guava-retrying-2.0.0.jar to known-dependencies.txt
* fix javadoc
* fix code smell
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
4 years ago
xiaozhaoying
de3a81cab5
support ali nail(ding talk) , issue:2458 ( #2594 )
...
* Create TextMessage.java
* Create DingTalkUtils.java
* Update alert.properties
* Update Constants.java
* Update Constants.java
* Create DingTalkMsgFormatter.java
* Update alert.properties
* Create DingTalkMsgFormatterTest.java
* Create DingTalkUtilsTest.java
* Update DingTalkUtils.java
* Create DingTalkManager.java
* Update EmailAlertPlugin.java
* feature: test send msg with proxy
* fix:delete comment and add exception info
* Update DingTalkUtilsTest.java
* Update DingTalkUtils.java
* Update DingTalkManager.java
* Update EmailAlertPlugin.java
* Update DingTalkUtilsTest.java
* Update DingTalkMsgFormatter.java
* Update DingTalkUtils.java
* Update DingTalkMsgFormatterTest.java
* Update DingTalkUtilsTest.java
* Update TextMessage.java
* Update DingTalkManager.java
* Delete TextMessage.java
* fix: add test case
* fix:add http client test case
* fix:delete ignore test case
* Update pom.xml
Co-authored-by: dailidong <dailidong66@gmail.com>
4 years ago
kezhenxu94
6e08b29254
Add a script to check the license ( #2552 )
...
* Add a script to check the license
### Motivation
Check licenses in the distribution package
### Modification
1. Add a script to check licenses, and list known licenses.
1. Remove unused dep license, lombok.
1. Remove unnecessary ignored patterns in .gitignore.
1. Add missing config items to apache-rat plugin, which just checked a small part of files before.
1. Add check to GitHub Actions process
### Result
1. Newly-added dependencies should be checked and confirmed.
1. Closes #1578
* Separate rat and dependency check
* Tee dependencies to stdout
* Add more debug messages
* Check light-weight task first
4 years ago
hgaol
906ffa09a9
resolve code checks
4 years ago
hgaol
85f3ac10a0
feat: add plugin management for alert service
4 years ago
qiaozhanwei
a12103b08c
no valid worker group,master can kill task directly ( #2541 )
...
* dispatch task fail will set task status failed
* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result
* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result
* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result
* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result
* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result
* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result
* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result
* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result
* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result
* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result
* 1,no worker condition , master will while ture wait for worker startup
2,worker response task status sync wait for result
* 1,task status statistics and process status statistics bug fix (#2357 )
2,worker group bug fix
* 1,task status statistics and process status statistics bug fix (#2357 )
2,worker group bug fix
* 1,task status statistics and process status statistics bug fix (#2357 )
2,worker group bug fix
* 1,task status statistics and process status statistics bug fix (#2357 )
2,worker group bug fix
* send mail error, #2466 bug fix
* send mail error, #2466 bug fix
* send mail error, #2466 bug fix
* send mail error, #2466 bug fix
* #2486 bug fix
* host and workergroup compatible
* EnterpriseWeChatUtils modify
* EnterpriseWeChatUtils modify
* EnterpriseWeChatUtils modify
* #2499 bug fix
* add comment
* revert comment
* revert comment
* #2499 buf fix
* #2499 bug fix
* #2499 bug fix
* #2499 bug fix
* #2499 bug fix
* #2499 bug fix
* #2499 bug fix
* no valid worker group,master can kill task directly
* no valid worker group,master can kill task directly
* no valid worker group,master can kill task directly
* no valid worker group,master can kill task directly
* no valid worker group,master can kill task directly
* no valid worker group,master can kill task directly
* no valid worker group,master can kill task directly
* no valid worker group,master can kill task directly
* no valid worker group,master can kill task directly
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
4 years ago
zixi0825
64d335d9f2
copy process feature #2412 ( #2471 )
...
* Update pom.xml (#2467 )
* add copy process
* add copy process 2
* add copy process 3
* add copy process 4
* add copy process 5
Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
4 years ago
lgcareer
44770a8767
[ fix #2442 ] It will give a error tip that resource not exist exwhen update resources. ( #2483 )
...
* fix #2442 and remove unavailable code
* revert verifyResourceName method
* Add ServiceException
* add ServiceExceptionTest
* update ServiceExceptionTest
* add ServiceExceptionTest in pom
4 years ago
dailidong
eec92ed1fc
simplify and optimize config ( #2469 )
...
* [Refactor worker] simplify and optimize config (#2386 )
* simplify config
* simplify config
* simplify and optimize config
* Update HadoopUtils.java
optimize HadoopUtils
* Update HadoopUtilsTest.java
* Update HadoopUtilsTest.java
* Update HadoopUtilsTest.java
* Update HttpUtils.java
* Update pom.xml
* Update HadoopUtilsTest.java
* Update HadoopUtilsTest.java
* Update HadoopUtilsTest.java
* Update HadoopUtilsTest.java
* Update HadoopUtilsTest.java
* Update HadoopUtilsTest.java
4 years ago
dailidong
0b405b6fc8
Update pom.xml
4 years ago
dailidong
dc36c77ba7
Update pom.xml ( #2467 )
4 years ago
dailidong
0f54149ae3
exclude HadoopUtils Test ( #2465 )
...
* Update pom.xml
* Update pom.xml
4 years ago
bao liang
4946e88872
move dependent and conditions to master server ( #2423 )
...
* move condition and dependent to master
* move conditions to master
* move conditions and dependent task to master
* move conditions to master
* update test
* add log
* add test for dependent task
* add test for dependent task
* update
* update
* refactor complexity code
* refactor complexity code
* add conditions task test
* add conditions task test
* update
* update host to host:port
* update logback.xml
Co-authored-by: baoliang <baoliang@analysys.com.cn>
4 years ago
zixi0825
ca66cfec17
solve code spell problem ( #2419 )
...
* solve code spell problem
* remove controller test
* add controller test
* sovle spell problem and sovle conflict
* sovle spell problem and sovle conflict
* sovle spell problem and add test
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
4 years ago
Han Gao
3c9ba0a3f3
api server exception management and code optimization ( #397 ) ( #2397 )
...
* api server exception management and code optimization (#397 )
* add unit test for exception handler.
* fix some style and naming issues
Co-authored-by: dailidong <dailidong66@gmail.com>
4 years ago
qiaozhanwei
ff535e766d
update UT in pom.xml
4 years ago
qiaozhanwei
9d06fe7755
update UT in pom.xml
4 years ago
qiaozhanwei
b2c92deace
update UT in pom.xml
4 years ago
qiaozhanwei
66d587c40d
update UT in pom.xml
4 years ago
qiaozhanwei
b1a773f341
update UT in pom.xml
4 years ago
qiaozhanwei
9de9f18c08
update UT in pom.xml
4 years ago
qiaozhanwei
4160c21d8f
update UT in pom.xml
4 years ago
qiaozhanwei
b581f02a21
add dao other mapper add in pom.xml
4 years ago
qiaozhanwei
f02e7e8b37
pom modify
4 years ago
qiaozhanwei
f65ab56d2d
UT modify
4 years ago
liwenhe1993
9dfc93e0a2
Exclude .mvn/jvm.config and .mvn/wrapper/maven-wrapper.properties ( #2387 )
...
* Remove .helmignore file
* Exclude .mvn/jvm.config and .mvn/wrapper/maven-wrapper.properties
When running rat, exclude .mvn/jvm.config and .mvn/wrapper/maven-wrapper.properties
4 years ago
qiaozhanwei
af2b3187b9
update pom.xml
4 years ago
qiaozhanwei
433b41dd7f
refactor-worker merge to dev bug fix
4 years ago
qiaozhanwei
94713cef79
refactor-worker merge to dev bug fix
4 years ago
zixi0825
69e000b542
#2282 fix workflow dependent bug ( #2329 )
...
* fix workflow dependent bug
* fix workflow dependent bug 2
* fix workflow dependent bug 2
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
4 years ago
liwenhe1993
526e5c91f1
Connection mode of adding Sid to Oracle ( #2254 )
...
* Connection mode of adding Sid to Oracle
* Remove code
* Add asf
* Add unit test
* Add unit test
* Add unit test
* solve the conflict
4 years ago
qiaozhanwei
68cb81fdf5
1,add UT in pom 2,refactor TaskUpdateQueue ( #2326 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* Encapsulate the parameters required by sqltask
* 1,Encapsulate the parameters required by sqltask
2,SQLTask optimization
* AbstractTask modify
* ProcedureTask optimization
* MasterSchedulerService modify
* TaskUpdateQueueConsumer modify
* test
* DataxTask process run debug
* DataxTask process run debug
* add protobuf dependency,MR、Spark task etc need this
* TaskUpdateQueueConsumer modify
* TaskExecutionContextBuilder set TaskInstance workgroup
* WorkerGroupService queryAllGroup modify
query available work group
* 1,get workergroup from zk modify
2,SpringConnectionFactory repeat load modify
* master and worker register ip use OSUtils.getHost()
* ProcessInstance host set ip:port format
* worker fault tolerance modify
* Constants and .env modify
* master fault tolerant bug modify
* UT add pom.xml
* timing online modify
* when taskResponse is faster than taskAck to db,task state will error
add async queue and new a thread reslove this problem
* TaskExecutionContext set host
* 1,TaskManager refactor
2, api start load server dolphinschedule-daemon.sh modify
* 1,TaskManager refactor
2, api start load server dolphinschedule-daemon.sh modify
* add UT in pom.xml
* revert dolphinscheduler-daemon.sh
* ZookeeperRegister use common.properties zookeeperRoot path
* api start exclude org.apache.dolphinscheduler.server.*
* ZookeeperRegister use common.properties zookeeperRoot path
* 1,api start load server filter
2,SHELL task exitStatusCode modify
* java doc error modify
* java doc error modify
* remove todo
* add UT in pom
* 1,add UT in pom
2,refactor TaskUpdateQueue
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
4 years ago
t1mon
efde8fd4e7
Refactor BaseDataSource and unified handling of JDBC resource releases. ( #2184 )
...
* Refactor BaseDataSource.isConnectable().
* Add BaseDataSource test.
* Unified handling of JDBC resource releases.
* Fix sonar check error.
* Remove useless logger.
* Refactor BaseDataSource.getJdbcUrl().
* Modify ConnectionUtils.releaseResource().
* Add javadoc
* Remove useless method.
* Add BaseDataSourceTest UT
4 years ago
qiaozhanwei
f112415b13
TaskManager refactor ( #2302 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* Encapsulate the parameters required by sqltask
* 1,Encapsulate the parameters required by sqltask
2,SQLTask optimization
* AbstractTask modify
* ProcedureTask optimization
* MasterSchedulerService modify
* TaskUpdateQueueConsumer modify
* test
* DataxTask process run debug
* DataxTask process run debug
* add protobuf dependency,MR、Spark task etc need this
* TaskUpdateQueueConsumer modify
* TaskExecutionContextBuilder set TaskInstance workgroup
* WorkerGroupService queryAllGroup modify
query available work group
* 1,get workergroup from zk modify
2,SpringConnectionFactory repeat load modify
* master and worker register ip use OSUtils.getHost()
* ProcessInstance host set ip:port format
* worker fault tolerance modify
* Constants and .env modify
* master fault tolerant bug modify
* UT add pom.xml
* timing online modify
* when taskResponse is faster than taskAck to db,task state will error
add async queue and new a thread reslove this problem
* TaskExecutionContext set host
* 1,TaskManager refactor
2, api start load server dolphinschedule-daemon.sh modify
* 1,TaskManager refactor
2, api start load server dolphinschedule-daemon.sh modify
* add UT in pom.xml
* revert dolphinscheduler-daemon.sh
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
5 years ago
qiaozhanwei
052f9d10bf
UT Coverage rate test ( #2276 )
...
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* 1, master persistent task
2. extract master and worker communication model
* add license
* modify javadoc error
* TaskExecutionContext create modify
* buildAckCommand taskInstanceId not set modify
* java doc error modify
* add comment
* ExecutorManager interface add generic type
* add TaskInstanceCacheManager receive Worker report result
* TaskInstance setExecutePath
* add TaskInstanceCacheManager to receive Worker Task result report
* TaskInstanceCacheManager add remove method
* add license
* add dispatcht task method
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* AbstractCommandExecutor remove db access
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* taskInstanceCache is null ,need load from db
* 1,worker TaskPros use TaskExecutionContext replase
2,Master kill Task , KillTaskProcessor modify
* worker remove db
* ShellTask modify
* master persistence processId and appIds
* master persistence processId and appIds
* master add kill task logic
* master add kill task logic
* master add kill task logic
* javadoc error modify
* remove chinese log
* executeDirectly method add Override
* remote module modify
* TaskKillResponseProcessor command type modify
* create buildKillCommand
* host add host:port format
* host add host:port format
* TaskAckProcessor modify
* TaskAckProcessor modify
* task prioriry refator
* remove ITaskQueue
* task prioriry refator
* remove ITaskQueue
* TaskPriority refactor
* remove logs
* WorkerServer refactor
* MasterSchedulerService modify
* WorkerConfig listen port modify
* modify master and worker listen port
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* cancelTaskInstance set TaskExecutionContext host,logPath,executePath
* Encapsulate the parameters required by sqltask
* 1,Encapsulate the parameters required by sqltask
2,SQLTask optimization
* AbstractTask modify
* ProcedureTask optimization
* MasterSchedulerService modify
* TaskUpdateQueueConsumer modify
* test
* DataxTask process run debug
* DataxTask process run debug
* add protobuf dependency,MR、Spark task etc need this
* TaskUpdateQueueConsumer modify
* TaskExecutionContextBuilder set TaskInstance workgroup
* WorkerGroupService queryAllGroup modify
query available work group
* 1,get workergroup from zk modify
2,SpringConnectionFactory repeat load modify
* master and worker register ip use OSUtils.getHost()
* ProcessInstance host set ip:port format
* worker fault tolerance modify
* Constants and .env modify
* master fault tolerant bug modify
* UT add pom.xml
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
5 years ago
khadgarmage
cddf803ba0
license and checkoutv2 fix ( #2261 )
5 years ago