Browse Source

update

data_quality_design
dailidong 4 years ago
parent
commit
ce330ca0ef
  1. 16
      README.md
  2. 24
      README_zh_CN.md
  3. 6
      dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/MasterServer.java
  4. 2
      dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterBaseTaskExecThread.java
  5. 8
      dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterExecThread.java
  6. 2
      dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterSchedulerThread.java
  7. 2
      dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterTaskExecThread.java
  8. 4
      dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/quartz/DruidConnectionProvider.java
  9. 3
      dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/quartz/QuartzExecutors.java
  10. 6
      dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/rpc/LogClient.java
  11. 4
      dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/utils/ProcessUtils.java
  12. 9
      dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/FetchTaskThread.java

16
README.md

@ -58,25 +58,25 @@ Overload processing: Task queue mechanism, the number of schedulable tasks on a
### Document
- <a href="https://dolphinscheduler.apache.org/en-us/docs/user_doc/backend-deployment.html" target="_blank">Backend deployment documentation</a>
- <a href="https://dolphinscheduler.apache.org/en-us/docs/1.2.0/user_doc/backend-deployment.html" target="_blank">Backend deployment documentation</a>
- <a href="https://dolphinscheduler.apache.org/en-us/docs/user_doc/frontend-deployment.html" target="_blank">Front-end deployment documentation</a>
- <a href="https://dolphinscheduler.apache.org/en-us/docs/1.2.0/user_doc/frontend-deployment.html" target="_blank">Front-end deployment documentation</a>
- [**User manual**](https://dolphinscheduler.apache.org/en-us/docs/user_doc/system-manual.html?_blank "System manual")
- [**User manual**](https://dolphinscheduler.apache.org/en-us/docs/1.2.0/user_doc/system-manual.html?_blank "System manual")
- [**Upgrade document**](https://dolphinscheduler.apache.org/en-us/docs/release/upgrade.html?_blank "Upgrade document")
- [**Upgrade document**](https://dolphinscheduler.apache.org/en-us/docs/1.2.0/release/upgrade.html?_blank "Upgrade document")
- <a href="http://106.75.43.194:8888" target="_blank">Online Demo</a>
More documentation please refer to <a href="https://dolphinscheduler.apache.org/en-us/docs/user_doc/quick-start.html" target="_blank">[DolphinScheduler online documentation]</a>
More documentation please refer to <a href="https://dolphinscheduler.apache.org/en-us/docs/1.2.0/user_doc/quick-start.html" target="_blank">[DolphinScheduler online documentation]</a>
### Recent R&D plan
Work plan of Dolphin Scheduler: [R&D plan](https://github.com/apache/incubator-dolphinscheduler/projects/1), Under the `In Develop` card is what is currently being developed, TODO card is to be done (including feature ideas)
### How to contribute code
### How to contribute
Welcome to participate in contributing code, please refer to the process of submitting the code:
[[How to contribute code](https://github.com/apache/incubator-dolphinscheduler/issues/310)]
Welcome to participate in contributing, please refer to the process of submitting the code:
[[How to contribute](https://dolphinscheduler.apache.org/en-us/docs/development/contribute.html)]
### How to Build

24
README_zh_CN.md

@ -44,27 +44,27 @@ Dolphin Scheduler Official Website
### 文档
- <a href="https://dolphinscheduler.apache.org/zh-cn/docs/user_doc/backend-deployment.html" target="_blank">后端部署文档</a>
- <a href="https://dolphinscheduler.apache.org/zh-cn/docs/1.2.0/user_doc/backend-deployment.html" target="_blank">后端部署文档</a>
- <a href="https://dolphinscheduler.apache.org/zh-cn/docs/user_doc/frontend-deployment.html" target="_blank">前端部署文档</a>
- <a href="https://dolphinscheduler.apache.org/zh-cn/docs/1.2.0/user_doc/frontend-deployment.html" target="_blank">前端部署文档</a>
- [**使用手册**](https://dolphinscheduler.apache.org/zh-cn/docs/user_doc/system-manual.html?_blank "系统使用手册")
- [**使用手册**](https://dolphinscheduler.apache.org/zh-cn/docs/1.2.0/user_doc/system-manual.html?_blank "系统使用手册")
- [**升级文档**](https://dolphinscheduler.apache.org/zh-cn/docs/release/upgrade.html?_blank "升级文档")
- [**升级文档**](https://dolphinscheduler.apache.org/zh-cn/docs/1.2.0/release/upgrade.html?_blank "升级文档")
- <a href="http://106.75.43.194:8888" target="_blank">我要体验</a>
更多文档请参考 <a href="https://dolphinscheduler.apache.org/zh-cn/docs/user_doc/quick-start.html" target="_blank">DolphinScheduler中文在线文档</a>
更多文档请参考 <a href="https://dolphinscheduler.apache.org/zh-cn/docs/1.2.0/user_doc/quick-start.html" target="_blank">DolphinScheduler中文在线文档</a>
### 近期研发计划
DolphinScheduler的工作计划:<a href="https://github.com/apache/incubator-dolphinscheduler/projects/1" target="_blank">研发计划</a> ,其中 In Develop卡片下是正在研发的功能,TODO卡片是待做事项(包括 feature ideas)
### 贡献代码
### 参与贡献
非常欢迎大家来参与贡献代码,提交代码流程请参考:
[[How to contribute code](https://github.com/apache/incubator-dolphinscheduler/issues/310)]
非常欢迎大家来参与贡献,贡献流程请参考:
[[参与贡献](https://dolphinscheduler.apache.org/zh-cn/docs/development/contribute.html)]
### How to Build
@ -88,12 +88,12 @@ Dolphin Scheduler使用了很多优秀的开源项目,比如google的guava、g
### 获得帮助
1. Submit an issue
1. Mail to dev-subscribe@dolphinscheduler.apache.org, follow the reply to subscribe the mail list. then you can send mail to dev@dolphinscheduler.apache.org.
1. Contact WeChat group manager, ID 510570367. This is for Mandarin(CN) discussion.
1. 提交issue
1. 先订阅邮件开发列表:[订阅邮件列表](https://dolphinscheduler.apache.org/zh-cn/docs/1.2.0/user_doc/subscribe.html), 订阅成功后发送邮件到dev@dolphinscheduler.apache.org.
1. 联系微信群助手(ID:dailidong66). 微信仅用于中国用户讨论.
### 版权
Please refer to [LICENSE](https://github.com/apache/incubator-dolphinscheduler/blob/dev/LICENSE) file.
请参考 [LICENSE](https://github.com/apache/incubator-dolphinscheduler/blob/dev/LICENSE) 文件.

6
dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/MasterServer.java

@ -141,7 +141,7 @@ public class MasterServer implements IStoppable {
} catch (SchedulerException e1) {
logger.error("QuartzExecutors shutdown failed : " + e1.getMessage(), e1);
}
logger.error("start Quartz failed : " + e.getMessage(), e);
logger.error("start Quartz failed", e);
}
@ -183,7 +183,7 @@ public class MasterServer implements IStoppable {
//thread sleep 3 seconds for thread quitely stop
Thread.sleep(3000L);
}catch (Exception e){
logger.warn("thread sleep exception:" + e.getMessage(), e);
logger.warn("thread sleep exception ", e);
}
try {
heartbeatMasterService.shutdownNow();
@ -228,7 +228,7 @@ public class MasterServer implements IStoppable {
} catch (Exception e) {
logger.error("master server stop exception : " + e.getMessage(), e);
logger.error("master server stop exception ", e);
System.exit(-1);
}
}

2
dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterBaseTaskExecThread.java

@ -140,7 +140,7 @@ public class MasterBaseTaskExecThread implements Callable<Boolean> {
}
Thread.sleep(commitRetryInterval);
} catch (Exception e) {
logger.error("task commit to mysql and queue failed : " + e.getMessage(),e);
logger.error("task commit to mysql and queue failed",e);
}
retryTimes += 1;
}

8
dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterExecThread.java

@ -170,7 +170,7 @@ public class MasterExecThread implements Runnable {
executeProcess();
}
}catch (Exception e){
logger.error("master exec thread exception: " + e.getMessage(), e);
logger.error("master exec thread exception", e);
logger.error("process execute failed, process id:{}", processInstance.getId());
processInstance.setState(ExecutionStatus.FAILURE);
processInstance.setEndTime(new Date());
@ -355,7 +355,7 @@ public class MasterExecThread implements Runnable {
try {
FileUtils.deleteDirectory(new File(execLocalPath));
} catch (IOException e) {
logger.error("delete exec dir failed : " + e.getMessage(), e);
logger.error("delete exec dir failed ", e);
}
}
}
@ -733,7 +733,7 @@ public class MasterExecThread implements Runnable {
Date endTime = DateUtils.getScheduleDate(cmdParam.get(CMDPARAM_COMPLEMENT_DATA_END_DATE));
return processInstance.getScheduleTime().equals(endTime);
} catch (Exception e) {
logger.error("complement end failed : " + e.getMessage(),e);
logger.error("complement end failed ",e);
return false;
}
}
@ -1005,7 +1005,7 @@ public class MasterExecThread implements Runnable {
return task;
}
}catch (Exception e){
logger.error("get recovery task instance failed : " + e.getMessage(),e);
logger.error("get recovery task instance failed ",e);
}
return null;
}

2
dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterSchedulerThread.java

@ -134,7 +134,7 @@ public class MasterSchedulerThread implements Runnable {
}
}
}catch (Exception e){
logger.error("master scheduler thread exception : " + e.getMessage(),e);
logger.error("master scheduler thread exception",e);
}finally{
AbstractZKClient.releaseMutex(mutex);
}

2
dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterTaskExecThread.java

@ -137,7 +137,7 @@ public class MasterTaskExecThread extends MasterBaseTaskExecThread {
processInstance = processDao.findProcessInstanceById(processInstance.getId());
Thread.sleep(Constants.SLEEP_TIME_MILLIS);
} catch (Exception e) {
logger.error("exception: "+ e.getMessage(),e);
logger.error("exception",e);
if (processInstance != null) {
logger.error("wait task quit failed, instance id:{}, task id:{}",
processInstance.getId(), taskInstance.getId());

4
dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/quartz/DruidConnectionProvider.java

@ -18,9 +18,9 @@ package org.apache.dolphinscheduler.server.quartz;
import com.alibaba.druid.pool.DruidDataSource;
import org.quartz.SchedulerException;
import org.quartz.utils.ConnectionProvider;
import java.sql.Connection;
import java.sql.SQLException;
import org.quartz.utils.ConnectionProvider;
/**
* druid connection provider
@ -137,7 +137,7 @@ public class DruidConnectionProvider implements ConnectionProvider {
datasource.setDriverClassName(this.driver);
} catch (Exception e) {
try {
throw new SchedulerException("Problem setting driver class name on datasource: " + e.getMessage(), e);
throw new SchedulerException("Problem setting driver class name on datasource", e);
} catch (SchedulerException e1) {
}
}

3
dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/quartz/QuartzExecutors.java

@ -27,7 +27,6 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.*;
import java.util.Calendar;
import java.util.concurrent.locks.ReadWriteLock;
import java.util.concurrent.locks.ReentrantReadWriteLock;
@ -198,7 +197,7 @@ public class QuartzExecutors {
} catch (Exception e) {
logger.error("add job failed", e);
throw new RuntimeException("add job failed:"+e.getMessage());
throw new RuntimeException("add job failed", e);
} finally {
lock.writeLock().unlock();
}

6
dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/rpc/LogClient.java

@ -102,7 +102,7 @@ public class LogClient {
retStrInfo = blockingStub.rollViewLog(pathParameter);
return retStrInfo.getMsg();
} catch (StatusRuntimeException e) {
logger.error("roll view log failed : " + e.getMessage(), e);
logger.error("roll view log failed", e);
return null;
}
}
@ -122,7 +122,7 @@ public class LogClient {
retStrInfo = blockingStub.viewLog(pathParameter);
return retStrInfo.getMsg();
} catch (StatusRuntimeException e) {
logger.error("view log failed : " + e.getMessage(), e);
logger.error("view log failed", e);
return null;
}
}
@ -142,7 +142,7 @@ public class LogClient {
retByteInfo = blockingStub.getLogBytes(pathParameter);
return retByteInfo.getData().toByteArray();
} catch (StatusRuntimeException e) {
logger.error("get log bytes failed : " + e.getMessage(), e);
logger.error("get log bytes failed ", e);
return null;
}
}

4
dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/utils/ProcessUtils.java

@ -308,7 +308,7 @@ public class ProcessUtils {
Runtime.getRuntime().exec(runCmd);
} catch (Exception e) {
logger.error("kill application failed : " + e.getMessage(), e);
logger.error("kill application failed", e);
}
}
}
@ -337,7 +337,7 @@ public class ProcessUtils {
killYarnJob(taskInstance);
} catch (Exception e) {
logger.error("kill failed : " + e.getMessage(), e);
logger.error("kill task failed", e);
}
}

9
dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/FetchTaskThread.java

@ -186,6 +186,10 @@ public class FetchTaskThread implements Runnable{
continue;
}
if(!checkWorkerGroup(taskInstance, OSUtils.getHost())){
continue;
}
Tenant tenant = processDao.getTenantForProcess(taskInstance.getProcessInstance().getTenantId(),
taskInstance.getProcessDefine().getUserId());
@ -203,11 +207,6 @@ public class FetchTaskThread implements Runnable{
logger.info("worker fetch taskId : {} from queue ", taskInstId);
if(!checkWorkerGroup(taskInstance, OSUtils.getHost())){
continue;
}
// local execute path
String execLocalPath = getExecLocalPath();

Loading…
Cancel
Save