Browse Source

update code

pull/3/MERGE
dailidong 5 years ago
parent
commit
c6d1557a2b
  1. 4
      README.md
  2. 7
      README_zh_CN.md
  3. 2
      dockerfile/conf/dolphinscheduler/conf/quartz.properties
  4. 34
      dolphinscheduler-api/pom.xml
  5. 8
      dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/ExecutorService.java
  6. 5
      dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/SchedulerService.java
  7. 17
      dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/model/DateInterval.java
  8. 31
      dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/queue/TaskQueueZkImpl.java
  9. 3
      dolphinscheduler-common/src/main/resources/quartz.properties
  10. 2
      dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/quartz/DruidConnectionProvider.java
  11. 2
      dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/quartz/ProcessScheduleJob.java
  12. 2
      dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/quartz/QuartzExecutors.java
  13. 4
      dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/MasterServer.java
  14. 14
      dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterExecThread.java
  15. 3
      dolphinscheduler-ui/src/js/conf/home/pages/resource/pages/udf/pages/function/_source/createUdf.vue
  16. 3
      dolphinscheduler-ui/src/js/conf/home/pages/resource/pages/udf/pages/function/_source/list.vue
  17. 6
      dolphinscheduler-ui/src/js/module/components/transfer/resource.vue
  18. 2
      pom.xml
  19. 2
      sql/upgrade/1.2.0_schema/mysql/dolphinscheduler_dml.sql

4
README.md

@ -64,7 +64,7 @@ Overload processing: Task queue mechanism, the number of schedulable tasks on a
- [**User manual**](https://dolphinscheduler.apache.org/en-us/docs/1.2.0/user_doc/system-manual.html?_blank "System manual")
- [**Upgrade document**](https://dolphinscheduler.apache.org/en-us/docs/1.2.0/release/upgrade.html?_blank "Upgrade document")
- [**Upgrade document**](https://dolphinscheduler.apache.org/en-us/docs/1.2.0/user_doc/upgrade.html?_blank "Upgrade document")
- <a href="http://106.75.43.194:8888" target="_blank">Online Demo</a>
@ -99,7 +99,7 @@ It is because of the shoulders of these open source projects that the birth of t
### Get Help
1. Submit an issue
1. Subscribe the mail list : https://dolphinscheduler.apache.org/en-us/docs/1.2.0/user_doc/subscribe.html. then send mail to dev@dolphinscheduler.apache.org
1. Subscribe the mail list : https://dolphinscheduler.apache.org/en-us/docs/development/subscribe.html. then send mail to dev@dolphinscheduler.apache.org
1. Contact WeChat group manager, ID 510570367. This is for Mandarin(CN) discussion.
### License

7
README_zh_CN.md

@ -50,7 +50,7 @@ Dolphin Scheduler Official Website
- [**使用手册**](https://dolphinscheduler.apache.org/zh-cn/docs/1.2.0/user_doc/system-manual.html?_blank "系统使用手册")
- [**升级文档**](https://dolphinscheduler.apache.org/zh-cn/docs/1.2.0/release/upgrade.html?_blank "升级文档")
- [**升级文档**](https://dolphinscheduler.apache.org/zh-cn/docs/1.2.0/user_doc/upgrade.html?_blank "升级文档")
- <a href="http://106.75.43.194:8888" target="_blank">我要体验</a>
@ -83,13 +83,12 @@ dolphinscheduler-dist/dolphinscheduler-src/target/apache-dolphinscheduler-incuba
### 感谢
Dolphin Scheduler使用了很多优秀的开源项目,比如google的guava、guice、grpc,netty,ali的bonecp,quartz,以及apache的众多开源项目等等,
正是由于站在这些开源项目的肩膀上,才有Dolphin Scheduler的诞生的可能。对此我们对使用的所有开源软件表示非常的感谢!我们也希望自己不仅是开源的受益者,也能成为开源的
贡献者,于是我们决定把易调度贡献出来,并承诺长期维护。也希望对开源有同样热情和信念的伙伴加入进来,一起为开源献出一份力!
正是由于站在这些开源项目的肩膀上,才有Dolphin Scheduler的诞生的可能。对此我们对使用的所有开源软件表示非常的感谢!我们也希望自己不仅是开源的受益者,也能成为开源的贡献者,也希望对开源有同样热情和信念的伙伴加入进来,一起为开源献出一份力!
### 获得帮助
1. 提交issue
1. 先订阅邮件开发列表:[订阅邮件列表](https://dolphinscheduler.apache.org/zh-cn/docs/1.2.0/user_doc/subscribe.html), 订阅成功后发送邮件到dev@dolphinscheduler.apache.org.
1. 先订阅邮件开发列表:[订阅邮件列表](https://dolphinscheduler.apache.org/zh-cn/docs/development/subscribe.html), 订阅成功后发送邮件到dev@dolphinscheduler.apache.org.
1. 联系微信群助手(ID:dailidong66). 微信仅用于中国用户讨论.
### 版权

2
dockerfile/conf/dolphinscheduler/conf/quartz.properties

@ -47,7 +47,7 @@ org.quartz.jobStore.dataSource = myDs
#============================================================================
# Configure Datasources
#============================================================================
org.quartz.dataSource.myDs.connectionProvider.class = org.apache.dolphinscheduler.server.quartz.DruidConnectionProvider
org.quartz.dataSource.myDs.connectionProvider.class = org.apache.dolphinscheduler.dao.quartz.DruidConnectionProvider
org.quartz.dataSource.myDs.driver = org.postgresql.Driver
org.quartz.dataSource.myDs.URL=jdbc:postgresql://127.0.0.1:5432/dolphinscheduler
org.quartz.dataSource.myDs.user=root

34
dolphinscheduler-api/pom.xml

@ -31,33 +31,21 @@
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-alert</artifactId>
<exclusions>
<exclusion>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-dao</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-dao</artifactId>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-server</artifactId>
<exclusions>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
</exclusion>
<exclusion>
<groupId>com.google</groupId>
<artifactId>netty</artifactId>
</exclusion>
<exclusion>
<artifactId>leveldbjni-all</artifactId>
<groupId>org.fusesource.leveldbjni</groupId>
</exclusion>
<exclusion>
<artifactId>protobuf-java</artifactId>
<groupId>com.google.protobuf</groupId>
</exclusion>
</exclusions>
</dependency>
<!--springboot-->
@ -92,8 +80,6 @@
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-aop</artifactId>

8
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/ExecutorService.java

@ -499,10 +499,6 @@ public class ExecutorService extends BaseService{
}
}
if ( start == null || end == null) {
return 0;
}
if(commandType == CommandType.COMPLEMENT_DATA){
runMode = (runMode == null) ? RunMode.RUN_MODE_SERIAL : runMode;
if(null != start && null != end && start.before(end)){
@ -550,7 +546,9 @@ public class ExecutorService extends BaseService{
}else{
command.setCommandParam(JSONUtils.toJson(cmdParam));
return processDao.createCommand(command);
}
}
return 0;
}
/**

5
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/SchedulerService.java

@ -35,8 +35,8 @@ import org.apache.dolphinscheduler.dao.mapper.ProcessDefinitionMapper;
import org.apache.dolphinscheduler.dao.mapper.ProjectMapper;
import org.apache.dolphinscheduler.dao.mapper.ScheduleMapper;
import org.apache.dolphinscheduler.dao.utils.cron.CronUtils;
import org.apache.dolphinscheduler.server.quartz.ProcessScheduleJob;
import org.apache.dolphinscheduler.server.quartz.QuartzExecutors;
import org.apache.dolphinscheduler.dao.quartz.ProcessScheduleJob;
import org.apache.dolphinscheduler.dao.quartz.QuartzExecutors;
import com.baomidou.mybatisplus.core.metadata.IPage;
import com.baomidou.mybatisplus.extension.plugins.pagination.Page;
import org.quartz.CronExpression;
@ -167,6 +167,7 @@ public class SchedulerService extends BaseService {
processDefinitionMapper.updateById(processDefinition);
putMsg(result, Status.SUCCESS);
result.put("scheduleId", scheduleObj.getId());
return result;
}

17
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/model/DateInterval.java

@ -34,13 +34,16 @@ public class DateInterval {
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
DateInterval that = (DateInterval) o;
return startTime.equals(that.startTime) &&
endTime.equals(that.endTime);
public boolean equals(Object obj) {
if (obj == null || getClass() != obj.getClass()) {
return false;
} else if (this == obj) {
return true;
} else {
DateInterval that = (DateInterval) obj;
return startTime.equals(that.startTime) &&
endTime.equals(that.endTime);
}
}
public Date getStartTime() {

31
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/queue/TaskQueueZkImpl.java

@ -30,7 +30,7 @@ import java.util.*;
/**
* A singleton of a task queue implemented with zookeeper
* tasks queue implemention
* tasks queue implementation
*/
@Service
public class TaskQueueZkImpl implements ITaskQueue {
@ -72,7 +72,7 @@ public class TaskQueueZkImpl implements ITaskQueue {
} catch (Exception e) {
logger.error("get all tasks from tasks queue exception",e);
}
return new ArrayList<>();
return Collections.emptyList();
}
/**
@ -196,11 +196,11 @@ public class TaskQueueZkImpl implements ITaskQueue {
}
}
List<String> taskslist = getTasksListFromTreeSet(tasksNum, taskTreeSet);
List<String> tasksList = getTasksListFromTreeSet(tasksNum, taskTreeSet);
logger.info("consume tasks: {},there still have {} tasks need to be executed", Arrays.toString(taskslist.toArray()), size - taskslist.size());
logger.info("consume tasks: {},there still have {} tasks need to be executed", Arrays.toString(tasksList.toArray()), size - tasksList.size());
return taskslist;
return tasksList;
}else{
Thread.sleep(Constants.SLEEP_TIME_MILLIS);
}
@ -208,7 +208,7 @@ public class TaskQueueZkImpl implements ITaskQueue {
} catch (Exception e) {
logger.error("add task to tasks queue exception",e);
}
return new ArrayList<String>();
return Collections.emptyList();
}
@ -221,15 +221,15 @@ public class TaskQueueZkImpl implements ITaskQueue {
public List<String> getTasksListFromTreeSet(int tasksNum, Set<String> taskTreeSet) {
Iterator<String> iterator = taskTreeSet.iterator();
int j = 0;
List<String> taskslist = new ArrayList<>(tasksNum);
List<String> tasksList = new ArrayList<>(tasksNum);
while(iterator.hasNext()){
if(j++ >= tasksNum){
break;
}
String task = iterator.next();
taskslist.add(getOriginTaskFormat(task));
tasksList.add(getOriginTaskFormat(task));
}
return taskslist;
return tasksList;
}
/**
@ -330,22 +330,13 @@ public class TaskQueueZkImpl implements ITaskQueue {
*/
@Override
public Set<String> smembers(String key) {
Set<String> tasksSet = new HashSet<>();
try {
List<String> list = zookeeperOperator.getChildrenKeys(getTasksPath(key));
for (String task : list) {
tasksSet.add(task);
}
return tasksSet;
return new HashSet<>(list);
} catch (Exception e) {
logger.error("get all tasks from tasks queue exception",e);
}
return tasksSet;
return Collections.emptySet();
}
/**

3
dolphinscheduler-common/src/main/resources/quartz.properties

@ -53,11 +53,12 @@ org.quartz.jobStore.tablePrefix = QRTZ_
org.quartz.jobStore.isClustered = true
org.quartz.jobStore.misfireThreshold = 60000
org.quartz.jobStore.clusterCheckinInterval = 5000
org.quartz.jobStore.acquireTriggersWithinLock=true
org.quartz.jobStore.dataSource = myDs
#============================================================================
# Configure Datasources
#============================================================================
org.quartz.dataSource.myDs.connectionProvider.class = org.apache.dolphinscheduler.server.quartz.DruidConnectionProvider
org.quartz.dataSource.myDs.connectionProvider.class = org.apache.dolphinscheduler.dao.quartz.DruidConnectionProvider
org.quartz.dataSource.myDs.maxConnections = 10
org.quartz.dataSource.myDs.validationQuery = select 1

2
dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/quartz/DruidConnectionProvider.java → dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/quartz/DruidConnectionProvider.java

@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.server.quartz;
package org.apache.dolphinscheduler.dao.quartz;
import com.alibaba.druid.pool.DruidDataSource;
import org.quartz.SchedulerException;

2
dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/quartz/ProcessScheduleJob.java → dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/quartz/ProcessScheduleJob.java

@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.server.quartz;
package org.apache.dolphinscheduler.dao.quartz;
import org.apache.dolphinscheduler.common.Constants;

2
dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/quartz/QuartzExecutors.java → dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/quartz/QuartzExecutors.java

@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.server.quartz;
package org.apache.dolphinscheduler.dao.quartz;
import org.apache.dolphinscheduler.common.Constants;
import org.apache.dolphinscheduler.common.utils.JSONUtils;

4
dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/MasterServer.java

@ -27,8 +27,8 @@ import org.apache.dolphinscheduler.common.utils.StringUtils;
import org.apache.dolphinscheduler.dao.ProcessDao;
import org.apache.dolphinscheduler.server.master.config.MasterConfig;
import org.apache.dolphinscheduler.server.master.runner.MasterSchedulerThread;
import org.apache.dolphinscheduler.server.quartz.ProcessScheduleJob;
import org.apache.dolphinscheduler.server.quartz.QuartzExecutors;
import org.apache.dolphinscheduler.dao.quartz.ProcessScheduleJob;
import org.apache.dolphinscheduler.dao.quartz.QuartzExecutors;
import org.apache.dolphinscheduler.server.zk.ZKMasterClient;
import org.quartz.SchedulerException;
import org.slf4j.Logger;

14
dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterExecThread.java

@ -187,7 +187,7 @@ public class MasterExecThread implements Runnable {
/**
* execute process
* @throws Exception excpetion
* @throws Exception exception
*/
private void executeProcess() throws Exception {
prepareProcess();
@ -197,7 +197,7 @@ public class MasterExecThread implements Runnable {
/**
* execute complement process
* @throws Exception excpetion
* @throws Exception exception
*/
private void executeComplementProcess() throws Exception {
@ -247,7 +247,7 @@ public class MasterExecThread implements Runnable {
// execute process ,waiting for end
runProcess();
// process instace failure ,no more complements
// process instance failure ,no more complements
if(!processInstance.getState().typeIsSuccess()){
logger.info("process {} state {}, complement not completely!",
processInstance.getId(), processInstance.getState());
@ -304,7 +304,7 @@ public class MasterExecThread implements Runnable {
/**
* prepare process parameter
* @throws Exception excpetion
* @throws Exception exception
*/
private void prepareProcess() throws Exception {
// init task queue
@ -332,7 +332,7 @@ public class MasterExecThread implements Runnable {
/**
* generate process dag
* @throws Exception excpetion
* @throws Exception exception
*/
private void buildFlowDag() throws Exception {
recoverNodeIdList = getStartTaskInstanceList(processInstance.getCommandParam());
@ -609,7 +609,7 @@ public class MasterExecThread implements Runnable {
/**
* query task instance by complete state
* @param state state
* @return task isntance list
* @return task instance list
*/
private List<TaskInstance> getCompleteTaskByState(ExecutionStatus state){
List<TaskInstance> resultList = new ArrayList<>();
@ -804,7 +804,7 @@ public class MasterExecThread implements Runnable {
}
/**
* add task to standy list
* add task to standby list
* @param taskInstance task instance
*/
private void addTaskToStandByList(TaskInstance taskInstance){

3
dolphinscheduler-ui/src/js/conf/home/pages/resource/pages/udf/pages/function/_source/createUdf.vue

@ -76,7 +76,8 @@
filterable
v-model="resourceId"
:disabled="isUpdate"
style="width: 200px">
:add-title="true"
style="width: 261px">
<x-option
v-for="city in udfResourceList"
:key="city.id"

3
dolphinscheduler-ui/src/js/conf/home/pages/resource/pages/udf/pages/function/_source/list.vue

@ -142,7 +142,7 @@ v-ps<template>
id: item.id
}).then(res => {
this.$refs[`poptip-${i}`][0].doClose()
this.$emit('_updateList')
this.$emit('on-update')
this.$message.success(res.msg)
}).catch(e => {
this.$refs[`poptip-${i}`][0].doClose()
@ -156,6 +156,7 @@ v-ps<template>
showMask: true,
escClose: true,
className: 'v-modal-custom',
width: '800px',
transitionName: 'opacityp',
render (h) {
return h(mCreateUdf, {

6
dolphinscheduler-ui/src/js/module/components/transfer/resource.vue

@ -17,7 +17,7 @@
<template>
<m-popup :ok-text="$t('Submit')" :nameText="type.name + $t('Authorize')" @ok="_ok" ref="popup">
<template slot="content">
<div class="clearfix transfer-model">
<div class="clearfix transfer-model" style="width: 660px">
<div>
<x-button-group v-model="checkedValue" size="small">
<x-button type="ghost" value="fileResource" @click="_ckFile">{{$t('File resources')}}</x-button>
@ -195,7 +195,7 @@
.transfer-model {
padding: 0 20px;
.select-list-box {
width: 220px;
width: 300px;
float: left;
border: 1px solid #dcdee2;
border-radius: 3px;
@ -237,7 +237,7 @@
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
width: 210px;
width: 290px;
display: inline-block;
}
&:hover {

2
pom.xml

@ -78,8 +78,6 @@
<slf4j.api.version>1.7.5</slf4j.api.version>
<slf4j.log4j12.version>1.7.5</slf4j.log4j12.version>
<commons.collections.version>3.2.2</commons.collections.version>
<commons.lang.version>2.3</commons.lang.version>
<commons.lang3.version>3.5</commons.lang3.version>
<commons.httpclient>3.0.1</commons.httpclient>
<commons.beanutils.version>1.7.0</commons.beanutils.version>
<commons.configuration.version>1.10</commons.configuration.version>

2
sql/upgrade/1.2.0_schema/mysql/dolphinscheduler_dml.sql

@ -20,6 +20,6 @@ UPDATE QRTZ_CRON_TRIGGERS SET SCHED_NAME='DolphinScheduler' WHERE SCHED_NAME='Ea
UPDATE QRTZ_TRIGGERS SET SCHED_NAME='DolphinScheduler' WHERE SCHED_NAME='EasyScheduler';
UPDATE QRTZ_FIRED_TRIGGERS SET SCHED_NAME='DolphinScheduler' WHERE SCHED_NAME='EasyScheduler';
UPDATE QRTZ_JOB_DETAILS SET SCHED_NAME='DolphinScheduler' WHERE SCHED_NAME='EasyScheduler';
UPDATE QRTZ_JOB_DETAILS SET JOB_CLASS_NAME='org.apache.dolphinscheduler.server.quartz.ProcessScheduleJob' WHERE JOB_CLASS_NAME='cn.escheduler.server.quartz.ProcessScheduleJob';
UPDATE QRTZ_JOB_DETAILS SET JOB_CLASS_NAME='org.apache.dolphinscheduler.dao.quartz.ProcessScheduleJob' WHERE JOB_CLASS_NAME='cn.escheduler.server.quartz.ProcessScheduleJob';
UPDATE QRTZ_LOCKS SET SCHED_NAME='DolphinScheduler' WHERE SCHED_NAME='EasyScheduler';
UPDATE QRTZ_SCHEDULER_STATE SET SCHED_NAME='DolphinScheduler' WHERE SCHED_NAME='EasyScheduler';
Loading…
Cancel
Save