diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md index be3c583f19..dc797cf0a8 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -26,7 +26,7 @@ A clear and concise description of what you expected to happen. If applicable, add screenshots to help explain your problem. -**Which version of Easy Scheduler:** +**Which version of Dolphin Scheduler:** -[1.1.0-preview] **Additional context** diff --git a/.github/ISSUE_TEMPLATE/question.md b/.github/ISSUE_TEMPLATE/question.md new file mode 100644 index 0000000000..59c8ce7716 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/question.md @@ -0,0 +1,23 @@ +--- +name: question +about: have a question wanted to be help +title: "[QUESTION] question title" +labels: question +assignees: '' + +--- + +*For better global communication, please give priority to using English description, thx! * + +**Describe the question** +A clear and concise description of what the question is. + + +**Which version of DolphinScheduler:** + -[1.1.0-preview] + +**Additional context** +Add any other context about the problem here. + +**Requirement or improvement +- Please describe about your requirements or improvement suggestions. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000000..8d3f81e711 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,17 @@ +name: CI + +on: [push] + +jobs: + build: + + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v1 + - name: Set up JDK 1.8 + uses: actions/setup-java@v1 + with: + java-version: 1.8 + - name: Build with Maven + run: mvn apache-rat:check --file pom.xml diff --git a/README.md b/README.md index 18e2df8da3..6352bd5f10 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,11 @@ -Easy Scheduler +Dolphin Scheduler ============ [![License](https://img.shields.io/badge/license-Apache%202-4EB1BA.svg)](https://www.apache.org/licenses/LICENSE-2.0.html) -[![Total Lines](https://tokei.rs/b1/github/analysys/EasyScheduler?category=lines)](https://github.com/analysys/EasyScheduler) +[![Total Lines](https://tokei.rs/b1/github/apache/Incubator-DolphinScheduler?category=lines)](https://github.com/apache/Incubator-DolphinScheduler) -> Easy Scheduler for Big Data +> Dolphin Scheduler for Big Data -[![Stargazers over time](https://starchart.cc/analysys/EasyScheduler.svg)](https://starchart.cc/analysys/EasyScheduler) +[![Stargazers over time](https://starchart.cc/apache/incubator-dolphinscheduler.svg)](https://starchart.cc/apache/incubator-dolphinscheduler) [![EN doc](https://img.shields.io/badge/document-English-blue.svg)](README.md) [![CN doc](https://img.shields.io/badge/文档-中文版-blue.svg)](README_zh_CN.md) @@ -32,12 +32,12 @@ Its main objectives are as follows: - There are more waiting partners to explore -### What's in Easy Scheduler +### What's in Dolphin Scheduler Stability | Easy to use | Features | Scalability | -- | -- | -- | -- Decentralized multi-master and multi-worker | Visualization process defines key information such as task status, task type, retry times, task running machine, visual variables and so on at a glance.  |  Support pause, recover operation | support custom task types -HA is supported by itself | All process definition operations are visualized, dragging tasks to draw DAGs, configuring data sources and resources. At the same time, for third-party systems, the api mode operation is provided. | Users on easyscheduler can achieve many-to-one or one-to-one mapping relationship through tenants and Hadoop users, which is very important for scheduling large data jobs. " | The scheduler uses distributed scheduling, and the overall scheduling capability will increase linearly with the scale of the cluster. Master and Worker support dynamic online and offline. +HA is supported by itself | All process definition operations are visualized, dragging tasks to draw DAGs, configuring data sources and resources. At the same time, for third-party systems, the api mode operation is provided. | Users on DolphinScheduler can achieve many-to-one or one-to-one mapping relationship through tenants and Hadoop users, which is very important for scheduling large data jobs. " | The scheduler uses distributed scheduling, and the overall scheduling capability will increase linearly with the scale of the cluster. Master and Worker support dynamic online and offline. Overload processing: Task queue mechanism, the number of schedulable tasks on a single machine can be flexibly configured, when too many tasks will be cached in the task queue, will not cause machine jam. | One-click deployment | Supports traditional shell tasks, and also support big data platform task scheduling: MR, Spark, SQL (mysql, postgresql, hive, sparksql), Python, Procedure, Sub_Process | | @@ -54,36 +54,38 @@ Overload processing: Task queue mechanism, the number of schedulable tasks on a ### Document -- Backend deployment documentation +- Backend deployment documentation -- Front-end deployment documentation +- Front-end deployment documentation -- [**User manual**](https://analysys.github.io/easyscheduler_docs_cn/系统使用手册.html?_blank "User manual") +- [**User manual**](https://dolphinscheduler.apache.org/en-us/docs/user_doc/system-manual.html?_blank "System manual") -- [**Upgrade document**](https://analysys.github.io/easyscheduler_docs_cn/升级文档.html?_blank "Upgrade document") +- [**Upgrade document**](https://dolphinscheduler.apache.org/en-us/docs/release/upgrade.html?_blank "Upgrade document") - Online Demo -More documentation please refer to [EasyScheduler online documentation] +More documentation please refer to [DolphinScheduler online documentation] ### Recent R&D plan -Work plan of Easy Scheduler: [R&D plan](https://github.com/analysys/EasyScheduler/projects/1), where `In Develop` card is the features of 1.1.0 version , TODO card is to be done (including feature ideas) +Work plan of Dolphin Scheduler: [R&D plan](https://github.com/apache/incubator-dolphinscheduler/projects/1), Under the `In Develop` card is what is currently being developed, TODO card is to be done (including feature ideas) ### How to contribute code Welcome to participate in contributing code, please refer to the process of submitting the code: -[[How to contribute code](https://github.com/analysys/EasyScheduler/issues/310)] +[[How to contribute code](https://github.com/apache/incubator-dolphinscheduler/issues/310)] ### Thanks -Easy Scheduler uses a lot of excellent open source projects, such as google guava, guice, grpc, netty, ali bonecp, quartz, and many open source projects of apache, etc. -It is because of the shoulders of these open source projects that the birth of the Easy Scheduler is possible. We are very grateful for all the open source software used! We also hope that we will not only be the beneficiaries of open source, but also be open source contributors, so we decided to contribute to easy scheduling and promised long-term updates. We also hope that partners who have the same passion and conviction for open source will join in and contribute to open source! +Dolphin Scheduler uses a lot of excellent open source projects, such as google guava, guice, grpc, netty, ali bonecp, quartz, and many open source projects of apache, etc. +It is because of the shoulders of these open source projects that the birth of the Dolphin Scheduler is possible. We are very grateful for all the open source software used! We also hope that we will not only be the beneficiaries of open source, but also be open source contributors. We also hope that partners who have the same passion and conviction for open source will join in and contribute to open source! ### Get Help -The fastest way to get response from our developers is to submit issues, or add our wechat : 510570367 +1. Submit an issue +1. Mail list: dev@dolphinscheduler.apache.org. Mail to dev-subscribe@dolphinscheduler.apache.org, follow the reply to subscribe the mail list. +1. Contact WeChat group manager, ID 510570367. This is for Mandarin(CN) discussion. ### License -Please refer to [LICENSE](https://github.com/analysys/EasyScheduler/blob/dev/LICENSE) file. +Please refer to [LICENSE](https://github.com/apache/incubator-dolphinscheduler/blob/dev/LICENSE) file. diff --git a/README_zh_CN.md b/README_zh_CN.md index 91cdd04669..f64fcda1f7 100644 --- a/README_zh_CN.md +++ b/README_zh_CN.md @@ -1,9 +1,9 @@ -Easy Scheduler +Dolphin Scheduler ============ [![License](https://img.shields.io/badge/license-Apache%202-4EB1BA.svg)](https://www.apache.org/licenses/LICENSE-2.0.html) [![Total Lines](https://tokei.rs/b1/github/analysys/EasyScheduler?category=lines)](https://github.com/analysys/EasyScheduler) -> Easy Scheduler for Big Data +> Dolphin Scheduler for Big Data @@ -30,9 +30,6 @@ Easy Scheduler - 支持国际化 - 还有更多等待伙伴们探索 -### 与同类调度系统的对比 - -![调度系统对比](http://geek.analysys.cn/static/upload/47/2019-03-01/9609ca82-cf8b-4d91-8dc0-0e2805194747.jpeg) ### 系统部分截图 @@ -44,38 +41,43 @@ Easy Scheduler ### 文档 -- 后端部署文档 +- 后端部署文档 -- 前端部署文档 +- 前端部署文档 -- [**使用手册**](https://analysys.github.io/easyscheduler_docs_cn/系统使用手册.html?_blank "系统使用手册") +- [**使用手册**](https://dolphinscheduler.apache.org/zh-cn/docs/user_doc/system-manual.html?_blank "系统使用手册") -- [**升级文档**](https://analysys.github.io/easyscheduler_docs_cn/升级文档.html?_blank "升级文档") +- [**升级文档**](https://dolphinscheduler.apache.org/zh-cn/docs/release/upgrade.html?_blank "升级文档") - 我要体验 -更多文档请参考 easyscheduler中文在线文档 +更多文档请参考 DolphinScheduler中文在线文档 ### 近期研发计划 -EasyScheduler的工作计划：研发计划 ，其中 In Develop卡片下是1.1.0版本的功能，TODO卡片是待做事项(包括 feature ideas) +DolphinScheduler的工作计划：研发计划 ，其中 In Develop卡片下是正在研发的功能，TODO卡片是待做事项(包括 feature ideas) ### 贡献代码 非常欢迎大家来参与贡献代码，提交代码流程请参考： -[[How to contribute code](https://github.com/analysys/EasyScheduler/issues/310)] +[[How to contribute code](https://github.com/apache/incubator-dolphinscheduler/issues/310)] ### 感谢 -Easy Scheduler使用了很多优秀的开源项目，比如google的guava、guice、grpc，netty，ali的bonecp，quartz，以及apache的众多开源项目等等， +Dolphin Scheduler使用了很多优秀的开源项目，比如google的guava、guice、grpc，netty，ali的bonecp，quartz，以及apache的众多开源项目等等， 正是由于站在这些开源项目的肩膀上，才有Easy Scheduler的诞生的可能。对此我们对使用的所有开源软件表示非常的感谢！我们也希望自己不仅是开源的受益者，也能成为开源的 贡献者，于是我们决定把易调度贡献出来，并承诺长期维护。也希望对开源有同样热情和信念的伙伴加入进来，一起为开源献出一份力！ -### 帮助 -The fastest way to get response from our developers is to submit issues, or add our wechat : 510570367 +### 获得帮助 +1. Submit an issue +1. Mail list: dev@dolphinscheduler.apache.org. Mail to dev-subscribe@dolphinscheduler.apache.org, follow the reply to subscribe the mail list. +1. Contact WeChat group manager, ID 510570367. This is for Mandarin(CN) discussion. + +### 版权 +Please refer to [LICENSE](https://github.com/apache/incubator-dolphinscheduler/blob/dev/LICENSE) file. diff --git a/docs/en_US/EasyScheduler Proposal.md b/docs/en_US/EasyScheduler Proposal.md index 552067fe47..6bcea73540 100644 --- a/docs/en_US/EasyScheduler Proposal.md +++ b/docs/en_US/EasyScheduler Proposal.md @@ -30,7 +30,7 @@ The codes are already under Apache License Version 2.0. We want to find a data processing tool with the following features: -- Easy to use，developers can build a ETL process with a very simple drag and drop operation. not only for ETL developers，people who can't write code also can use this tool for ETL operation such as system adminitrator. +- Easy to use，developers can build a ETL process with a very simple drag and drop operation. not only for ETL developers，people who can't write code also can use this tool for ETL operation such as system administrator. - Solving the problem of "complex task dependencies" , and it can monitor the ETL running status. - Support multi-tenant. - Support many task types: Shell, MR, Spark, SQL (mysql, postgresql, hive, sparksql), Python, Sub_Process, Procedure, etc. @@ -73,7 +73,7 @@ Thus, it is very unlikely that EasyScheduler becomes orphaned. EasyScheduler's core developers have been running it as a community-oriented open source project for some time, several of them already have experience working with open source communities, they are also active in presto, alluxio and other projects. At the same time, we will learn more open source experiences by following the Apache way in our incubator journey. -### Homogenous Developers +### Homogeneous Developers The current developers work across a variety of organizations including Analysys, guandata and hydee; some individual developers are accepted as developers of EasyScheduler as well. @@ -110,7 +110,7 @@ The project consists of three distinct codebases: core and document. The address ## Source and Intellectual Property Submission Plan -As soon as EasyScheduler is approved to join Apache Incubator, Analysys will provide the Software Grant Agreement(SGA) and intial committers will submit ICLA(s). The code is already licensed under the Apache Software License, version 2.0. +As soon as EasyScheduler is approved to join Apache Incubator, Analysys will provide the Software Grant Agreement(SGA) and initial committers will submit ICLA(s). The code is already licensed under the Apache Software License, version 2.0. ## External Dependencies diff --git a/docs/en_US/architecture-design.md b/docs/en_US/architecture-design.md index e1c8f01f34..0587993d05 100644 --- a/docs/en_US/architecture-design.md +++ b/docs/en_US/architecture-design.md @@ -266,7 +266,7 @@ In the early scheduling design, if there is no priority design and fair scheduli /** * task log appender */ - Public class TaskLogAppender extends FileAppender { ... @@ -297,7 +297,7 @@ Generate a log in the form of /process definition id/process instance id/task in /** * task log filter */ -Public class TaskLogFilter extends Filter { @Override Public FilterReply decide(ILoggingEvent event) { @@ -313,4 +313,4 @@ Public class TaskLogFilter extends Filter "Queue Manage" - > "Creat Queue" +- "Security" - > "Queue Manage" - > "Create Queue"

@@ -403,7 +403,7 @@ conf/common/hadoop.properties try { // execute response = httpclient.execute(httpPost); - // eponse status code 200 + // response status code 200 if (response.getStatusLine().getStatusCode() == 200) { String content = EntityUtils.toString(response.getEntity(), "UTF-8"); System.out.println(content); @@ -533,7 +533,7 @@ conf/common/hadoop.properties

-- Datasource: The data source type of stored procedure supports MySQL and POSTGRRESQL, and chooses the corresponding data source. +- Datasource: The data source type of stored procedure supports MySQL and POSTGRESQL, and chooses the corresponding data source. - Method: The method name of the stored procedure - Custom parameters: Custom parameter types of stored procedures support IN and OUT, and data types support nine data types: VARCHAR, INTEGER, LONG, FLOAT, DOUBLE, DATE, TIME, TIMESTAMP and BOOLEAN. diff --git a/docs/zh_CN/前端开发文档.md b/docs/zh_CN/前端开发文档.md index b3e8aa82d9..f805f5ed8c 100644 --- a/docs/zh_CN/前端开发文档.md +++ b/docs/zh_CN/前端开发文档.md @@ -64,7 +64,7 @@ npm install node-sass --unsafe-perm //单独安装node-sass依赖 访问地址 `http://localhost:8888/#/` -#### Liunx下使用node启动并且守护进程 +#### Linux下使用node启动并且守护进程 安装pm2 `npm install -g pm2` @@ -237,7 +237,7 @@ export default { ├── conditions ├── conditions.vue └── _source - └── serach.vue + └── search.vue └── util.js ``` diff --git a/docs/zh_CN/系统使用手册.md b/docs/zh_CN/系统使用手册.md index 4d0cc21195..348cc2b36a 100644 --- a/docs/zh_CN/系统使用手册.md +++ b/docs/zh_CN/系统使用手册.md @@ -391,7 +391,7 @@ conf/common/hadoop.properties try { // execute response = httpclient.execute(httpPost); - // eponse status code 200 + // response status code 200 if (response.getStatusLine().getStatusCode() == 200) { String content = EntityUtils.toString(response.getEntity(), "UTF-8"); System.out.println(content); diff --git a/docs/zh_CN/系统架构设计.md b/docs/zh_CN/系统架构设计.md index 8c3ef4e053..61e522b6cd 100644 --- a/docs/zh_CN/系统架构设计.md +++ b/docs/zh_CN/系统架构设计.md @@ -253,7 +253,7 @@ Master Scheduler线程一旦发现任务实例为” 需要容错”状态，则 /** * task log appender */ - public class TaskLogAppender extends FileAppender { ... @@ -286,7 +286,7 @@ Master Scheduler线程一旦发现任务实例为” 需要容错”状态，则 /** * task log filter */ -public class TaskLogFilter extends Filter { @Override public FilterReply decide(ILoggingEvent event) { diff --git a/escheduler-alert/src/main/java/cn/escheduler/alert/AlertServer.java b/escheduler-alert/src/main/java/cn/escheduler/alert/AlertServer.java index 8de3a65d6c..8d27f6f770 100644 --- a/escheduler-alert/src/main/java/cn/escheduler/alert/AlertServer.java +++ b/escheduler-alert/src/main/java/cn/escheduler/alert/AlertServer.java @@ -20,36 +20,40 @@ import cn.escheduler.alert.runner.AlertSender; import cn.escheduler.alert.utils.Constants; import cn.escheduler.common.thread.Stopper; import cn.escheduler.dao.AlertDao; -import cn.escheduler.dao.DaoFactory; import cn.escheduler.dao.model.Alert; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.boot.CommandLineRunner; +import org.springframework.boot.SpringApplication; +import org.springframework.context.annotation.ComponentScan; import java.util.List; /** * alert of start */ -public class AlertServer { +@ComponentScan("cn.escheduler") +public class AlertServer implements CommandLineRunner { private static final Logger logger = LoggerFactory.getLogger(AlertServer.class); /** * Alert Dao */ - private AlertDao alertDao = DaoFactory.getDaoInstance(AlertDao.class); + private AlertDao alertDao; private AlertSender alertSender; - private static AlertServer instance; - - private AlertServer() { + private static volatile AlertServer instance; + public AlertServer(AlertDao alertDao) { + this.alertDao = alertDao; } - public static AlertServer getInstance(){ + public static AlertServer getInstance(AlertDao alertDao){ if (null == instance) { synchronized (AlertServer.class) { if(null == instance) { - instance = new AlertServer(); + instance = new AlertServer(alertDao); } } } @@ -72,8 +76,13 @@ public class AlertServer { public static void main(String[] args){ + SpringApplication app = new SpringApplication(AlertServer.class); + app.run(args); + } - AlertServer alertServer = AlertServer.getInstance(); + @Override + public void run(String... strings) throws Exception { + AlertServer alertServer = AlertServer.getInstance(alertDao); alertServer.start(); } } diff --git a/escheduler-alert/src/main/resources/application_alert.properties b/escheduler-alert/src/main/resources/application_alert.properties new file mode 100644 index 0000000000..cbc81b06e6 --- /dev/null +++ b/escheduler-alert/src/main/resources/application_alert.properties @@ -0,0 +1,5 @@ +logging.config=classpath:alert_logback.xml + +# server port +server.port=7789 + diff --git a/escheduler-api/src/main/java/cn/escheduler/api/CombinedApplicationServer.java b/escheduler-api/src/main/java/cn/escheduler/api/CombinedApplicationServer.java index fc421944e9..70535488aa 100644 --- a/escheduler-api/src/main/java/cn/escheduler/api/CombinedApplicationServer.java +++ b/escheduler-api/src/main/java/cn/escheduler/api/CombinedApplicationServer.java @@ -51,7 +51,7 @@ public class CombinedApplicationServer extends SpringBootServletInitializer { LoggerServer server = new LoggerServer(); server.start(); - AlertServer alertServer = AlertServer.getInstance(); + AlertServer alertServer = AlertServer.getInstance(alertDao); alertServer.start(); } } diff --git a/escheduler-api/src/main/java/cn/escheduler/api/service/ProcessDefinitionService.java b/escheduler-api/src/main/java/cn/escheduler/api/service/ProcessDefinitionService.java index 45d0b46e79..ed92ce5a02 100644 --- a/escheduler-api/src/main/java/cn/escheduler/api/service/ProcessDefinitionService.java +++ b/escheduler-api/src/main/java/cn/escheduler/api/service/ProcessDefinitionService.java @@ -547,8 +547,27 @@ public class ProcessDefinitionService extends BaseDAGService { sqlParameters.put("datasourceName", dataSource.getName()); } taskNode.put("params", sqlParameters); + }else if(taskType.equals(TaskType.DEPENDENT.name())){ + JSONObject dependentParameters = JSONUtils.parseObject(taskNode.getString("dependence")); + if(dependentParameters != null){ + JSONArray dependTaskList = (JSONArray) dependentParameters.get("dependTaskList"); + for (int j = 0; j < dependTaskList.size(); j++) { + JSONObject dependentTaskModel = dependTaskList.getJSONObject(j); + JSONArray dependItemList = (JSONArray) dependentTaskModel.get("dependItemList"); + for (int k = 0; k < dependItemList.size(); k++) { + JSONObject dependentItem = dependItemList.getJSONObject(k); + int definitionId = dependentItem.getInteger("definitionId"); + ProcessDefinition definition = processDefineMapper.queryByDefineId(definitionId); + if(definition != null){ + dependentItem.put("projectName",definition.getProjectName()); + dependentItem.put("definitionName",definition.getName()); + } + } + } + taskNode.put("dependence", dependentParameters); } } + } } jsonObject.put("tasks", jsonArray); processDefinition.setProcessDefinitionJson(jsonObject.toString()); @@ -570,7 +589,7 @@ public class ProcessDefinitionService extends BaseDAGService { row.put("scheduleEndTime", schedule.getEndTime()); row.put("scheduleCrontab", schedule.getCrontab()); row.put("scheduleFailureStrategy", schedule.getFailureStrategy()); - row.put("scheduleReleaseState", schedule.getReleaseState()); + row.put("scheduleReleaseState", ReleaseState.OFFLINE); row.put("scheduleProcessInstancePriority", schedule.getProcessInstancePriority()); if(schedule.getId() == -1){ row.put("scheduleWorkerGroupId", -1); @@ -643,16 +662,19 @@ public class ProcessDefinitionService extends BaseDAGService { projectName = json.get("projectName").toString(); } else { putMsg(result, Status.DATA_IS_NULL, "processDefinitionName"); + return result; } if (ObjectUtils.allNotNull(json.get("processDefinitionName"))) { processDefinitionName = json.get("processDefinitionName").toString(); } else { putMsg(result, Status.DATA_IS_NULL, "processDefinitionName"); + return result; } if (ObjectUtils.allNotNull(json.get("processDefinitionJson"))) { processDefinitionJson = json.get("processDefinitionJson").toString(); } else { putMsg(result, Status.DATA_IS_NULL, "processDefinitionJson"); + return result; } if (ObjectUtils.allNotNull(json.get("processDefinitionDesc"))) { processDefinitionDesc = json.get("processDefinitionDesc").toString(); @@ -664,17 +686,46 @@ public class ProcessDefinitionService extends BaseDAGService { processDefinitionConnects = json.get("processDefinitionConnects").toString(); } + Project project = projectMapper.queryByName(projectName); + if(project != null){ + processDefinitionName = recursionProcessDefinitionName(project.getId(), processDefinitionName,1); + } + JSONObject jsonObject = JSONUtils.parseObject(processDefinitionJson); JSONArray jsonArray = (JSONArray) jsonObject.get("tasks"); for (int j = 0; j < jsonArray.size(); j++) { JSONObject taskNode = jsonArray.getJSONObject(j); - JSONObject sqlParameters = JSONUtils.parseObject(taskNode.getString("params")); - List dataSources = dataSourceMapper.queryDataSourceByName(sqlParameters.getString("datasourceName")); - if (dataSources.size() > 0) { - DataSource dataSource = dataSources.get(0); - sqlParameters.put("datasource", dataSource.getId()); + String taskType = taskNode.getString("type"); + if(taskType.equals(TaskType.SQL.name()) || taskType.equals(TaskType.PROCEDURE.name())) { + JSONObject sqlParameters = JSONUtils.parseObject(taskNode.getString("params")); + List dataSources = dataSourceMapper.queryDataSourceByName(sqlParameters.getString("datasourceName")); + if (dataSources.size() > 0) { + DataSource dataSource = dataSources.get(0); + sqlParameters.put("datasource", dataSource.getId()); + } + taskNode.put("params", sqlParameters); + }else if(taskType.equals(TaskType.DEPENDENT.name())){ + JSONObject dependentParameters = JSONUtils.parseObject(taskNode.getString("dependence")); + if(dependentParameters != null){ + JSONArray dependTaskList = (JSONArray) dependentParameters.get("dependTaskList"); + for (int h = 0; h < dependTaskList.size(); h++) { + JSONObject dependentTaskModel = dependTaskList.getJSONObject(h); + JSONArray dependItemList = (JSONArray) dependentTaskModel.get("dependItemList"); + for (int k = 0; k < dependItemList.size(); k++) { + JSONObject dependentItem = dependItemList.getJSONObject(k); + Project dependentItemProject = projectMapper.queryByName(dependentItem.getString("projectName")); + if(dependentItemProject != null){ + ProcessDefinition definition = processDefineMapper.queryByDefineName(dependentItemProject.getId(),dependentItem.getString("definitionName")); + if(definition != null){ + dependentItem.put("projectId",dependentItemProject.getId()); + dependentItem.put("definitionId",definition.getId()); + } + } + } + } + taskNode.put("dependence", dependentParameters); + } } - taskNode.put("params", sqlParameters); } jsonObject.put("tasks", jsonArray); @@ -1112,5 +1163,20 @@ public class ProcessDefinitionService extends BaseDAGService { return graph.hasCycle(); } + private String recursionProcessDefinitionName(Integer projectId,String processDefinitionName,int num){ + ProcessDefinition processDefinition = processDefineMapper.queryByDefineName(projectId, processDefinitionName); + if (processDefinition != null) { + if(num>1){ + String str = processDefinitionName.substring(0,processDefinitionName.length() - 3); + processDefinitionName = str + "("+num+")"; + }else{ + processDefinitionName = processDefinition.getName() + "("+num+")"; + } + }else{ + return processDefinitionName; + } + return recursionProcessDefinitionName(projectId,processDefinitionName,num + 1); + } + } diff --git a/escheduler-api/src/main/java/cn/escheduler/api/utils/ZooKeeperState.java b/escheduler-api/src/main/java/cn/escheduler/api/utils/ZooKeeperState.java index 7d29e8618f..a40cf05341 100644 --- a/escheduler-api/src/main/java/cn/escheduler/api/utils/ZooKeeperState.java +++ b/escheduler-api/src/main/java/cn/escheduler/api/utils/ZooKeeperState.java @@ -1,3 +1,19 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ package cn.escheduler.api.utils; import org.apache.commons.lang3.StringUtils; @@ -9,7 +25,7 @@ import java.io.IOException; import java.util.Scanner; /** - * zookeeper状态监控:4字口诀 + * zookeeper状态监控:4字口诀 * */ public class ZooKeeperState { @@ -77,7 +93,7 @@ public class ZooKeeperState { } scannerForWchs.close(); } - + String consText = cmd("cons"); if (StringUtils.isNotBlank(consText)) { Scanner scannerForCons = new Scanner(consText); @@ -205,7 +221,7 @@ public class ZooKeeperState { + ", watches=" + watches + ", connections=" + connections + "]"; } - - - + + + } diff --git a/escheduler-api/src/main/java/cn/escheduler/api/utils/ZookeeperMonitor.java b/escheduler-api/src/main/java/cn/escheduler/api/utils/ZookeeperMonitor.java index dc6f95d7ec..4bbe35d6b7 100644 --- a/escheduler-api/src/main/java/cn/escheduler/api/utils/ZookeeperMonitor.java +++ b/escheduler-api/src/main/java/cn/escheduler/api/utils/ZookeeperMonitor.java @@ -1,3 +1,19 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ package cn.escheduler.api.utils; import cn.escheduler.common.enums.ZKNodeType; @@ -57,14 +73,14 @@ public class ZookeeperMonitor extends AbstractZKClient{ if(StringUtils.isNotBlank(zookeeperServers)){ String[] zookeeperServersArray = zookeeperServers.split(","); - + for (String zookeeperServer : zookeeperServersArray) { ZooKeeperState state = new ZooKeeperState(zookeeperServer); boolean ok = state.ruok(); if(ok){ state.getZookeeperInfo(); } - + String hostName = zookeeperServer; int connections = state.getConnections(); int watches = state.getWatches(); diff --git a/escheduler-api/src/test/java/cn/escheduler/api/HttpClientTest.java b/escheduler-api/src/test/java/cn/escheduler/api/HttpClientTest.java index b2495a2f9d..1d527b563c 100644 --- a/escheduler-api/src/test/java/cn/escheduler/api/HttpClientTest.java +++ b/escheduler-api/src/test/java/cn/escheduler/api/HttpClientTest.java @@ -59,7 +59,7 @@ public class HttpClientTest { try { // execute response = httpclient.execute(httpPost); - // eponse status code 200 + // response status code 200 if (response.getStatusLine().getStatusCode() == 200) { String content = EntityUtils.toString(response.getEntity(), "UTF-8"); logger.info(content); @@ -96,7 +96,7 @@ public class HttpClientTest { try { // execute http get request response = httpclient.execute(httpGet); - // reponse status code 200 + // response status code 200 if (response.getStatusLine().getStatusCode() == 200) { String content = EntityUtils.toString(response.getEntity(), "UTF-8"); logger.info("start--------------->"); @@ -139,7 +139,7 @@ public class HttpClientTest { try { // execute http get request response = httpclient.execute(httpGet); - // reponse status code 200 + // response status code 200 if (response.getStatusLine().getStatusCode() == 200) { String content = EntityUtils.toString(response.getEntity(), "UTF-8"); logger.info("start--------------->"); diff --git a/escheduler-api/src/test/java/cn/escheduler/api/controller/MonitorControllerTest.java b/escheduler-api/src/test/java/cn/escheduler/api/controller/MonitorControllerTest.java index e4c6103cd0..79bd958246 100644 --- a/escheduler-api/src/test/java/cn/escheduler/api/controller/MonitorControllerTest.java +++ b/escheduler-api/src/test/java/cn/escheduler/api/controller/MonitorControllerTest.java @@ -1,3 +1,19 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ package cn.escheduler.api.controller; import cn.escheduler.api.enums.Status; @@ -79,4 +95,4 @@ public class MonitorControllerTest extends AbstractControllerTest { Assert.assertEquals(Status.SUCCESS.getCode(),result.getCode().intValue()); logger.info(mvcResult.getResponse().getContentAsString()); } -} \ No newline at end of file +} diff --git a/escheduler-api/src/test/java/cn/escheduler/api/utils/ZookeeperMonitorUtilsTest.java b/escheduler-api/src/test/java/cn/escheduler/api/utils/ZookeeperMonitorUtilsTest.java index c79d6ea31d..3e590756e4 100644 --- a/escheduler-api/src/test/java/cn/escheduler/api/utils/ZookeeperMonitorUtilsTest.java +++ b/escheduler-api/src/test/java/cn/escheduler/api/utils/ZookeeperMonitorUtilsTest.java @@ -1,3 +1,19 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ package cn.escheduler.api.utils; import cn.escheduler.common.model.MasterServer; @@ -27,4 +43,4 @@ public class ZookeeperMonitorUtilsTest { } -} \ No newline at end of file +} diff --git a/escheduler-common/pom.xml b/escheduler-common/pom.xml index b5f1d58620..cabdaf41b6 100644 --- a/escheduler-common/pom.xml +++ b/escheduler-common/pom.xml @@ -83,10 +83,6 @@ javax.servlet servlet-api - - log4j - log4j - org.apache.curator curator-client @@ -519,6 +515,10 @@ log4j-web org.apache.logging.log4j + + log4j-1.2-api + org.apache.logging.log4j + @@ -572,6 +572,12 @@ org.mortbay.jetty jsp-2.1 + + + org.mortbay.jetty + servlet-api-2.5 + + diff --git a/escheduler-common/src/main/java/cn/escheduler/common/Constants.java b/escheduler-common/src/main/java/cn/escheduler/common/Constants.java index 10a4460678..8522992f9f 100644 --- a/escheduler-common/src/main/java/cn/escheduler/common/Constants.java +++ b/escheduler-common/src/main/java/cn/escheduler/common/Constants.java @@ -696,6 +696,10 @@ public final class Constants { */ public static final String SPARK_QUEUE = "--queue"; + /** + * --queue --qu + */ + public static final String FLINK_QUEUE = "--qu"; /** * exit code success @@ -906,4 +910,18 @@ public final class Constants { * hive conf */ public static final String HIVE_CONF = "hiveconf:"; + + //flink 任务 + public static final String FLINK_YARN_CLUSTER = "yarn-cluster"; + public static final String FLINK_RUN_MODE = "-m"; + public static final String FLINK_YARN_SLOT = "-ys"; + public static final String FLINK_APP_NAME = "-ynm"; + public static final String FLINK_TASK_MANAGE = "-yn"; + + public static final String FLINK_JOB_MANAGE_MEM = "-yjm"; + public static final String FLINK_TASK_MANAGE_MEM = "-ytm"; + public static final String FLINK_detach = "-d"; + public static final String FLINK_MAIN_CLASS = "-c"; + + } diff --git a/escheduler-common/src/main/java/cn/escheduler/common/enums/HttpCheckCondition.java b/escheduler-common/src/main/java/cn/escheduler/common/enums/HttpCheckCondition.java new file mode 100644 index 0000000000..b88ec01980 --- /dev/null +++ b/escheduler-common/src/main/java/cn/escheduler/common/enums/HttpCheckCondition.java @@ -0,0 +1,30 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package cn.escheduler.common.enums; + +/** + * http check condition + */ +public enum HttpCheckCondition { + /** + * 0 status_code_default:200 + * 1 status_code_custom + * 2 body_contains + * 3 body_not_contains + */ + STATUS_CODE_DEFAULT,STATUS_CODE_CUSTOM, BODY_CONTAINS, BODY_NOT_CONTAINS +} \ No newline at end of file diff --git a/escheduler-common/src/main/java/cn/escheduler/common/enums/HttpMethod.java b/escheduler-common/src/main/java/cn/escheduler/common/enums/HttpMethod.java new file mode 100644 index 0000000000..32cbf1c42f --- /dev/null +++ b/escheduler-common/src/main/java/cn/escheduler/common/enums/HttpMethod.java @@ -0,0 +1,31 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package cn.escheduler.common.enums; + +/** + * http method + */ +public enum HttpMethod { + /** + * 0 get + * 1 post + * 2 head + * 3 put + * 4 delete + */ + GET, POST, HEAD, PUT, DELETE +} \ No newline at end of file diff --git a/escheduler-common/src/main/java/cn/escheduler/common/enums/HttpParametersType.java b/escheduler-common/src/main/java/cn/escheduler/common/enums/HttpParametersType.java new file mode 100644 index 0000000000..21f4d8b3cc --- /dev/null +++ b/escheduler-common/src/main/java/cn/escheduler/common/enums/HttpParametersType.java @@ -0,0 +1,29 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package cn.escheduler.common.enums; + +/** + * http parameters type + */ +public enum HttpParametersType { + /** + * 0 parameter; + * 1 body; + * 2 headers; + */ + PARAMETER,BODY,HEADERS +} diff --git a/escheduler-common/src/main/java/cn/escheduler/common/enums/TaskType.java b/escheduler-common/src/main/java/cn/escheduler/common/enums/TaskType.java index 1d589167e3..a212261db6 100644 --- a/escheduler-common/src/main/java/cn/escheduler/common/enums/TaskType.java +++ b/escheduler-common/src/main/java/cn/escheduler/common/enums/TaskType.java @@ -29,8 +29,10 @@ public enum TaskType { * 5 SPARK * 6 PYTHON * 7 DEPENDENT + * 8 FLINK + * 9 HTTP */ - SHELL,SQL, SUB_PROCESS,PROCEDURE,MR,SPARK,PYTHON,DEPENDENT; + SHELL,SQL, SUB_PROCESS,PROCEDURE,MR,SPARK,PYTHON,DEPENDENT,FLINK,HTTP; public static boolean typeIsNormalTask(String typeName) { TaskType taskType = TaskType.valueOf(typeName); diff --git a/escheduler-common/src/main/java/cn/escheduler/common/enums/ZKNodeType.java b/escheduler-common/src/main/java/cn/escheduler/common/enums/ZKNodeType.java index 371231f727..af44439455 100644 --- a/escheduler-common/src/main/java/cn/escheduler/common/enums/ZKNodeType.java +++ b/escheduler-common/src/main/java/cn/escheduler/common/enums/ZKNodeType.java @@ -1,3 +1,20 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + package cn.escheduler.common.enums; /** diff --git a/escheduler-common/src/main/java/cn/escheduler/common/process/HttpProperty.java b/escheduler-common/src/main/java/cn/escheduler/common/process/HttpProperty.java new file mode 100644 index 0000000000..ed7cc46a38 --- /dev/null +++ b/escheduler-common/src/main/java/cn/escheduler/common/process/HttpProperty.java @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package cn.escheduler.common.process; + +import cn.escheduler.common.enums.HttpParametersType; + +import java.util.Objects; + +public class HttpProperty { + /** + * key + */ + private String prop; + + /** + * httpParametersType + */ + private HttpParametersType httpParametersType; + + /** + * value + */ + private String value; + + public HttpProperty() { + } + + public HttpProperty(String prop, HttpParametersType httpParametersType, String value) { + this.prop = prop; + this.httpParametersType = httpParametersType; + this.value = value; + } + + /** + * getter method + * + * @return the prop + * @see HttpProperty#prop + */ + public String getProp() { + return prop; + } + + /** + * setter method + * + * @param prop the prop to set + * @see HttpProperty#prop + */ + public void setProp(String prop) { + this.prop = prop; + } + + /** + * getter method + * + * @return the value + * @see HttpProperty#value + */ + public String getValue() { + return value; + } + + /** + * setter method + * + * @param value the value to set + * @see HttpProperty#value + */ + public void setValue(String value) { + this.value = value; + } + + public HttpParametersType getHttpParametersType() { + return httpParametersType; + } + + public void setHttpParametersType(HttpParametersType httpParametersType) { + this.httpParametersType = httpParametersType; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + HttpProperty property = (HttpProperty) o; + return Objects.equals(prop, property.prop) && + Objects.equals(value, property.value); + } + + + @Override + public int hashCode() { + return Objects.hash(prop, value); + } + + @Override + public String toString() { + return "HttpProperty{" + + "prop='" + prop + '\'' + + ", httpParametersType=" + httpParametersType + + ", value='" + value + '\'' + + '}'; + } + + +} diff --git a/escheduler-common/src/main/java/cn/escheduler/common/shell/AbstractShell.java b/escheduler-common/src/main/java/cn/escheduler/common/shell/AbstractShell.java index 1250732632..0880c4d5bb 100644 --- a/escheduler-common/src/main/java/cn/escheduler/common/shell/AbstractShell.java +++ b/escheduler-common/src/main/java/cn/escheduler/common/shell/AbstractShell.java @@ -157,7 +157,7 @@ public abstract class AbstractShell { BufferedReader inReader = new BufferedReader(new InputStreamReader(process .getInputStream())); - final StringBuffer errMsg = new StringBuffer(); + final StringBuilder errMsg = new StringBuilder(); // read error and input streams as this would free up the buffers // free the error stream buffer diff --git a/escheduler-common/src/main/java/cn/escheduler/common/task/flink/FlinkParameters.java b/escheduler-common/src/main/java/cn/escheduler/common/task/flink/FlinkParameters.java new file mode 100644 index 0000000000..54dfcb7103 --- /dev/null +++ b/escheduler-common/src/main/java/cn/escheduler/common/task/flink/FlinkParameters.java @@ -0,0 +1,219 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package cn.escheduler.common.task.flink; + +import cn.escheduler.common.enums.ProgramType; +import cn.escheduler.common.process.ResourceInfo; +import cn.escheduler.common.task.AbstractParameters; + +import java.util.List; +import java.util.stream.Collectors; + +/** + * spark parameters + */ +public class FlinkParameters extends AbstractParameters { + + /** + * major jar + */ + private ResourceInfo mainJar; + + /** + * major class + */ + private String mainClass; + + /** + * deploy mode yarn-cluster yarn-client yarn-local + */ + private String deployMode; + + /** + * arguments + */ + private String mainArgs; + + /** + * slot个数 + */ + private int slot; + + /** + *Yarn application的名字 + */ + + private String appName; + + /** + * taskManager 数量 + */ + private int taskManager; + + /** + * jobManagerMemory 内存大小 + */ + private String jobManagerMemory ; + + /** + * taskManagerMemory内存大小 + */ + private String taskManagerMemory; + + /** + * resource list + */ + private List resourceList; + + /** + * The YARN queue to submit to + */ + private String queue; + + /** + * other arguments + */ + private String others; + + /** + * program type + * 0 JAVA,1 SCALA,2 PYTHON + */ + private ProgramType programType; + + public ResourceInfo getMainJar() { + return mainJar; + } + + public void setMainJar(ResourceInfo mainJar) { + this.mainJar = mainJar; + } + + public String getMainClass() { + return mainClass; + } + + public void setMainClass(String mainClass) { + this.mainClass = mainClass; + } + + public String getDeployMode() { + return deployMode; + } + + public void setDeployMode(String deployMode) { + this.deployMode = deployMode; + } + + public String getMainArgs() { + return mainArgs; + } + + public void setMainArgs(String mainArgs) { + this.mainArgs = mainArgs; + } + + public int getSlot() { + return slot; + } + + public void setSlot(int slot) { + this.slot = slot; + } + + public String getAppName() { + return appName; + } + + public void setAppName(String appName) { + this.appName = appName; + } + + public int getTaskManager() { + return taskManager; + } + + public void setTaskManager(int taskManager) { + this.taskManager = taskManager; + } + + public String getJobManagerMemory() { + return jobManagerMemory; + } + + public void setJobManagerMemory(String jobManagerMemory) { + this.jobManagerMemory = jobManagerMemory; + } + + public String getTaskManagerMemory() { + return taskManagerMemory; + } + + public void setTaskManagerMemory(String taskManagerMemory) { + this.taskManagerMemory = taskManagerMemory; + } + + public String getQueue() { + return queue; + } + + public void setQueue(String queue) { + this.queue = queue; + } + + public List getResourceList() { + return resourceList; + } + + public void setResourceList(List resourceList) { + this.resourceList = resourceList; + } + + public String getOthers() { + return others; + } + + public void setOthers(String others) { + this.others = others; + } + + public ProgramType getProgramType() { + return programType; + } + + public void setProgramType(ProgramType programType) { + this.programType = programType; + } + + @Override + public boolean checkParameters() { + return mainJar != null && programType != null; + } + + + @Override + public List getResourceFilesList() { + if(resourceList !=null ) { + this.resourceList.add(mainJar); + return resourceList.stream() + .map(p -> p.getRes()).collect(Collectors.toList()); + } + return null; + } + + +} diff --git a/escheduler-common/src/main/java/cn/escheduler/common/task/http/HttpParameters.java b/escheduler-common/src/main/java/cn/escheduler/common/task/http/HttpParameters.java new file mode 100644 index 0000000000..dc23f738b8 --- /dev/null +++ b/escheduler-common/src/main/java/cn/escheduler/common/task/http/HttpParameters.java @@ -0,0 +1,108 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package cn.escheduler.common.task.http; + +import cn.escheduler.common.enums.HttpCheckCondition; +import cn.escheduler.common.enums.HttpMethod; +import cn.escheduler.common.process.HttpProperty; +import cn.escheduler.common.task.AbstractParameters; +import org.apache.commons.lang.StringUtils; + +import java.util.ArrayList; +import java.util.List; + +/** + * http parameter + */ +public class HttpParameters extends AbstractParameters { + /** + * url + */ + private String url; + + /** + * httpMethod + */ + private HttpMethod httpMethod; + + /** + * http params + */ + private List httpParams; + + /** + * httpCheckCondition + */ + private HttpCheckCondition httpCheckCondition = HttpCheckCondition.STATUS_CODE_DEFAULT; + + /** + * condition + */ + private String condition; + + + + @Override + public boolean checkParameters() { + return StringUtils.isNotEmpty(url); + } + + @Override + public List getResourceFilesList() { + return new ArrayList<>(); + } + + public String getUrl() { + return url; + } + + public void setUrl(String url) { + this.url = url; + } + + public HttpMethod getHttpMethod() { + return httpMethod; + } + + public void setHttpMethod(HttpMethod httpMethod) { + this.httpMethod = httpMethod; + } + + public List getHttpParams() { + return httpParams; + } + + public void setHttpParams(List httpParams) { + this.httpParams = httpParams; + } + + public HttpCheckCondition getHttpCheckCondition() { + return httpCheckCondition; + } + + public void setHttpCheckCondition(HttpCheckCondition httpCheckCondition) { + this.httpCheckCondition = httpCheckCondition; + } + + public String getCondition() { + return condition; + } + + public void setCondition(String condition) { + this.condition = condition; + } +} diff --git a/escheduler-common/src/main/java/cn/escheduler/common/thread/ThreadPoolExecutors.java b/escheduler-common/src/main/java/cn/escheduler/common/thread/ThreadPoolExecutors.java index 8329e8c18a..ea03026d3a 100644 --- a/escheduler-common/src/main/java/cn/escheduler/common/thread/ThreadPoolExecutors.java +++ b/escheduler-common/src/main/java/cn/escheduler/common/thread/ThreadPoolExecutors.java @@ -40,7 +40,7 @@ public class ThreadPoolExecutors { private static final Logger logger = LoggerFactory.getLogger(ThreadPoolExecutors.class); private static Executor executor; - private static ThreadPoolExecutors threadPoolExecutors; + private static volatile ThreadPoolExecutors threadPoolExecutors; private ThreadPoolExecutors(){} diff --git a/escheduler-common/src/main/java/cn/escheduler/common/utils/HadoopUtils.java b/escheduler-common/src/main/java/cn/escheduler/common/utils/HadoopUtils.java index 6bc33eca42..bcea353a60 100644 --- a/escheduler-common/src/main/java/cn/escheduler/common/utils/HadoopUtils.java +++ b/escheduler-common/src/main/java/cn/escheduler/common/utils/HadoopUtils.java @@ -288,9 +288,11 @@ public class HadoopUtils implements Closeable { if (dstPath.isFile()) { if (overwrite) { dstPath.delete(); + }else{ + throw new IOException("destination file already exists!"); } } else { - logger.error("destination file must be a file"); + throw new IOException("destination file must be a file!"); } } diff --git a/escheduler-common/src/main/java/cn/escheduler/common/utils/TaskParametersUtils.java b/escheduler-common/src/main/java/cn/escheduler/common/utils/TaskParametersUtils.java index feff4141da..b3fcade4a7 100644 --- a/escheduler-common/src/main/java/cn/escheduler/common/utils/TaskParametersUtils.java +++ b/escheduler-common/src/main/java/cn/escheduler/common/utils/TaskParametersUtils.java @@ -19,6 +19,8 @@ package cn.escheduler.common.utils; import cn.escheduler.common.enums.TaskType; import cn.escheduler.common.task.AbstractParameters; import cn.escheduler.common.task.dependent.DependentParameters; +import cn.escheduler.common.task.flink.FlinkParameters; +import cn.escheduler.common.task.http.HttpParameters; import cn.escheduler.common.task.mr.MapreduceParameters; import cn.escheduler.common.task.procedure.ProcedureParameters; import cn.escheduler.common.task.python.PythonParameters; @@ -63,6 +65,10 @@ public class TaskParametersUtils { return JSONUtils.parseObject(parameter, PythonParameters.class); case DEPENDENT: return JSONUtils.parseObject(parameter, DependentParameters.class); + case FLINK: + return JSONUtils.parseObject(parameter, FlinkParameters.class); + case HTTP: + return JSONUtils.parseObject(parameter, HttpParameters.class); default: return null; } diff --git a/escheduler-common/src/main/resources/common/common.properties b/escheduler-common/src/main/resources/common/common.properties index 27d525f8f5..5de705c8f8 100644 --- a/escheduler-common/src/main/resources/common/common.properties +++ b/escheduler-common/src/main/resources/common/common.properties @@ -35,7 +35,7 @@ login.user.keytab.path=/opt/hdfs.headless.keytab escheduler.env.path=/opt/.escheduler_env.sh #resource.view.suffixs -resource.view.suffixs=txt,log,sh,conf,cfg,py,java,sql,hql,xml +resource.view.suffixs=txt,log,sh,conf,cfg,py,java,sql,hql,xml,properties # is development state? default "false" development.state=true diff --git a/escheduler-common/src/test/java/cn/escheduler/common/utils/IpUtilsTest.java b/escheduler-common/src/test/java/cn/escheduler/common/utils/IpUtilsTest.java index 11a03a2334..ec286257bc 100644 --- a/escheduler-common/src/test/java/cn/escheduler/common/utils/IpUtilsTest.java +++ b/escheduler-common/src/test/java/cn/escheduler/common/utils/IpUtilsTest.java @@ -1,3 +1,20 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + package cn.escheduler.common.utils; import org.junit.Assert; @@ -38,4 +55,4 @@ public class IpUtilsTest { Assert.assertEquals(ip, i1); Assert.assertEquals(ip2, i2); } -} \ No newline at end of file +} diff --git a/escheduler-dao/src/main/java/cn/escheduler/dao/ProcessDao.java b/escheduler-dao/src/main/java/cn/escheduler/dao/ProcessDao.java index 73ae868c80..dee22d6f36 100644 --- a/escheduler-dao/src/main/java/cn/escheduler/dao/ProcessDao.java +++ b/escheduler-dao/src/main/java/cn/escheduler/dao/ProcessDao.java @@ -1024,11 +1024,11 @@ public class ProcessDao extends AbstractBaseDao { } /** - * ${processInstancePriority}_${processInstanceId}_${taskInstancePriority}_${taskId}_${task executed by ip1},${ip2}... + * ${processInstancePriority}_${processInstanceId}_${taskInstancePriority}_${taskInstanceId}_${task executed by ip1},${ip2}... * * The tasks with the highest priority are selected by comparing the priorities of the above four levels from high to low. * - * 流程实例优先级_流程实例id_任务优先级_任务id_任务执行机器ip1，ip2... high <- low + * 流程实例优先级_流程实例id_任务优先级_任务实例id_任务执行机器ip1，ip2... high <- low * * @param taskInstance * @return diff --git a/escheduler-dao/src/main/java/cn/escheduler/dao/mapper/ErrorCommandMapperProvider.java b/escheduler-dao/src/main/java/cn/escheduler/dao/mapper/ErrorCommandMapperProvider.java index 9ea05229ef..f4766e38ec 100644 --- a/escheduler-dao/src/main/java/cn/escheduler/dao/mapper/ErrorCommandMapperProvider.java +++ b/escheduler-dao/src/main/java/cn/escheduler/dao/mapper/ErrorCommandMapperProvider.java @@ -1,3 +1,19 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ package cn.escheduler.dao.mapper; import cn.escheduler.common.enums.*; diff --git a/escheduler-dao/src/main/java/cn/escheduler/dao/mapper/ProcessInstanceMapperProvider.java b/escheduler-dao/src/main/java/cn/escheduler/dao/mapper/ProcessInstanceMapperProvider.java index 78165e3f9a..9bbd3e336e 100644 --- a/escheduler-dao/src/main/java/cn/escheduler/dao/mapper/ProcessInstanceMapperProvider.java +++ b/escheduler-dao/src/main/java/cn/escheduler/dao/mapper/ProcessInstanceMapperProvider.java @@ -354,7 +354,7 @@ public class ProcessInstanceMapperProvider { * @return */ public String listByStatus(Map parameter) { - StringBuffer strStates = new StringBuffer(); + StringBuilder strStates = new StringBuilder(); int[] stateArray = (int[]) parameter.get("states"); for(int i=0;i parameter) { - StringBuffer strStates = new StringBuffer(); + StringBuilder strStates = new StringBuilder(); int[] stateArray = (int[]) parameter.get("states"); for(int i=0;i parameter) { - StringBuffer strStates = new StringBuffer(); + StringBuilder strStates = new StringBuilder(); int[] stateArray = (int[]) parameter.get("states"); for(int i=0;i parameter) { - StringBuffer strStates = new StringBuffer(); + StringBuilder strStates = new StringBuilder(); int[] stateArray = (int[]) parameter.get("states"); for(int i=0;i parameter) { - StringBuffer strIds = new StringBuffer(); + StringBuilder strIds = new StringBuilder(); int[] idsArray = (int[]) parameter.get("processDefineIds"); for(int i=0;i parameter) { - StringBuffer strStates = new StringBuffer(); + StringBuilder strStates = new StringBuilder(); int[] stateArray = (int[]) parameter.get("states"); for(int i=0;i parameter) { - StringBuffer strStates = new StringBuffer(); + StringBuilder strStates = new StringBuilder(); int[] stateArray = (int[]) parameter.get("states"); for(int i=0;i parameter) { - StringBuffer strStates = new StringBuffer(); + StringBuilder strStates = new StringBuilder(); int[] stateArray = (int[]) parameter.get("states"); int state = ExecutionStatus.NEED_FAULT_TOLERANCE.ordinal(); for(int i=0;i parameter){ - StringBuffer taskIdsStr = new StringBuffer(); + StringBuilder taskIdsStr = new StringBuilder(); int[] stateArray = (int[]) parameter.get("taskIds"); for(int i=0;i METHOD_ROLL_VIEW_LOG = getRollViewLogMethod(); @@ -46,7 +62,7 @@ public final class LogViewServiceGrpc { if ((getRollViewLogMethod = LogViewServiceGrpc.getRollViewLogMethod) == null) { synchronized (LogViewServiceGrpc.class) { if ((getRollViewLogMethod = LogViewServiceGrpc.getRollViewLogMethod) == null) { - LogViewServiceGrpc.getRollViewLogMethod = getRollViewLogMethod = + LogViewServiceGrpc.getRollViewLogMethod = getRollViewLogMethod = io.grpc.MethodDescriptor.newBuilder() .setType(io.grpc.MethodDescriptor.MethodType.UNARY) .setFullMethodName(generateFullMethodName( @@ -64,7 +80,7 @@ public final class LogViewServiceGrpc { return getRollViewLogMethod; } @io.grpc.ExperimentalApi("https://github.com/grpc/grpc-java/issues/1901") - @java.lang.Deprecated // Use {@link #getViewLogMethod()} instead. + @java.lang.Deprecated // Use {@link #getViewLogMethod()} instead. public static final io.grpc.MethodDescriptor METHOD_VIEW_LOG = getViewLogMethod(); @@ -78,7 +94,7 @@ public final class LogViewServiceGrpc { if ((getViewLogMethod = LogViewServiceGrpc.getViewLogMethod) == null) { synchronized (LogViewServiceGrpc.class) { if ((getViewLogMethod = LogViewServiceGrpc.getViewLogMethod) == null) { - LogViewServiceGrpc.getViewLogMethod = getViewLogMethod = + LogViewServiceGrpc.getViewLogMethod = getViewLogMethod = io.grpc.MethodDescriptor.newBuilder() .setType(io.grpc.MethodDescriptor.MethodType.UNARY) .setFullMethodName(generateFullMethodName( @@ -96,7 +112,7 @@ public final class LogViewServiceGrpc { return getViewLogMethod; } @io.grpc.ExperimentalApi("https://github.com/grpc/grpc-java/issues/1901") - @java.lang.Deprecated // Use {@link #getGetLogBytesMethod()} instead. + @java.lang.Deprecated // Use {@link #getGetLogBytesMethod()} instead. public static final io.grpc.MethodDescriptor METHOD_GET_LOG_BYTES = getGetLogBytesMethod(); @@ -110,7 +126,7 @@ public final class LogViewServiceGrpc { if ((getGetLogBytesMethod = LogViewServiceGrpc.getGetLogBytesMethod) == null) { synchronized (LogViewServiceGrpc.class) { if ((getGetLogBytesMethod = LogViewServiceGrpc.getGetLogBytesMethod) == null) { - LogViewServiceGrpc.getGetLogBytesMethod = getGetLogBytesMethod = + LogViewServiceGrpc.getGetLogBytesMethod = getGetLogBytesMethod = io.grpc.MethodDescriptor.newBuilder() .setType(io.grpc.MethodDescriptor.MethodType.UNARY) .setFullMethodName(generateFullMethodName( diff --git a/escheduler-server/src/main/java/cn/escheduler/server/rpc/LoggerServer.java b/escheduler-server/src/main/java/cn/escheduler/server/rpc/LoggerServer.java index 1dde113545..19fae81b56 100644 --- a/escheduler-server/src/main/java/cn/escheduler/server/rpc/LoggerServer.java +++ b/escheduler-server/src/main/java/cn/escheduler/server/rpc/LoggerServer.java @@ -98,7 +98,6 @@ public class LoggerServer { request.getLimit()); List list = readFile(request.getPath(), request.getSkipLineNum(), request.getLimit()); StringBuilder sb = new StringBuilder(); - boolean errorLineFlag = false; for (String line : list){ sb.append(line + "\r\n"); } @@ -196,7 +195,6 @@ public class LoggerServer { StringBuilder sb = new StringBuilder(); try { br = new BufferedReader(new InputStreamReader(new FileInputStream(path))); - boolean errorLineFlag = false; while ((line = br.readLine()) != null){ sb.append(line + "\r\n"); } diff --git a/escheduler-server/src/main/java/cn/escheduler/server/utils/FlinkArgsUtils.java b/escheduler-server/src/main/java/cn/escheduler/server/utils/FlinkArgsUtils.java new file mode 100644 index 0000000000..b7b69db78f --- /dev/null +++ b/escheduler-server/src/main/java/cn/escheduler/server/utils/FlinkArgsUtils.java @@ -0,0 +1,124 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package cn.escheduler.server.utils; + + +import cn.escheduler.common.Constants; +import cn.escheduler.common.enums.ProgramType; +import cn.escheduler.common.task.flink.FlinkParameters; +import org.apache.commons.lang.StringUtils; +import org.slf4j.LoggerFactory; + +import java.util.ArrayList; +import java.util.List; + + +/** + * spark args utils + */ +public class FlinkArgsUtils { + + /** + * build args + * + * @param param + * @return + */ + private static final org.slf4j.Logger logger = LoggerFactory.getLogger(FlinkArgsUtils.class); + + public static List buildArgs(FlinkParameters param) { + List args = new ArrayList<>(); + String deployMode = "cluster"; + if (StringUtils.isNotEmpty(param.getDeployMode())) { + deployMode = param.getDeployMode(); + + } + if (!"local".equals(deployMode)) { + args.add(Constants.FLINK_RUN_MODE); //-m + + args.add(Constants.FLINK_YARN_CLUSTER); //yarn-cluster + + + if (param.getSlot() != 0) { + args.add(Constants.FLINK_YARN_SLOT); + args.add(String.format("%d", param.getSlot())); //-ys + } + + if (StringUtils.isNotEmpty(param.getAppName())) { //-ynm + args.add(Constants.FLINK_APP_NAME); + args.add(param.getAppName()); + } + + if (param.getTaskManager() != 0) { //-yn + args.add(Constants.FLINK_TASK_MANAGE); + args.add(String.format("%d", param.getTaskManager())); + } + + if (StringUtils.isNotEmpty(param.getJobManagerMemory())) { + args.add(Constants.FLINK_JOB_MANAGE_MEM); + args.add(param.getJobManagerMemory()); //-yjm + } + + if (StringUtils.isNotEmpty(param.getTaskManagerMemory())) { // -ytm + args.add(Constants.FLINK_TASK_MANAGE_MEM); + args.add(param.getTaskManagerMemory()); + } + + args.add(Constants.FLINK_detach); //-d + + + } + + if (param.getProgramType() != null) { + if (param.getProgramType() != ProgramType.PYTHON) { + if (StringUtils.isNotEmpty(param.getMainClass())) { + args.add(Constants.FLINK_MAIN_CLASS); //-c + args.add(param.getMainClass()); //main class + } + } + } + + if (param.getMainJar() != null) { + args.add(param.getMainJar().getRes()); + } + + if (StringUtils.isNotEmpty(param.getMainArgs())) { + args.add(param.getMainArgs()); + } + + // --files --conf --libjar ... + if (StringUtils.isNotEmpty(param.getOthers())) { + String others = param.getOthers(); + if (!others.contains("--qu")) { + if (StringUtils.isNotEmpty(param.getQueue()) && !deployMode.equals("local")) { + args.add(Constants.FLINK_QUEUE); + args.add(param.getQueue()); + } + } + args.add(param.getOthers()); + } else if (StringUtils.isNotEmpty(param.getQueue()) && !deployMode.equals("local")) { + args.add(Constants.FLINK_QUEUE); + args.add(param.getQueue()); + + } + + + return args; + } + + +} \ No newline at end of file diff --git a/escheduler-server/src/main/java/cn/escheduler/server/worker/task/AbstractTask.java b/escheduler-server/src/main/java/cn/escheduler/server/worker/task/AbstractTask.java index 213f4fd3f9..6472873d8b 100644 --- a/escheduler-server/src/main/java/cn/escheduler/server/worker/task/AbstractTask.java +++ b/escheduler-server/src/main/java/cn/escheduler/server/worker/task/AbstractTask.java @@ -22,6 +22,7 @@ import cn.escheduler.common.enums.TaskRecordStatus; import cn.escheduler.common.enums.TaskType; import cn.escheduler.common.process.Property; import cn.escheduler.common.task.AbstractParameters; +import cn.escheduler.common.task.flink.FlinkParameters; import cn.escheduler.common.task.mr.MapreduceParameters; import cn.escheduler.common.task.procedure.ProcedureParameters; import cn.escheduler.common.task.python.PythonParameters; @@ -178,6 +179,8 @@ public abstract class AbstractTask { case SPARK: paramsClass = SparkParameters.class; break; + case FLINK: + paramsClass = FlinkParameters.class; case PYTHON: paramsClass = PythonParameters.class; break; diff --git a/escheduler-server/src/main/java/cn/escheduler/server/worker/task/TaskManager.java b/escheduler-server/src/main/java/cn/escheduler/server/worker/task/TaskManager.java index e23a29ae08..de96ff25be 100644 --- a/escheduler-server/src/main/java/cn/escheduler/server/worker/task/TaskManager.java +++ b/escheduler-server/src/main/java/cn/escheduler/server/worker/task/TaskManager.java @@ -19,6 +19,8 @@ package cn.escheduler.server.worker.task; import cn.escheduler.common.enums.TaskType; import cn.escheduler.server.worker.task.dependent.DependentTask; +import cn.escheduler.server.worker.task.flink.FlinkTask; +import cn.escheduler.server.worker.task.http.HttpTask; import cn.escheduler.server.worker.task.mr.MapReduceTask; import cn.escheduler.server.worker.task.processdure.ProcedureTask; import cn.escheduler.server.worker.task.python.PythonTask; @@ -55,10 +57,14 @@ public class TaskManager { return new MapReduceTask(props, logger); case SPARK: return new SparkTask(props, logger); + case FLINK: + return new FlinkTask(props, logger); case PYTHON: return new PythonTask(props, logger); case DEPENDENT: return new DependentTask(props, logger); + case HTTP: + return new HttpTask(props, logger); default: logger.error("unsupport task type: {}", taskType); throw new IllegalArgumentException("not support task type"); diff --git a/escheduler-server/src/main/java/cn/escheduler/server/worker/task/flink/FlinkTask.java b/escheduler-server/src/main/java/cn/escheduler/server/worker/task/flink/FlinkTask.java new file mode 100644 index 0000000000..bf6f0cc9fb --- /dev/null +++ b/escheduler-server/src/main/java/cn/escheduler/server/worker/task/flink/FlinkTask.java @@ -0,0 +1,118 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package cn.escheduler.server.worker.task.flink; + +import cn.escheduler.common.process.Property; +import cn.escheduler.common.task.AbstractParameters; +import cn.escheduler.common.task.flink.FlinkParameters; +import cn.escheduler.common.utils.JSONUtils; +import cn.escheduler.common.utils.ParameterUtils; +import cn.escheduler.dao.model.ProcessInstance; +import cn.escheduler.server.utils.FlinkArgsUtils; +import cn.escheduler.server.utils.ParamUtils; +import cn.escheduler.server.worker.task.AbstractYarnTask; +import cn.escheduler.server.worker.task.TaskProps; +import org.apache.commons.lang3.StringUtils; +import org.slf4j.Logger; + +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +/** + * flink task + */ +public class FlinkTask extends AbstractYarnTask { + + /** + * flink command + */ + private static final String FLINK_COMMAND = "flink"; + private static final String FLINK_RUN = "run"; + + /** + * flink parameters + */ + private FlinkParameters flinkParameters; + + public FlinkTask(TaskProps props, Logger logger) { + super(props, logger); + } + + @Override + public void init() { + + logger.info("flink task params {}", taskProps.getTaskParams()); + + flinkParameters = JSONUtils.parseObject(taskProps.getTaskParams(), FlinkParameters.class); + + if (!flinkParameters.checkParameters()) { + throw new RuntimeException("flink task params is not valid"); + } + flinkParameters.setQueue(taskProps.getQueue()); + + if (StringUtils.isNotEmpty(flinkParameters.getMainArgs())) { + String args = flinkParameters.getMainArgs(); + // get process instance by task instance id + ProcessInstance processInstance = processDao.findProcessInstanceByTaskId(taskProps.getTaskInstId()); + + /** + * combining local and global parameters + */ + Map paramsMap = ParamUtils.convert(taskProps.getUserDefParamsMap(), + taskProps.getDefinedParams(), + flinkParameters.getLocalParametersMap(), + processInstance.getCmdTypeIfComplement(), + processInstance.getScheduleTime()); + + logger.info("param Map : {}", paramsMap); + if (paramsMap != null ){ + + args = ParameterUtils.convertParameterPlaceholders(args, ParamUtils.convert(paramsMap)); + logger.info("param args : {}", args); + } + flinkParameters.setMainArgs(args); + } + } + + /** + * create command + * @return + */ + @Override + protected String buildCommand() { + List args = new ArrayList<>(); + + args.add(FLINK_COMMAND); + args.add(FLINK_RUN); + logger.info("flink task args : {}", args); + // other parameters + args.addAll(FlinkArgsUtils.buildArgs(flinkParameters)); + + String command = ParameterUtils + .convertParameterPlaceholders(String.join(" ", args), taskProps.getDefinedParams()); + + logger.info("flink task command : {}", command); + + return command; + } + + @Override + public AbstractParameters getParameters() { + return flinkParameters; + } +} diff --git a/escheduler-server/src/main/java/cn/escheduler/server/worker/task/http/HttpTask.java b/escheduler-server/src/main/java/cn/escheduler/server/worker/task/http/HttpTask.java new file mode 100644 index 0000000000..552e4bd828 --- /dev/null +++ b/escheduler-server/src/main/java/cn/escheduler/server/worker/task/http/HttpTask.java @@ -0,0 +1,270 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package cn.escheduler.server.worker.task.http; + + +import cn.escheduler.common.enums.HttpMethod; +import cn.escheduler.common.enums.HttpParametersType; +import cn.escheduler.common.process.HttpProperty; +import cn.escheduler.common.process.Property; +import cn.escheduler.common.task.AbstractParameters; +import cn.escheduler.common.task.http.HttpParameters; +import cn.escheduler.common.utils.Bytes; +import cn.escheduler.common.utils.DateUtils; +import cn.escheduler.common.utils.ParameterUtils; +import cn.escheduler.dao.DaoFactory; +import cn.escheduler.dao.ProcessDao; +import cn.escheduler.dao.model.ProcessInstance; +import cn.escheduler.server.utils.ParamUtils; +import cn.escheduler.server.worker.task.AbstractTask; +import cn.escheduler.server.worker.task.TaskProps; +import com.alibaba.fastjson.JSONObject; +import org.apache.commons.io.Charsets; +import org.apache.commons.lang3.StringUtils; +import org.apache.http.HttpEntity; +import org.apache.http.ParseException; +import org.apache.http.client.config.RequestConfig; +import org.apache.http.client.methods.CloseableHttpResponse; +import org.apache.http.client.methods.HttpUriRequest; +import org.apache.http.client.methods.RequestBuilder; +import org.apache.http.entity.StringEntity; +import org.apache.http.impl.client.CloseableHttpClient; +import org.apache.http.impl.client.HttpClientBuilder; +import org.apache.http.impl.client.HttpClients; +import org.apache.http.util.EntityUtils; +import org.slf4j.Logger; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +/** + * http task + */ +public class HttpTask extends AbstractTask { + + private HttpParameters httpParameters; + + /** + * process database access + */ + private ProcessDao processDao; + + /** + * Convert mill seconds to second unit + */ + protected static final int MAX_CONNECTION_MILLISECONDS = 60000; + + protected static final String APPLICATION_JSON = "application/json"; + + protected String output; + + + public HttpTask(TaskProps props, Logger logger) { + super(props, logger); + this.processDao = DaoFactory.getDaoInstance(ProcessDao.class); + } + + @Override + public void init() { + logger.info("http task params {}", taskProps.getTaskParams()); + this.httpParameters = JSONObject.parseObject(taskProps.getTaskParams(), HttpParameters.class); + + if (!httpParameters.checkParameters()) { + throw new RuntimeException("http task params is not valid"); + } + } + + @Override + public void handle() throws Exception { + String threadLoggerInfoName = String.format("TaskLogInfo-%s", taskProps.getTaskAppId()); + Thread.currentThread().setName(threadLoggerInfoName); + + long startTime = System.currentTimeMillis(); + String statusCode = null; + String body = null; + try(CloseableHttpClient client = createHttpClient()) { + try(CloseableHttpResponse response = sendRequest(client)) { + statusCode = String.valueOf(getStatusCode(response)); + body = getResponseBody(response); + exitStatusCode = validResponse(body, statusCode); + long costTime = System.currentTimeMillis() - startTime; + logger.info("startTime: {}, httpUrl: {}, httpMethod: {}, costTime : {}Millisecond, statusCode : {}, body : {}, log : {}", + DateUtils.format2Readable(startTime), httpParameters.getUrl(),httpParameters.getHttpMethod(), costTime, statusCode, body, output); + }catch (Exception e) { + appendMessage(e.toString()); + exitStatusCode = -1; + logger.error("httpUrl[" + httpParameters.getUrl() + "] connection failed："+output, e); + } + } catch (Exception e) { + appendMessage(e.toString()); + exitStatusCode = -1; + logger.error("httpUrl[" + httpParameters.getUrl() + "] connection failed："+output, e); + } + } + + protected CloseableHttpResponse sendRequest(CloseableHttpClient client) throws IOException { + RequestBuilder builder = createRequestBuilder(); + ProcessInstance processInstance = processDao.findProcessInstanceByTaskId(taskProps.getTaskInstId()); + + Map paramsMap = ParamUtils.convert(taskProps.getUserDefParamsMap(), + taskProps.getDefinedParams(), + httpParameters.getLocalParametersMap(), + processInstance.getCmdTypeIfComplement(), + processInstance.getScheduleTime()); + List httpPropertyList = new ArrayList<>(); + if(httpParameters.getHttpParams() != null && httpParameters.getHttpParams().size() > 0){ + for (HttpProperty httpProperty: httpParameters.getHttpParams()) { + String jsonObject = JSONObject.toJSONString(httpProperty); + String params = ParameterUtils.convertParameterPlaceholders(jsonObject,ParamUtils.convert(paramsMap)); + logger.info("http request params：{}",params); + httpPropertyList.add(JSONObject.parseObject(params,HttpProperty.class)); + } + } + addRequestParams(builder,httpPropertyList); + HttpUriRequest request = builder.setUri(httpParameters.getUrl()).build(); + setHeaders(request,httpPropertyList); + return client.execute(request); + } + + protected String getResponseBody(CloseableHttpResponse httpResponse) throws ParseException, IOException { + if (httpResponse == null) { + return null; + } + HttpEntity entity = httpResponse.getEntity(); + if (entity == null) { + return null; + } + String webPage = EntityUtils.toString(entity, Bytes.UTF8_ENCODING); + return webPage; + } + + protected int getStatusCode(CloseableHttpResponse httpResponse) { + int status = httpResponse.getStatusLine().getStatusCode(); + return status; + } + + protected int validResponse(String body, String statusCode){ + int exitStatusCode = 0; + switch (httpParameters.getHttpCheckCondition()) { + case BODY_CONTAINS: + if (StringUtils.isEmpty(body) || !body.contains(httpParameters.getCondition())) { + appendMessage(httpParameters.getUrl() + " doesn contain " + + httpParameters.getCondition()); + exitStatusCode = -1; + } + break; + case BODY_NOT_CONTAINS: + if (StringUtils.isEmpty(body) || body.contains(httpParameters.getCondition())) { + appendMessage(httpParameters.getUrl() + " contains " + + httpParameters.getCondition()); + exitStatusCode = -1; + } + break; + case STATUS_CODE_CUSTOM: + if (!statusCode.equals(httpParameters.getCondition())) { + appendMessage(httpParameters.getUrl() + " statuscode: " + statusCode + ", Must be: " + httpParameters.getCondition()); + exitStatusCode = -1; + } + break; + default: + if (!"200".equals(statusCode)) { + appendMessage(httpParameters.getUrl() + " statuscode: " + statusCode + ", Must be: 200"); + exitStatusCode = -1; + } + break; + } + return exitStatusCode; + } + + public String getOutput() { + return output; + } + + protected void appendMessage(String message) { + if (output == null) { + output = ""; + } + if (message != null && !message.trim().isEmpty()) { + output += message; + } + } + + protected void addRequestParams(RequestBuilder builder,List httpPropertyList) { + if(httpPropertyList != null && httpPropertyList.size() > 0){ + JSONObject jsonParam = new JSONObject(); + for (HttpProperty property: httpPropertyList){ + if(property.getHttpParametersType() != null){ + if (property.getHttpParametersType().equals(HttpParametersType.PARAMETER)){ + builder.addParameter(property.getProp(), property.getValue()); + }else if(property.getHttpParametersType().equals(HttpParametersType.BODY)){ + jsonParam.put(property.getProp(), property.getValue()); + } + } + } + StringEntity postingString = new StringEntity(jsonParam.toString(), Charsets.UTF_8); + postingString.setContentEncoding(Bytes.UTF8_ENCODING); + postingString.setContentType(APPLICATION_JSON); + builder.setEntity(postingString); + } + } + + protected void setHeaders(HttpUriRequest request,List httpPropertyList) { + if(httpPropertyList != null && httpPropertyList.size() > 0){ + for (HttpProperty property: httpPropertyList){ + if(property.getHttpParametersType() != null) { + if (property.getHttpParametersType().equals(HttpParametersType.HEADERS)) { + request.addHeader(property.getProp(), property.getValue()); + } + } + } + } + } + + protected CloseableHttpClient createHttpClient() { + final RequestConfig requestConfig = requestConfig(); + HttpClientBuilder httpClientBuilder; + httpClientBuilder = HttpClients.custom().setDefaultRequestConfig(requestConfig); + return httpClientBuilder.build(); + } + + private RequestConfig requestConfig() { + return RequestConfig.custom().setSocketTimeout(MAX_CONNECTION_MILLISECONDS).setConnectTimeout(MAX_CONNECTION_MILLISECONDS).build(); + } + + protected RequestBuilder createRequestBuilder() { + if (httpParameters.getHttpMethod().equals(HttpMethod.GET)) { + return RequestBuilder.get(); + } else if (httpParameters.getHttpMethod().equals(HttpMethod.POST)) { + return RequestBuilder.post(); + } else if (httpParameters.getHttpMethod().equals(HttpMethod.HEAD)) { + return RequestBuilder.head(); + } else if (httpParameters.getHttpMethod().equals(HttpMethod.PUT)) { + return RequestBuilder.put(); + } else if (httpParameters.getHttpMethod().equals(HttpMethod.DELETE)) { + return RequestBuilder.delete(); + } else { + return null; + } + } + + @Override + public AbstractParameters getParameters() { + return this.httpParameters; + } +} diff --git a/escheduler-server/src/main/resources/application_worker.properties b/escheduler-server/src/main/resources/application_worker.properties index 39983e4fba..b7e3c88b24 100644 --- a/escheduler-server/src/main/resources/application_worker.properties +++ b/escheduler-server/src/main/resources/application_worker.properties @@ -1,4 +1,4 @@ -logging.config=classpath:master_logback.xml +logging.config=classpath:worker_logback.xml # server port server.port=7788 diff --git a/escheduler-server/src/test/java/cn/escheduler/server/worker/EnvFileTest.java b/escheduler-server/src/test/java/cn/escheduler/server/worker/EnvFileTest.java index 341a74e20c..7e75babe08 100644 --- a/escheduler-server/src/test/java/cn/escheduler/server/worker/EnvFileTest.java +++ b/escheduler-server/src/test/java/cn/escheduler/server/worker/EnvFileTest.java @@ -1,3 +1,19 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ package cn.escheduler.server.worker; import org.apache.commons.lang.StringUtils; diff --git a/escheduler-server/src/test/java/cn/escheduler/server/zk/ZKWorkerClientTest.java b/escheduler-server/src/test/java/cn/escheduler/server/zk/ZKWorkerClientTest.java index 005b338cb7..8c08cab965 100644 --- a/escheduler-server/src/test/java/cn/escheduler/server/zk/ZKWorkerClientTest.java +++ b/escheduler-server/src/test/java/cn/escheduler/server/zk/ZKWorkerClientTest.java @@ -1,3 +1,19 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ package cn.escheduler.server.zk; import cn.escheduler.common.Constants; @@ -34,4 +50,4 @@ public class ZKWorkerClientTest { Assert.assertEquals(1, ipList.size()); } -} \ No newline at end of file +} diff --git a/escheduler-ui/build/webpack.config.prod.js b/escheduler-ui/build/webpack.config.prod.js index 595de11076..4a287feb8e 100644 --- a/escheduler-ui/build/webpack.config.prod.js +++ b/escheduler-ui/build/webpack.config.prod.js @@ -100,8 +100,10 @@ const config = merge.smart(baseConfig, { sourceMap: true, uglifyOptions: { compress: { + warnings: false, + drop_debugger: true, drop_console: true, - drop_debugger: true + pure_funcs: ['console.log']//移除console }, comments: function (n, c) { /*! IMPORTANT: Please preserve 3rd-party library license info, inspired from @allex/amd-build-worker/config/jsplumb.js */ diff --git a/escheduler-ui/package.json b/escheduler-ui/package.json index c6a2a92e40..3a8d532e01 100644 --- a/escheduler-ui/package.json +++ b/escheduler-ui/package.json @@ -1,5 +1,5 @@ { - "name": "escheduler", + "name": "dolphinscheduler", "version": "1.0.0", "description": "调度平台前端项目", "author": "gongzijian ", @@ -23,7 +23,6 @@ "babel-plugin-syntax-jsx": "^6.18.0", "babel-plugin-transform-class-properties": "^6.24.1", "babel-plugin-transform-object-rest-spread": "^6.26.0", - "babel-plugin-transform-remove-console": "^6.9.4", "babel-plugin-transform-runtime": "^6.23.0", "babel-plugin-transform-vue-jsx": "^3.5.0", "babel-preset-env": "^1.6.1", diff --git a/escheduler-ui/src/combo/1.0.0/3rd.css b/escheduler-ui/src/combo/1.0.0/3rd.css index 658a682e97..31ad26482d 100644 --- a/escheduler-ui/src/combo/1.0.0/3rd.css +++ b/escheduler-ui/src/combo/1.0.0/3rd.css @@ -1,3 +1,19 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ .hljs{display:block;overflow-x:auto;padding:0.5em;background:white;color:black}.hljs-comment,.hljs-quote,.hljs-variable{color:#008000}.hljs-keyword,.hljs-selector-tag,.hljs-built_in,.hljs-name,.hljs-tag{color:#00f}.hljs-string,.hljs-title,.hljs-section,.hljs-attribute,.hljs-literal,.hljs-template-tag,.hljs-template-variable,.hljs-type,.hljs-addition{color:#a31515}.hljs-deletion,.hljs-selector-attr,.hljs-selector-pseudo,.hljs-meta{color:#2b91af}.hljs-doctag{color:#808080}.hljs-attr{color:#f00}.hljs-symbol,.hljs-bullet,.hljs-link{color:#00b0e8}.hljs-emphasis{font-style:italic}.hljs-strong{font-weight:bold} .jtk-node{position:absolute}.jtk-group{position:absolute;overflow:visible}[jtk-group-content]{position:relative}.katavorio-clone-drag{pointer-events:none}.jtk-surface{overflow:hidden!important;position:relative;cursor:move;cursor:-moz-grab;cursor:-webkit-grab;touch-action:none}.jtk-surface-panning{cursor:-moz-grabbing;cursor:-webkit-grabbing;-webkit-touch-callout:none;-webkit-user-select:none;-khtml-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none}.jtk-surface-canvas{overflow:visible!important}.jtk-surface-droppable-node{touch-action:none}.jtk-surface-nopan{overflow:scroll!important;cursor:default}.jtk-surface-tile{border:none;outline:0;margin:0;-webkit-transition:opacity .3s ease .15s;-moz-transition:opacity .3s ease .15s;-o-transition:opacity .3s ease .15s;-ms-transition:opacity .3s ease .15s;transition:opacity .3s ease .15s}.jtk-lasso{border:2px solid #3177b8;background-color:#f5f5f5;opacity:.5;display:none;z-index:20000;position:absolute}.jtk-lasso-select-defeat *{-webkit-touch-callout:none;-webkit-user-select:none;-khtml-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none}.jtk-lasso-mask{position:fixed;z-index:20000;display:none;opacity:.5;background-color:#07234e;top:0;bottom:0;left:0;right:0}.jtk-surface-selected-element{border:2px dashed #f76258!important}.jtk-surface-pan{background-color:Azure;opacity:.4;text-align:center;cursor:pointer;z-index:2;-webkit-transition:background-color .15s ease-in;-moz-transition:background-color .15s ease-in;-o-transition:background-color .15s ease-in;transition:background-color .15s ease-in}.jtk-surface-pan-bottom,.jtk-surface-pan-top{width:100%;height:20px}.jtk-surface-pan-bottom:hover,.jtk-surface-pan-left:hover,.jtk-surface-pan-right:hover,.jtk-surface-pan-top:hover{opacity:.6;background-color:#3177b8;color:#fff;font-weight:700}.jtk-surface-pan-left,.jtk-surface-pan-right{width:20px;height:100%;line-height:40}.jtk-surface-pan-active,.jtk-surface-pan-active:hover{background-color:#f76258}.jtk-miniview{overflow:hidden!important;width:125px;height:125px;position:relative;background-color:#b2c9cd;border:1px solid #e2e6cd;border-radius:4px;opacity:.8}.jtk-miniview-panner{border:5px dotted #f5f5f5;opacity:.4;background-color:#4f6f7e;cursor:move;cursor:-moz-grab;cursor:-webkit-grab}.jtk-miniview-panning{cursor:-moz-grabbing;cursor:-webkit-grabbing}.jtk-miniview-element{background-color:#607a86;position:absolute}.jtk-miniview-group-element{background:0 0;border:2px solid #607a86}.jtk-miniview-collapse{color:#f5f5f5;position:absolute;font-size:18px;top:-1px;right:3px;cursor:pointer;font-weight:700}.jtk-miniview-collapse:before{content:"\2012"}.jtk-miniview-collapsed{background-color:#449ea6;border-radius:4px;height:22px;margin-right:0;padding:4px;width:21px}.jtk-miniview-collapsed .jtk-miniview-element,.jtk-miniview-collapsed .jtk-miniview-panner{visibility:hidden}.jtk-miniview-collapsed .jtk-miniview-collapse:before{content:"+"}.jtk-miniview-collapse:hover{color:#e4f013}.jtk-dialog-underlay{left:0;right:0;top:0;bottom:0;position:fixed;z-index:100000;opacity:.8;background-color:#ccc;display:none}.jtk-dialog-overlay{position:fixed;z-index:100001;display:none;background-color:#fff;font-family:"Open Sans",sans-serif;padding:7px;box-shadow:0 0 5px gray;overflow:hidden}.jtk-dialog-overlay-x{max-height:0;transition:max-height .5s ease-in;-moz-transition:max-height .5s ease-in;-ms-transition:max-height .5s ease-in;-o-transition:max-height .5s ease-in;-webkit-transition:max-height .5s ease-in}.jtk-dialog-overlay-y{max-width:0;transition:max-width .5s ease-in;-moz-transition:max-width .5s ease-in;-ms-transition:max-width .5s ease-in;-o-transition:max-width .5s ease-in;-webkit-transition:max-width .5s ease-in}.jtk-dialog-overlay-top{top:20px}.jtk-dialog-overlay-bottom{bottom:20px}.jtk-dialog-overlay-left{left:20px}.jtk-dialog-overlay-right{right:20px}.jtk-dialog-overlay-x.jtk-dialog-overlay-visible{max-height:1000px}.jtk-dialog-overlay-y.jtk-dialog-overlay-visible{max-width:1000px}.jtk-dialog-buttons{text-align:right;margin-top:5px}.jtk-dialog-button{border:none;cursor:pointer;margin-right:5px;min-width:56px;background-color:#fff;outline:1px solid #ccc}.jtk-dialog-button:hover{color:#fff;background-color:#234b5e}.jtk-dialog-title{text-align:left;font-size:14px;margin-bottom:9px}.jtk-dialog-content{font-size:12px;text-align:left;min-width:250px;margin:0 14px}.jtk-dialog-content ul{width:100%;padding-left:0}.jtk-dialog-content label{cursor:pointer;font-weight:inherit}.jtk-dialog-overlay input,.jtk-dialog-overlay textarea{background-color:#fff;border:1px solid #ccc;color:#333;font-size:14px;font-style:normal;outline:0;padding:6px 4px;margin-right:6px}.jtk-dialog-overlay input:focus,.jtk-dialog-overlay textarea:focus{background-color:#cbeae1;border:1px solid #83b8a8;color:#333;font-size:14px;font-style:normal;outline:0}.jtk-draw-skeleton{position:absolute;left:0;right:0;top:0;bottom:0;outline:2px solid #84acb3;opacity:.8}.jtk-draw-handle{position:absolute;width:7px;height:7px;background-color:#84acb3}.jtk-draw-handle-tl{left:0;top:0;cursor:nw-resize}.jtk-draw-handle-tr{right:0;top:0;cursor:ne-resize}.jtk-draw-handle-bl{left:0;bottom:0;cursor:sw-resize}.jtk-draw-handle-br{bottom:0;right:0;cursor:se-resize}.jtk-draw-drag{display:none;position:absolute;left:50%;top:50%;margin-left:-10px;margin-top:-10px;width:20px;height:20px;background-color:#84acb3;cursor:move}.jtk-drag-select-defeat *{-webkit-touch-callout:none;-webkit-user-select:none;-khtml-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none} .CodeMirror{font-family:monospace;height:300px;color:#000;direction:ltr}.CodeMirror-lines{padding:4px 0}.CodeMirror pre{padding:0 4px}.CodeMirror-gutter-filler,.CodeMirror-scrollbar-filler{background-color:#fff}.CodeMirror-gutters{border-right:1px solid #ddd;background-color:#f7f7f7;white-space:nowrap}.CodeMirror-linenumber{padding:0 3px 0 5px;min-width:20px;text-align:right;color:#999;white-space:nowrap}.CodeMirror-guttermarker{color:#000}.CodeMirror-guttermarker-subtle{color:#999}.CodeMirror-cursor{border-left:1px solid #000;border-right:none;width:0}.CodeMirror div.CodeMirror-secondarycursor{border-left:1px solid silver}.cm-fat-cursor .CodeMirror-cursor{width:auto;border:0!important;background:#7e7}.cm-fat-cursor div.CodeMirror-cursors{z-index:1}.cm-fat-cursor-mark{background-color:rgba(20,255,20,.5);-webkit-animation:blink 1.06s steps(1) infinite;-moz-animation:blink 1.06s steps(1) infinite;animation:blink 1.06s steps(1) infinite}.cm-animate-fat-cursor{width:auto;border:0;-webkit-animation:blink 1.06s steps(1) infinite;-moz-animation:blink 1.06s steps(1) infinite;animation:blink 1.06s steps(1) infinite;background-color:#7e7}@-moz-keyframes blink{50%{background-color:transparent}}@-webkit-keyframes blink{50%{background-color:transparent}}@keyframes blink{50%{background-color:transparent}}.cm-tab{display:inline-block;text-decoration:inherit}.CodeMirror-rulers{position:absolute;left:0;right:0;top:-50px;bottom:-20px;overflow:hidden}.CodeMirror-ruler{border-left:1px solid #ccc;top:0;bottom:0;position:absolute}.cm-s-default .cm-header{color:#00f}.cm-s-default .cm-quote{color:#090}.cm-negative{color:#d44}.cm-positive{color:#292}.cm-header,.cm-strong{font-weight:700}.cm-em{font-style:italic}.cm-link{text-decoration:underline}.cm-strikethrough{text-decoration:line-through}.cm-s-default .cm-keyword{color:#708}.cm-s-default .cm-atom{color:#219}.cm-s-default .cm-number{color:#164}.cm-s-default .cm-def{color:#00f}.cm-s-default .cm-variable-2{color:#05a}.cm-s-default .cm-type,.cm-s-default .cm-variable-3{color:#085}.cm-s-default .cm-comment{color:#a50}.cm-s-default .cm-string{color:#a11}.cm-s-default .cm-string-2{color:#f50}.cm-s-default .cm-meta{color:#555}.cm-s-default .cm-qualifier{color:#555}.cm-s-default .cm-builtin{color:#30a}.cm-s-default .cm-bracket{color:#997}.cm-s-default .cm-tag{color:#170}.cm-s-default .cm-attribute{color:#00c}.cm-s-default .cm-hr{color:#999}.cm-s-default .cm-link{color:#00c}.cm-s-default .cm-error{color:red}.cm-invalidchar{color:red}.CodeMirror-composing{border-bottom:2px solid}div.CodeMirror span.CodeMirror-matchingbracket{color:#0b0}div.CodeMirror span.CodeMirror-nonmatchingbracket{color:#a22}.CodeMirror-matchingtag{background:rgba(255,150,0,.3)}.CodeMirror-activeline-background{background:#e8f2ff}.CodeMirror{position:relative;overflow:hidden;background:#fff}.CodeMirror-scroll{overflow:scroll!important;margin-bottom:-30px;margin-right:-30px;padding-bottom:30px;height:100%;outline:0;position:relative}.CodeMirror-sizer{position:relative;border-right:30px solid transparent}.CodeMirror-gutter-filler,.CodeMirror-hscrollbar,.CodeMirror-scrollbar-filler,.CodeMirror-vscrollbar{position:absolute;z-index:6;display:none}.CodeMirror-vscrollbar{right:0;top:0;overflow-x:hidden;overflow-y:scroll}.CodeMirror-hscrollbar{bottom:0;left:0;overflow-y:hidden;overflow-x:scroll}.CodeMirror-scrollbar-filler{right:0;bottom:0}.CodeMirror-gutter-filler{left:0;bottom:0}.CodeMirror-gutters{position:absolute;left:0;top:0;min-height:100%;z-index:3}.CodeMirror-gutter{white-space:normal;height:100%;display:inline-block;vertical-align:top;margin-bottom:-30px}.CodeMirror-gutter-wrapper{position:absolute;z-index:4;background:0 0!important;border:none!important}.CodeMirror-gutter-background{position:absolute;top:0;bottom:0;z-index:4}.CodeMirror-gutter-elt{position:absolute;cursor:default;z-index:4}.CodeMirror-gutter-wrapper ::selection{background-color:transparent}.CodeMirror-gutter-wrapper ::-moz-selection{background-color:transparent}.CodeMirror-lines{cursor:text;min-height:1px}.CodeMirror pre{-moz-border-radius:0;-webkit-border-radius:0;border-radius:0;border-width:0;background:0 0;font-family:inherit;font-size:inherit;margin:0;white-space:pre;word-wrap:normal;line-height:inherit;color:inherit;z-index:2;position:relative;overflow:visible;-webkit-tap-highlight-color:transparent;-webkit-font-variant-ligatures:contextual;font-variant-ligatures:contextual}.CodeMirror-wrap pre{word-wrap:break-word;white-space:pre-wrap;word-break:normal}.CodeMirror-linebackground{position:absolute;left:0;right:0;top:0;bottom:0;z-index:0}.CodeMirror-linewidget{position:relative;z-index:2;padding:.1px}.CodeMirror-rtl pre{direction:rtl}.CodeMirror-code{outline:0}.CodeMirror-gutter,.CodeMirror-gutters,.CodeMirror-linenumber,.CodeMirror-scroll,.CodeMirror-sizer{-moz-box-sizing:content-box;box-sizing:content-box}.CodeMirror-measure{position:absolute;width:100%;height:0;overflow:hidden;visibility:hidden}.CodeMirror-cursor{position:absolute;pointer-events:none}.CodeMirror-measure pre{position:static}div.CodeMirror-cursors{visibility:hidden;position:relative;z-index:3}div.CodeMirror-dragcursors{visibility:visible}.CodeMirror-focused div.CodeMirror-cursors{visibility:visible}.CodeMirror-selected{background:#d9d9d9}.CodeMirror-focused .CodeMirror-selected{background:#d7d4f0}.CodeMirror-crosshair{cursor:crosshair}.CodeMirror-line::selection,.CodeMirror-line>span::selection,.CodeMirror-line>span>span::selection{background:#d7d4f0}.CodeMirror-line::-moz-selection,.CodeMirror-line>span::-moz-selection,.CodeMirror-line>span>span::-moz-selection{background:#d7d4f0}.cm-searching{background-color:#ffa;background-color:rgba(255,255,0,.4)}.cm-force-border{padding-right:.1px}@media print{.CodeMirror div.CodeMirror-cursors{visibility:hidden}}.cm-tab-wrap-hack:after{content:''}span.CodeMirror-selectedtext{background:0 0} diff --git a/escheduler-ui/src/combo/1.0.0/local.js b/escheduler-ui/src/combo/1.0.0/local.js index ec6f9abe6c..1d0d62ecd5 100644 --- a/escheduler-ui/src/combo/1.0.0/local.js +++ b/escheduler-ui/src/combo/1.0.0/local.js @@ -1 +1,17 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ Not found \ No newline at end of file diff --git a/escheduler-ui/src/font/demo.css b/escheduler-ui/src/font/demo.css index a67054a0a0..9abf38fc19 100644 --- a/escheduler-ui/src/font/demo.css +++ b/escheduler-ui/src/font/demo.css @@ -1,3 +1,19 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ /* Logo 字体 */ @font-face { font-family: "iconfont logo"; diff --git a/escheduler-ui/src/font/iconfont.css b/escheduler-ui/src/font/iconfont.css index 77b5d0724d..0a7b84a393 100644 --- a/escheduler-ui/src/font/iconfont.css +++ b/escheduler-ui/src/font/iconfont.css @@ -1,3 +1,19 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ @font-face {font-family: "iconfont"; src: url('iconfont.eot?t=1554803141579'); /* IE9 */ src: url('iconfont.eot?t=1554803141579#iefix') format('embedded-opentype'), /* IE6-IE8 */ diff --git a/escheduler-ui/src/font/iconfont.js b/escheduler-ui/src/font/iconfont.js index 00ab236f61..14779bbc05 100644 --- a/escheduler-ui/src/font/iconfont.js +++ b/escheduler-ui/src/font/iconfont.js @@ -1 +1,17 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ !function(s){var c,h='',l=(c=document.getElementsByTagName("script"))[c.length-1].getAttribute("data-injectcss");if(l&&!s.__iconfont__svg__cssinject__){s.__iconfont__svg__cssinject__=!0;try{document.write("")}catch(c){console&&console.log(c)}}!function(c){if(document.addEventListener)if(~["complete","loaded","interactive"].indexOf(document.readyState))setTimeout(c,0);else{var l=function(){document.removeEventListener("DOMContentLoaded",l,!1),c()};document.addEventListener("DOMContentLoaded",l,!1)}else document.attachEvent&&(h=c,a=s.document,i=!1,o=function(){i||(i=!0,h())},(t=function(){try{a.documentElement.doScroll("left")}catch(c){return void setTimeout(t,50)}o()})(),a.onreadystatechange=function(){"complete"==a.readyState&&(a.onreadystatechange=null,o())});var h,a,i,o,t}(function(){var c,l;(c=document.createElement("div")).innerHTML=h,h=null,(l=c.getElementsByTagName("svg")[0])&&(l.setAttribute("aria-hidden","true"),l.style.position="absolute",l.style.width=0,l.style.height=0,l.style.overflow="hidden",function(c,l){l.firstChild?function(c,l){l.parentNode.insertBefore(c,l)}(c,l.firstChild):l.appendChild(c)}(l,document.body))})}(window); \ No newline at end of file diff --git a/escheduler-ui/src/js/conf/home/App.vue b/escheduler-ui/src/js/conf/home/App.vue index 65969452e9..97110e580c 100644 --- a/escheduler-ui/src/js/conf/home/App.vue +++ b/escheduler-ui/src/js/conf/home/App.vue @@ -1,3 +1,19 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */