lenboo
5 years ago
185 changed files with 4371 additions and 1647 deletions
@ -0,0 +1,77 @@
|
||||
Easy Scheduler |
||||
============ |
||||
[](https://www.apache.org/licenses/LICENSE-2.0.html) |
||||
|
||||
> Easy Scheduler for Big Data |
||||
|
||||
**设计特点：** 一个分布式易扩展的可视化DAG工作流任务调度系统。致力于解决数据处理流程中错综复杂的依赖关系，使调度系统在数据处理流程中`开箱即用`。 |
||||
其主要目标如下： |
||||
- 以DAG图的方式将Task按照任务的依赖关系关联起来，可实时可视化监控任务的运行状态 |
||||
- 支持丰富的任务类型：Shell、MR、Spark、SQL(mysql、postgresql、hive、sparksql),Python,Sub_Process、Procedure等 |
||||
- 支持工作流定时调度、依赖调度、手动调度、手动暂停/停止/恢复，同时支持失败重试/告警、从指定节点恢复失败、Kill任务等操作 |
||||
- 支持工作流优先级、任务优先级及任务的故障转移及任务超时告警/失败 |
||||
- 支持工作流全局参数及节点自定义参数设置 |
||||
- 支持资源文件的在线上传/下载，管理等，支持在线文件创建、编辑 |
||||
- 支持任务日志在线查看及滚动、在线下载日志等 |
||||
- 实现集群HA，通过Zookeeper实现Master集群和Worker集群去中心化 |
||||
- 支持对`Master/Worker` cpu load，memory，cpu在线查看 |
||||
- 支持工作流运行历史树形/甘特图展示、支持任务状态统计、流程状态统计 |
||||
- 支持补数 |
||||
- 支持多租户 |
||||
- 支持国际化 |
||||
- 还有更多等待伙伴们探索 |
||||
|
||||
### 与同类调度系统的对比 |
||||
|
||||
 |
||||
|
||||
### 系统部分截图 |
||||
|
||||
 |
||||
|
||||
 |
||||
|
||||
 |
||||
|
||||
### 文档 |
||||
|
||||
- <a href="https://analysys.github.io/easyscheduler_docs_cn/后端部署文档.html" target="_blank">后端部署文档</a> |
||||
|
||||
- <a href="https://analysys.github.io/easyscheduler_docs_cn/前端部署文档.html" target="_blank">前端部署文档</a> |
||||
|
||||
- [**使用手册**](https://analysys.github.io/easyscheduler_docs_cn/系统使用手册.html?_blank "系统使用手册") |
||||
|
||||
- [**升级文档**](https://analysys.github.io/easyscheduler_docs_cn/升级文档.html?_blank "升级文档") |
||||
|
||||
- <a href="http://52.82.13.76:8888" target="_blank">我要体验</a> |
||||
|
||||
更多文档请参考 <a href="https://analysys.github.io/easyscheduler_docs_cn/" target="_blank">easyscheduler中文在线文档</a> |
||||
|
||||
|
||||
### 近期研发计划 |
||||
|
||||
EasyScheduler的工作计划：<a href="https://github.com/analysys/EasyScheduler/projects/1" target="_blank">研发计划</a> ，其中 In Develop卡片下是1.1.0版本的功能，TODO卡片是待做事项(包括 feature ideas) |
||||
|
||||
### 贡献代码 |
||||
|
||||
非常欢迎大家来参与贡献代码，提交代码流程请参考： |
||||
https://github.com/analysys/EasyScheduler/blob/master/CONTRIBUTING.md |
||||
|
||||
|
||||
### 感谢 |
||||
|
||||
Easy Scheduler使用了很多优秀的开源项目，比如google的guava、guice、grpc，netty，ali的bonecp，quartz，以及apache的众多开源项目等等， |
||||
正是由于站在这些开源项目的肩膀上，才有Easy Scheduler的诞生的可能。对此我们对使用的所有开源软件表示非常的感谢！我们也希望自己不仅是开源的受益者，也能成为开源的 |
||||
贡献者，于是我们决定把易调度贡献出来，并承诺长期维护。也希望对开源有同样热情和信念的伙伴加入进来，一起为开源献出一份力！ |
||||
|
||||
|
||||
### 帮助 |
||||
The fastest way to get response from our developers is to submit issues, or add our wechat : 510570367 |
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@ -0,0 +1,31 @@
|
||||
Easy Scheduler Release 1.0.4 |
||||
=== |
||||
Easy Scheduler 1.0.4是1.x系列中的第五个版本。 |
||||
|
||||
增强： |
||||
=== |
||||
- [[EasyScheduler-482]](https://github.com/analysys/EasyScheduler/issues/482)sql任务中的邮件标题增加了对自定义变量的支持 |
||||
- [[EasyScheduler-483]](https://github.com/analysys/EasyScheduler/issues/483)sql任务中的发邮件失败,则此sql任务为失败 |
||||
- [[EasyScheduler-484]](https://github.com/analysys/EasyScheduler/issues/484)修改sql任务中自定义变量的替换规则,支持多个单引号和双引号的替换 |
||||
- [[EasyScheduler-485]](https://github.com/analysys/EasyScheduler/issues/485)创建资源文件时，增加对该资源文件是否在hdfs上已存在的验证 |
||||
- [[EasyScheduler-486]](https://github.com/analysys/EasyScheduler/issues/486)shell进程退出，yarn状态非终态等待判断 |
||||
|
||||
修复 |
||||
=== |
||||
- [[EasyScheduler-198]](https://github.com/analysys/EasyScheduler/issues/198) 流程定义列表根据定时状态和更新时间进行排序 |
||||
- [[EasyScheduler-419]](https://github.com/analysys/EasyScheduler/issues/419) 修复在线创建文件，hdfs文件未创建，却返回成功 |
||||
- [[EasyScheduler-481]](https://github.com/analysys/EasyScheduler/issues/481)修复job不存在定时无法下线的问题 |
||||
- [[EasyScheduler-425]](https://github.com/analysys/EasyScheduler/issues/425) kill任务时增加对其子进程的kill |
||||
- [[EasyScheduler-422]](https://github.com/analysys/EasyScheduler/issues/422) 修复更新资源文件时更新时间和大小未更新的问题 |
||||
- [[EasyScheduler-431]](https://github.com/analysys/EasyScheduler/issues/431) 修复删除租户时,如果未启动hdfs,则删除租户失败的问题 |
||||
|
||||
|
||||
感谢： |
||||
=== |
||||
最后但最重要的是，没有以下伙伴的贡献就没有新版本的诞生： |
||||
|
||||
Baoqi, jimmy201602, samz406, petersear, millionfor, hyperknob, fanguanqun, yangqinlong, qq389401879, feloxx, coding-now, hymzcn, nysyxxg, chgxtony, gj-zhang, xianhu, sunnyingit, |
||||
zhengqiangtan |
||||
|
||||
以及微信群里众多的热心伙伴！在此非常感谢！ |
||||
|
||||
@ -0,0 +1,23 @@
|
||||
Easy Scheduler Release 1.0.4 |
||||
=== |
||||
Easy Scheduler 1.0.4是1.x系列中的第个版本。 |
||||
|
||||
增强： |
||||
=== |
||||
- [[EasyScheduler-597]](https://github.com/analysys/EasyScheduler/issues/597)child process cannot extend father's receivers and cc |
||||
|
||||
修复 |
||||
=== |
||||
- [[EasyScheduler-516]](https://github.com/analysys/EasyScheduler/issues/516)The task instance of MR cannot stop in some cases |
||||
- [[EasyScheduler-594]](https://github.com/analysys/EasyScheduler/issues/594)soft kill task 后 进程依旧存在(父进程 子进程) |
||||
|
||||
|
||||
感谢： |
||||
=== |
||||
最后但最重要的是，没有以下伙伴的贡献就没有新版本的诞生： |
||||
|
||||
Baoqi, jimmy201602, samz406, petersear, millionfor, hyperknob, fanguanqun, yangqinlong, qq389401879, feloxx, coding-now, hymzcn, nysyxxg, chgxtony, gj-zhang, xianhu, sunnyingit, |
||||
zhengqiangtan, chinashenkai |
||||
|
||||
以及微信群里众多的热心伙伴！在此非常感谢！ |
||||
|
||||
@ -0,0 +1,55 @@
|
||||
Easy Scheduler Release 1.1.0 |
||||
=== |
||||
Easy Scheduler 1.1.0是1.x系列中的第六个版本。 |
||||
|
||||
新特性： |
||||
=== |
||||
- [[EasyScheduler-391](https://github.com/analysys/EasyScheduler/issues/391)] run a process under a specified tenement user |
||||
- [[EasyScheduler-288](https://github.com/analysys/EasyScheduler/issues/288)] Feature/qiye_weixin |
||||
- [[EasyScheduler-189](https://github.com/analysys/EasyScheduler/issues/189)] Kerberos等安全支持 |
||||
- [[EasyScheduler-398](https://github.com/analysys/EasyScheduler/issues/398)]管理员，有租户（install.sh设置默认租户），可以创建资源、项目和数据源（限制有一个管理员） |
||||
- [[EasyScheduler-293](https://github.com/analysys/EasyScheduler/issues/293)]点击运行流程时候选择的参数，没有地方可查看，也没有保存 |
||||
- [[EasyScheduler-401](https://github.com/analysys/EasyScheduler/issues/401)]定时很容易定时每秒一次，定时完成以后可以在页面显示一下下次触发时间 |
||||
- [[EasyScheduler-493](https://github.com/analysys/EasyScheduler/pull/493)]add datasource kerberos auth and FAQ modify and add resource upload s3 |
||||
|
||||
|
||||
增强： |
||||
=== |
||||
- [[EasyScheduler-227](https://github.com/analysys/EasyScheduler/issues/227)] upgrade spring-boot to 2.1.x and spring to 5.x |
||||
- [[EasyScheduler-434](https://github.com/analysys/EasyScheduler/issues/434)] worker节点数量 zk和mysql中不一致 |
||||
- [[EasyScheduler-435](https://github.com/analysys/EasyScheduler/issues/435)]邮箱格式的验证 |
||||
- [[EasyScheduler-441](https://github.com/analysys/EasyScheduler/issues/441)] 禁止运行节点加入已完成节点检测 |
||||
- [[EasyScheduler-400](https://github.com/analysys/EasyScheduler/issues/400)] 首页页面，队列统计不和谐，命令统计无数据 |
||||
- [[EasyScheduler-395](https://github.com/analysys/EasyScheduler/issues/395)] 对于容错恢复的流程，状态不能为 **正在运行 |
||||
- [[EasyScheduler-529](https://github.com/analysys/EasyScheduler/issues/529)] optimize poll task from zookeeper |
||||
- [[EasyScheduler-242](https://github.com/analysys/EasyScheduler/issues/242)]worker-server节点获取任务性能问题 |
||||
- [[EasyScheduler-352](https://github.com/analysys/EasyScheduler/issues/352)]worker 分组, 队列消费问题 |
||||
- [[EasyScheduler-461](https://github.com/analysys/EasyScheduler/issues/461)]查看数据源参数，需要加密账号密码信息 |
||||
- [[EasyScheduler-396](https://github.com/analysys/EasyScheduler/issues/396)]Dockerfile优化，并关联Dockerfile和github实现自动打镜像 |
||||
- [[EasyScheduler-389](https://github.com/analysys/EasyScheduler/issues/389)]service monitor cannot find the change of master/worker |
||||
- [[EasyScheduler-511](https://github.com/analysys/EasyScheduler/issues/511)]support recovery process from stop/kill nodes. |
||||
- [[EasyScheduler-399](https://github.com/analysys/EasyScheduler/issues/399)]HadoopUtils指定用户操作，而不是 **部署用户 |
||||
|
||||
修复： |
||||
=== |
||||
- [[EasyScheduler-394](https://github.com/analysys/EasyScheduler/issues/394)] master&worker部署在同一台机器上时，如果重启master&worker服务，会导致之前调度的任务无法继续调度 |
||||
- [[EasyScheduler-469](https://github.com/analysys/EasyScheduler/issues/469)]Fix naming errors,monitor page |
||||
- [[EasyScheduler-392](https://github.com/analysys/EasyScheduler/issues/392)]Feature request: fix email regex check |
||||
- [[EasyScheduler-405](https://github.com/analysys/EasyScheduler/issues/405)]定时修改/添加页面，开始时间和结束时间不能相同 |
||||
- [[EasyScheduler-517](https://github.com/analysys/EasyScheduler/issues/517)]补数 - 子工作流 - 时间参数 |
||||
- [[EasyScheduler-532](https://github.com/analysys/EasyScheduler/issues/532)]python节点不执行的问题 |
||||
- [[EasyScheduler-543](https://github.com/analysys/EasyScheduler/issues/543)]optimize datasource connection params safety |
||||
- [[EasyScheduler-569](https://github.com/analysys/EasyScheduler/issues/569)]定时任务无法真正停止 |
||||
- [[EasyScheduler-463](https://github.com/analysys/EasyScheduler/issues/463)]邮箱验证不支持非常见后缀邮箱 |
||||
|
||||
|
||||
|
||||
|
||||
感谢： |
||||
=== |
||||
最后但最重要的是，没有以下伙伴的贡献就没有新版本的诞生： |
||||
|
||||
Baoqi, jimmy201602, samz406, petersear, millionfor, hyperknob, fanguanqun, yangqinlong, qq389401879, chgxtony, Stanfan, lfyee, thisnew, hujiang75277381, sunnyingit, lgbo-ustc, ivivi, lzy305, JackIllkid, telltime, lipengbo2018, wuchunfu, telltime |
||||
|
||||
以及微信群里众多的热心伙伴！在此非常感谢！ |
||||
|
||||
|
After Width: | Height: | Size: 104 KiB |
|
After Width: | Height: | Size: 36 KiB |
|
After Width: | Height: | Size: 106 KiB |
|
After Width: | Height: | Size: 36 KiB |
@ -0,0 +1,57 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
package cn.escheduler.alert.manager; |
||||
|
||||
import cn.escheduler.alert.utils.Constants; |
||||
import cn.escheduler.alert.utils.EnterpriseWeChatUtils; |
||||
import cn.escheduler.dao.model.Alert; |
||||
import org.slf4j.Logger; |
||||
import org.slf4j.LoggerFactory; |
||||
|
||||
import java.io.IOException; |
||||
import java.util.Arrays; |
||||
import java.util.HashMap; |
||||
import java.util.List; |
||||
import java.util.Map; |
||||
|
||||
/** |
||||
* Enterprise WeChat Manager |
||||
*/ |
||||
public class EnterpriseWeChatManager { |
||||
private static final Logger logger = LoggerFactory.getLogger(MsgManager.class); |
||||
/** |
||||
* Enterprise We Chat send |
||||
* @param alert |
||||
*/ |
||||
public Map<String,Object> send(Alert alert, String token){ |
||||
Map<String,Object> retMap = new HashMap<>(); |
||||
retMap.put(Constants.STATUS, false); |
||||
String agentId = EnterpriseWeChatUtils.enterpriseWeChatAgentId; |
||||
String users = EnterpriseWeChatUtils.enterpriseWeChatUsers; |
||||
List<String> userList = Arrays.asList(users.split(",")); |
||||
logger.info("send message {}",alert); |
||||
String msg = EnterpriseWeChatUtils.makeUserSendMsg(userList, agentId,EnterpriseWeChatUtils.markdownByAlert(alert)); |
||||
try { |
||||
EnterpriseWeChatUtils.sendEnterpriseWeChat(Constants.UTF_8, msg, token); |
||||
} catch (IOException e) { |
||||
logger.error(e.getMessage(),e); |
||||
} |
||||
retMap.put(Constants.STATUS, true); |
||||
return retMap; |
||||
} |
||||
|
||||
} |
||||
@ -0,0 +1,248 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
package cn.escheduler.alert.utils; |
||||
|
||||
import cn.escheduler.common.enums.ShowType; |
||||
import cn.escheduler.dao.model.Alert; |
||||
import com.alibaba.fastjson.JSON; |
||||
|
||||
import com.google.common.reflect.TypeToken; |
||||
import org.apache.commons.lang3.StringUtils; |
||||
import org.apache.http.HttpEntity; |
||||
import org.apache.http.client.methods.CloseableHttpResponse; |
||||
import org.apache.http.client.methods.HttpGet; |
||||
import org.apache.http.client.methods.HttpPost; |
||||
import org.apache.http.entity.StringEntity; |
||||
import org.apache.http.impl.client.CloseableHttpClient; |
||||
import org.apache.http.impl.client.HttpClients; |
||||
import org.apache.http.util.EntityUtils; |
||||
import org.slf4j.Logger; |
||||
import org.slf4j.LoggerFactory; |
||||
|
||||
import java.io.IOException; |
||||
import java.util.*; |
||||
|
||||
import static cn.escheduler.alert.utils.PropertyUtils.getString; |
||||
|
||||
/** |
||||
* Enterprise WeChat utils |
||||
*/ |
||||
public class EnterpriseWeChatUtils { |
||||
|
||||
public static final Logger logger = LoggerFactory.getLogger(EnterpriseWeChatUtils.class); |
||||
|
||||
private static final String enterpriseWeChatCorpId = getString(Constants.ENTERPRISE_WECHAT_CORP_ID); |
||||
|
||||
private static final String enterpriseWeChatSecret = getString(Constants.ENTERPRISE_WECHAT_SECRET); |
||||
|
||||
private static final String enterpriseWeChatTokenUrl = getString(Constants.ENTERPRISE_WECHAT_TOKEN_URL); |
||||
private static String enterpriseWeChatTokenUrlReplace = enterpriseWeChatTokenUrl |
||||
.replaceAll("\\$corpId", enterpriseWeChatCorpId) |
||||
.replaceAll("\\$secret", enterpriseWeChatSecret); |
||||
|
||||
private static final String enterpriseWeChatPushUrl = getString(Constants.ENTERPRISE_WECHAT_PUSH_URL); |
||||
|
||||
private static final String enterpriseWeChatTeamSendMsg = getString(Constants.ENTERPRISE_WECHAT_TEAM_SEND_MSG); |
||||
|
||||
private static final String enterpriseWeChatUserSendMsg = getString(Constants.ENTERPRISE_WECHAT_USER_SEND_MSG); |
||||
|
||||
public static final String enterpriseWeChatAgentId = getString(Constants.ENTERPRISE_WECHAT_AGENT_ID); |
||||
|
||||
public static final String enterpriseWeChatUsers = getString(Constants.ENTERPRISE_WECHAT_USERS); |
||||
|
||||
/** |
||||
* get Enterprise WeChat token info |
||||
* @return token string info |
||||
* @throws IOException |
||||
*/ |
||||
public static String getToken() throws IOException { |
||||
String resp; |
||||
|
||||
CloseableHttpClient httpClient = HttpClients.createDefault(); |
||||
HttpGet httpGet = new HttpGet(enterpriseWeChatTokenUrlReplace); |
||||
CloseableHttpResponse response = httpClient.execute(httpGet); |
||||
try { |
||||
HttpEntity entity = response.getEntity(); |
||||
resp = EntityUtils.toString(entity, Constants.UTF_8); |
||||
EntityUtils.consume(entity); |
||||
} finally { |
||||
response.close(); |
||||
} |
||||
|
||||
Map<String, Object> map = JSON.parseObject(resp, |
||||
new TypeToken<Map<String, Object>>() { |
||||
}.getType()); |
||||
return map.get("access_token").toString(); |
||||
} |
||||
|
||||
/** |
||||
* make team single Enterprise WeChat message |
||||
* @param toParty |
||||
* @param agentId |
||||
* @param msg |
||||
* @return Enterprise WeChat send message |
||||
*/ |
||||
public static String makeTeamSendMsg(String toParty, String agentId, String msg) { |
||||
return enterpriseWeChatTeamSendMsg.replaceAll("\\$toParty", toParty) |
||||
.replaceAll("\\$agentId", agentId) |
||||
.replaceAll("\\$msg", msg); |
||||
} |
||||
|
||||
/** |
||||
* make team multi Enterprise WeChat message |
||||
* @param toParty |
||||
* @param agentId |
||||
* @param msg |
||||
* @return Enterprise WeChat send message |
||||
*/ |
||||
public static String makeTeamSendMsg(Collection<String> toParty, String agentId, String msg) { |
||||
String listParty = FuncUtils.mkString(toParty, "|"); |
||||
return enterpriseWeChatTeamSendMsg.replaceAll("\\$toParty", listParty) |
||||
.replaceAll("\\$agentId", agentId) |
||||
.replaceAll("\\$msg", msg); |
||||
} |
||||
|
||||
/** |
||||
* make team single user message |
||||
* @param toUser |
||||
* @param agentId |
||||
* @param msg |
||||
* @return Enterprise WeChat send message |
||||
*/ |
||||
public static String makeUserSendMsg(String toUser, String agentId, String msg) { |
||||
return enterpriseWeChatUserSendMsg.replaceAll("\\$toUser", toUser) |
||||
.replaceAll("\\$agentId", agentId) |
||||
.replaceAll("\\$msg", msg); |
||||
} |
||||
|
||||
/** |
||||
* make team multi user message |
||||
* @param toUser |
||||
* @param agentId |
||||
* @param msg |
||||
* @return Enterprise WeChat send message |
||||
*/ |
||||
public static String makeUserSendMsg(Collection<String> toUser, String agentId, String msg) { |
||||
String listUser = FuncUtils.mkString(toUser, "|"); |
||||
return enterpriseWeChatUserSendMsg.replaceAll("\\$toUser", listUser) |
||||
.replaceAll("\\$agentId", agentId) |
||||
.replaceAll("\\$msg", msg); |
||||
} |
||||
|
||||
/** |
||||
* send Enterprise WeChat |
||||
* @param charset |
||||
* @param data |
||||
* @param token |
||||
* @return Enterprise WeChat resp, demo: {"errcode":0,"errmsg":"ok","invaliduser":""} |
||||
* @throws IOException |
||||
*/ |
||||
public static String sendEnterpriseWeChat(String charset, String data, String token) throws IOException { |
||||
String enterpriseWeChatPushUrlReplace = enterpriseWeChatPushUrl.replaceAll("\\$token", token); |
||||
|
||||
CloseableHttpClient httpclient = HttpClients.createDefault(); |
||||
HttpPost httpPost = new HttpPost(enterpriseWeChatPushUrlReplace); |
||||
httpPost.setEntity(new StringEntity(data, charset)); |
||||
CloseableHttpResponse response = httpclient.execute(httpPost); |
||||
String resp; |
||||
try { |
||||
HttpEntity entity = response.getEntity(); |
||||
resp = EntityUtils.toString(entity, charset); |
||||
EntityUtils.consume(entity); |
||||
} finally { |
||||
response.close(); |
||||
} |
||||
logger.info("Enterprise WeChat send [{}], param:{}, resp:{}", enterpriseWeChatPushUrl, data, resp); |
||||
return resp; |
||||
} |
||||
|
||||
/** |
||||
* convert table to markdown style |
||||
* @param title |
||||
* @param content |
||||
* @return |
||||
*/ |
||||
public static String markdownTable(String title,String content){ |
||||
List<LinkedHashMap> mapItemsList = JSONUtils.toList(content, LinkedHashMap.class); |
||||
StringBuilder contents = new StringBuilder(200); |
||||
for (LinkedHashMap mapItems : mapItemsList){ |
||||
|
||||
Set<Map.Entry<String, String>> entries = mapItems.entrySet(); |
||||
|
||||
Iterator<Map.Entry<String, String>> iterator = entries.iterator(); |
||||
|
||||
StringBuilder t = new StringBuilder(String.format("`%s`%s",title,Constants.MARKDOWN_ENTER)); |
||||
while (iterator.hasNext()){ |
||||
|
||||
Map.Entry<String, String> entry = iterator.next(); |
||||
t.append(Constants.MARKDOWN_QUOTE); |
||||
t.append(entry.getKey()).append(":").append(entry.getValue()); |
||||
t.append(Constants.MARKDOWN_ENTER); |
||||
} |
||||
|
||||
contents.append(t); |
||||
} |
||||
return contents.toString(); |
||||
} |
||||
|
||||
/** |
||||
* convert text to markdown style |
||||
* @param title |
||||
* @param content |
||||
* @return |
||||
*/ |
||||
public static String markdownText(String title,String content){ |
||||
if (StringUtils.isNotEmpty(content)){ |
||||
List<String> list; |
||||
try { |
||||
list = JSONUtils.toList(content,String.class); |
||||
}catch (Exception e){ |
||||
logger.error("json format exception",e); |
||||
return null; |
||||
} |
||||
|
||||
StringBuilder contents = new StringBuilder(100); |
||||
contents.append(String.format("`%s`\n",title)); |
||||
for (String str : list){ |
||||
contents.append(Constants.MARKDOWN_QUOTE); |
||||
contents.append(str); |
||||
contents.append(Constants.MARKDOWN_ENTER); |
||||
} |
||||
|
||||
return contents.toString(); |
||||
|
||||
} |
||||
return null; |
||||
} |
||||
|
||||
/** |
||||
* Determine the mardown style based on the show type of the alert |
||||
* @param alert |
||||
* @return |
||||
*/ |
||||
public static String markdownByAlert(Alert alert){ |
||||
String result = ""; |
||||
if (alert.getShowType() == ShowType.TABLE) { |
||||
result = markdownTable(alert.getTitle(),alert.getContent()); |
||||
}else if(alert.getShowType() == ShowType.TEXT){ |
||||
result = markdownText(alert.getTitle(),alert.getContent()); |
||||
} |
||||
return result; |
||||
|
||||
} |
||||
|
||||
} |
||||
@ -0,0 +1,35 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
package cn.escheduler.alert.utils; |
||||
|
||||
public class FuncUtils { |
||||
|
||||
static public String mkString(Iterable<String> list, String split) { |
||||
StringBuilder sb = new StringBuilder(); |
||||
boolean first = true; |
||||
for (String item : list) { |
||||
if (first) { |
||||
first = false; |
||||
} else { |
||||
sb.append(split); |
||||
} |
||||
sb.append(item); |
||||
} |
||||
return sb.toString(); |
||||
} |
||||
|
||||
} |
||||
@ -0,0 +1,54 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
package cn.escheduler.api; |
||||
|
||||
import cn.escheduler.alert.AlertServer; |
||||
import cn.escheduler.dao.ProcessDao; |
||||
import cn.escheduler.server.master.MasterServer; |
||||
import cn.escheduler.server.rpc.LoggerServer; |
||||
import cn.escheduler.server.worker.WorkerServer; |
||||
import org.springframework.boot.SpringApplication; |
||||
import org.springframework.boot.autoconfigure.SpringBootApplication; |
||||
import org.springframework.boot.web.servlet.ServletComponentScan; |
||||
import org.springframework.boot.web.servlet.support.SpringBootServletInitializer; |
||||
import org.springframework.context.ConfigurableApplicationContext; |
||||
import org.springframework.context.annotation.ComponentScan; |
||||
import springfox.documentation.swagger2.annotations.EnableSwagger2; |
||||
|
||||
@SpringBootApplication |
||||
@ServletComponentScan |
||||
@ComponentScan("cn.escheduler") |
||||
@EnableSwagger2 |
||||
public class CombinedApplicationServer extends SpringBootServletInitializer { |
||||
|
||||
public static void main(String[] args) throws Exception { |
||||
|
||||
ConfigurableApplicationContext context = SpringApplication.run(ApiApplicationServer.class, args); |
||||
ProcessDao processDao = context.getBean(ProcessDao.class); |
||||
MasterServer master = new MasterServer(processDao); |
||||
master.run(processDao); |
||||
|
||||
WorkerServer workerServer = new WorkerServer(); |
||||
workerServer.run(); |
||||
|
||||
LoggerServer server = new LoggerServer(); |
||||
server.start(); |
||||
|
||||
AlertServer alertServer = AlertServer.getInstance(); |
||||
alertServer.start(); |
||||
} |
||||
} |
||||
@ -1,42 +0,0 @@
|
||||
<!-- Logback configuration. See http://logback.qos.ch/manual/index.html --> |
||||
<configuration scan="true" scanPeriod="120 seconds"> |
||||
<logger name="org.apache.zookeeper" level="WARN"/> |
||||
<logger name="org.apache.hbase" level="WARN"/> |
||||
<logger name="org.apache.hadoop" level="WARN"/> |
||||
|
||||
<property name="log.base" value="logs" /> |
||||
|
||||
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender"> |
||||
<encoder> |
||||
<pattern> |
||||
[%level] %date{yyyy-MM-dd HH:mm:ss.SSS} %logger{96}:[%line] - %msg%n |
||||
</pattern> |
||||
<charset>UTF-8</charset> |
||||
</encoder> |
||||
</appender> |
||||
|
||||
<appender name="APISERVERLOGFILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> |
||||
<!-- Log level filter --> |
||||
<filter class="ch.qos.logback.classic.filter.ThresholdFilter"> |
||||
<level>INFO</level> |
||||
</filter> |
||||
<file>${log.base}/escheduler-api-server.log</file> |
||||
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy"> |
||||
<fileNamePattern>${log.base}/escheduler-api-server.%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern> |
||||
<maxHistory>168</maxHistory> |
||||
<maxFileSize>64MB</maxFileSize> |
||||
</rollingPolicy> |
||||
|
||||
<encoder> |
||||
<pattern> |
||||
[%level] %date{yyyy-MM-dd HH:mm:ss.SSS} %logger{96}:[%line] - %msg%n |
||||
</pattern> |
||||
<charset>UTF-8</charset> |
||||
</encoder> |
||||
|
||||
</appender> |
||||
|
||||
<root level="INFO"> |
||||
<appender-ref ref="STDOUT" /> |
||||
</root> |
||||
</configuration> |
||||
@ -0,0 +1,29 @@
|
||||
package cn.escheduler.api.utils; |
||||
|
||||
import cn.escheduler.dao.model.MasterServer; |
||||
import org.junit.Assert; |
||||
import org.junit.Test; |
||||
|
||||
|
||||
import java.util.List; |
||||
|
||||
public class ZookeeperMonitorUtilsTest { |
||||
|
||||
|
||||
@Test |
||||
public void testGetMasterLsit(){ |
||||
|
||||
ZookeeperMonitor zookeeperMonitor = new ZookeeperMonitor(); |
||||
|
||||
|
||||
List<MasterServer> masterServerList = zookeeperMonitor.getMasterServers(); |
||||
|
||||
List<MasterServer> workerServerList = zookeeperMonitor.getWorkerServers(); |
||||
|
||||
Assert.assertEquals(masterServerList.size(), 1); |
||||
Assert.assertEquals(workerServerList.size(), 1); |
||||
|
||||
|
||||
} |
||||
|
||||
} |
||||
@ -0,0 +1,29 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
package cn.escheduler.common.enums; |
||||
|
||||
/** |
||||
* data base types |
||||
*/ |
||||
public enum ResUploadType { |
||||
/** |
||||
* 0 hdfs |
||||
* 1 s3 |
||||
* 2 none |
||||
*/ |
||||
HDFS,S3,NONE |
||||
} |
||||
@ -0,0 +1,35 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
package cn.escheduler.common.enums; |
||||
|
||||
|
||||
/** |
||||
* task record status |
||||
* |
||||
*/ |
||||
public enum TaskRecordStatus { |
||||
|
||||
/** |
||||
* status： |
||||
* 0 sucess |
||||
* 1 failure |
||||
* 2 exception |
||||
*/ |
||||
SUCCESS,FAILURE,EXCEPTION |
||||
|
||||
|
||||
} |
||||
@ -0,0 +1,15 @@
|
||||
package cn.escheduler.common.enums; |
||||
|
||||
/** |
||||
* zk node type |
||||
*/ |
||||
public enum ZKNodeType { |
||||
|
||||
/** |
||||
* 0 do not send warning; |
||||
* 1 send if process success; |
||||
* 2 send if process failed; |
||||
* 3 send if process ending; |
||||
*/ |
||||
MASTER, WORKER, DEAD_SERVER, TASK_QUEUE; |
||||
} |
||||
@ -0,0 +1,64 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
package cn.escheduler.common.utils; |
||||
|
||||
|
||||
import org.slf4j.Logger; |
||||
import org.slf4j.LoggerFactory; |
||||
|
||||
|
||||
/** |
||||
* http utils |
||||
*/ |
||||
public class IpUtils { |
||||
|
||||
private static final Logger logger = LoggerFactory.getLogger(IpUtils.class); |
||||
public static final String DOT = "."; |
||||
|
||||
/** |
||||
* ip str to long <p> |
||||
* |
||||
* @param ipStr ip string |
||||
*/ |
||||
public static Long ipToLong(String ipStr) { |
||||
String[] ipSet = ipStr.split("\\" + DOT); |
||||
|
||||
return Long.parseLong(ipSet[0]) << 24 | Long.parseLong(ipSet[1]) << 16 | Long.parseLong(ipSet[2]) << 8 | Long.parseLong(ipSet[3]); |
||||
} |
||||
|
||||
/** |
||||
* long to ip |
||||
* @param ipLong the long number converted from IP |
||||
* @return String |
||||
*/ |
||||
public static String longToIp(long ipLong) { |
||||
long[] ipNumbers = new long[4]; |
||||
long tmp = 0xFF; |
||||
ipNumbers[0] = ipLong >> 24 & tmp; |
||||
ipNumbers[1] = ipLong >> 16 & tmp; |
||||
ipNumbers[2] = ipLong >> 8 & tmp; |
||||
ipNumbers[3] = ipLong & tmp; |
||||
|
||||
StringBuilder sb = new StringBuilder(16); |
||||
sb.append(ipNumbers[0]).append(DOT) |
||||
.append(ipNumbers[1]).append(DOT) |
||||
.append(ipNumbers[2]).append(DOT) |
||||
.append(ipNumbers[3]); |
||||
return sb.toString(); |
||||
} |
||||
|
||||
} |
||||
@ -0,0 +1,41 @@
|
||||
package cn.escheduler.common.utils; |
||||
|
||||
import org.junit.Assert; |
||||
import org.junit.Test; |
||||
|
||||
import static org.junit.Assert.*; |
||||
|
||||
public class IpUtilsTest { |
||||
|
||||
@Test |
||||
public void ipToLong() { |
||||
|
||||
String ip = "192.168.110.1"; |
||||
String ip2 = "0.0.0.0"; |
||||
long longNumber = IpUtils.ipToLong(ip); |
||||
long longNumber2 = IpUtils.ipToLong(ip2); |
||||
System.out.println(longNumber); |
||||
Assert.assertEquals(longNumber, 3232263681L); |
||||
Assert.assertEquals(longNumber2, 0L); |
||||
|
||||
String ip3 = "255.255.255.255"; |
||||
long longNumber3 = IpUtils.ipToLong(ip3); |
||||
System.out.println(longNumber3); |
||||
Assert.assertEquals(longNumber3, 4294967295L); |
||||
|
||||
} |
||||
|
||||
@Test |
||||
public void longToIp() { |
||||
|
||||
String ip = "192.168.110.1"; |
||||
String ip2 = "0.0.0.0"; |
||||
long longNum = 3232263681L; |
||||
String i1 = IpUtils.longToIp(longNum); |
||||
|
||||
String i2 = IpUtils.longToIp(0); |
||||
|
||||
Assert.assertEquals(ip, i1); |
||||
Assert.assertEquals(ip2, i2); |
||||
} |
||||
} |
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in new issue