HuangWei
3 years ago
committed by
GitHub
28 changed files with 883 additions and 15 deletions
@ -0,0 +1,76 @@ |
|||||||
|
# OpenMLDB Node |
||||||
|
|
||||||
|
## Overview |
||||||
|
|
||||||
|
[OpenMLDB](https://openmldb.ai/) is an excellent open source machine learning database, providing a full-stack |
||||||
|
FeatureOps solution for production. |
||||||
|
|
||||||
|
OpenMLDB task plugin used to execute tasks on OpenMLDB cluster. |
||||||
|
|
||||||
|
## Create Task |
||||||
|
|
||||||
|
- Click `Project -> Management-Project -> Name-Workflow Definition`, and click the "Create Workflow" button to enter the |
||||||
|
DAG editing page. |
||||||
|
- Drag from the toolbar <img src="/img/tasks/icons/openmldb.png" width="15"/> task node to canvas. |
||||||
|
|
||||||
|
## Task Example |
||||||
|
|
||||||
|
First, introduce some general parameters of DolphinScheduler |
||||||
|
|
||||||
|
- **Node name**: The node name in a workflow definition is unique. |
||||||
|
- **Run flag**: Identifies whether this node schedules normally, if it does not need to execute, select |
||||||
|
the `prohibition execution`. |
||||||
|
- **Descriptive information**: Describe the function of the node. |
||||||
|
- **Task priority**: When the number of worker threads is insufficient, execute in the order of priority from high |
||||||
|
to low, and tasks with the same priority will execute in a first-in first-out order. |
||||||
|
- **Worker grouping**: Assign tasks to the machines of the worker group to execute. If `Default` is selected, |
||||||
|
randomly select a worker machine for execution. |
||||||
|
- **Environment Name**: Configure the environment name in which run the script. |
||||||
|
- **Times of failed retry attempts**: The number of times the task failed to resubmit. |
||||||
|
- **Failed retry interval**: The time interval (unit minute) for resubmitting the task after a failed task. |
||||||
|
- **Delayed execution time**: The time (unit minute) that a task delays in execution. |
||||||
|
- **Timeout alarm**: Check the timeout alarm and timeout failure. When the task runs exceed the "timeout", an alarm |
||||||
|
email will send and the task execution will fail. |
||||||
|
- **Predecessor task**: Selecting a predecessor task for the current task, will set the selected predecessor task as |
||||||
|
upstream of the current task. |
||||||
|
|
||||||
|
### OpenMLDB Parameters |
||||||
|
|
||||||
|
**Task Parameter** |
||||||
|
|
||||||
|
- **zookeeper** ：OpenMLDB cluster zookeeper address, e.g. 127.0.0.1:2181. |
||||||
|
- **zookeeper path** : OpenMLDB cluster zookeeper path, e.g. /openmldb. |
||||||
|
- **Execute Mode** ：determine the init mode, offline or online. You can switch it in sql statement. |
||||||
|
- **SQL statement** ：SQL statement. |
||||||
|
- Custom parameters: It is the user-defined parameters of Python, which will replace the content with \${variable} in the script. |
||||||
|
|
||||||
|
Here are some examples: |
||||||
|
|
||||||
|
#### Load data |
||||||
|
|
||||||
|
 |
||||||
|
|
||||||
|
We use `LOAD DATA` to load data into OpenMLDB cluster. We select `offline` here, so it will load to offline storage. |
||||||
|
|
||||||
|
#### Feature extraction |
||||||
|
|
||||||
|
 |
||||||
|
|
||||||
|
We use `SELECT INTO` to do feature extraction. We select `offline` here, so it will run sql on offline engine. |
||||||
|
|
||||||
|
## Environment to prepare |
||||||
|
|
||||||
|
### Start the OpenMLDB cluster |
||||||
|
|
||||||
|
You should create an OpenMLDB cluster first. If in production env, please check [deploy OpenMLDB](https://openmldb.ai/docs/en/v0.5/deploy/install_deploy.html). |
||||||
|
|
||||||
|
You can follow [run OpenMLDB in docker](https://openmldb.ai/docs/zh/v0.5/quickstart/openmldb_quickstart.html#id11) |
||||||
|
to a quick start. |
||||||
|
|
||||||
|
### Python env |
||||||
|
|
||||||
|
The OpenMLDB task will use OpenMLDB Python SDK to connect OpenMLDB cluster. So you should have the Python env. |
||||||
|
|
||||||
|
We will use `python3` by default. You can set `PYTHON_HOME` to use your custom python env. |
||||||
|
|
||||||
|
Make sure you have installed OpenMLDB Python SDK in the host where the worker server running, using `pip install openmldb`. |
||||||
@ -0,0 +1,68 @@ |
|||||||
|
# OpenMLDB 节点 |
||||||
|
|
||||||
|
## 综述 |
||||||
|
|
||||||
|
[OpenMLDB](https://openmldb.ai/) 是一个优秀的开源机器学习数据库，提供生产级数据及特征开发全栈解决方案。 |
||||||
|
|
||||||
|
OpenMLDB任务组件可以连接OpenMLDB集群执行任务。 |
||||||
|
|
||||||
|
## 创建任务 |
||||||
|
|
||||||
|
- 点击项目管理-项目名称-工作流定义，点击“创建工作流”按钮，进入 DAG 编辑页面； |
||||||
|
- 拖动工具栏的 <img src="/img/tasks/icons/openmldb.png" width="15"/> 任务节点到画板中。 |
||||||
|
|
||||||
|
## 任务样例 |
||||||
|
|
||||||
|
首先介绍一些DS通用参数： |
||||||
|
|
||||||
|
- **节点名称** ：设置任务的名称。一个工作流定义中的节点名称是唯一的。 |
||||||
|
- **运行标志** ：标识这个节点是否能正常调度,如果不需要执行，可以打开禁止执行开关。 |
||||||
|
- **描述** ：描述该节点的功能。 |
||||||
|
- **任务优先级** ：worker 线程数不足时，根据优先级从高到低依次执行，优先级一样时根据先进先出原则执行。 |
||||||
|
- **Worker 分组** ：任务分配给 worker 组的机器执行，选择 Default，会随机选择一台 worker 机执行。 |
||||||
|
- **环境名称** ：配置运行脚本的环境。 |
||||||
|
- **失败重试次数** ：任务失败重新提交的次数。 |
||||||
|
- **失败重试间隔** ：任务失败重新提交任务的时间间隔，以分钟为单位。 |
||||||
|
- **延迟执行时间** ：任务延迟执行的时间，以分钟为单位。 |
||||||
|
- **超时告警** ：勾选超时告警、超时失败，当任务超过"超时时长"后，会发送告警邮件并且任务执行失败。 |
||||||
|
- **前置任务** ：选择当前任务的前置任务，会将被选择的前置任务设置为当前任务的上游。 |
||||||
|
|
||||||
|
### OpenMLDB 参数 |
||||||
|
|
||||||
|
**任务参数** |
||||||
|
|
||||||
|
- **zookeeper地址** ：OpenMLDB集群连接地址中的zookeeper地址, e.g. 127.0.0.1:2181。 |
||||||
|
- **zookeeper路径** : OpenMLDB集群连接地址中的zookeeper路径, e.g. /openmldb。 |
||||||
|
- **执行模式** ：初始执行模式（离线/在线），你可以在sql语句中随时切换。 |
||||||
|
- **SQL语句** ：SQL语句。 |
||||||
|
- 自定义参数：是PYTHON局部的用户自定义参数，会替换脚本中以${变量}的内容。 |
||||||
|
|
||||||
|
下面有几个例子： |
||||||
|
|
||||||
|
#### 导入数据 |
||||||
|
|
||||||
|
 |
||||||
|
|
||||||
|
我们使用`LOAD DATA`语句导入数据到OpenMLDB集群。因为选择的是离线执行模式，所以将会导入数据到离线存储中。 |
||||||
|
|
||||||
|
#### 特征抽取 |
||||||
|
|
||||||
|
 |
||||||
|
|
||||||
|
我们使用`SELECT INTO`进行特征抽取。因为选择的是离线执行模式，所以会使用离线引擎做特征计算。 |
||||||
|
|
||||||
|
## 环境准备 |
||||||
|
|
||||||
|
### OpenMLDB 启动 |
||||||
|
|
||||||
|
执行任务之前，你需要启动OpenMLDB集群。如果是在生产环境，请参考[deploy OpenMLDB](https://openmldb.ai/docs/zh/v0.5/deploy/install_deploy.html). |
||||||
|
|
||||||
|
你可以参考[在docker中运行OpenMLDB集群](https://openmldb.ai/docs/zh/v0.5/quickstart/openmldb_quickstart.html#id11) 快速启动。 |
||||||
|
|
||||||
|
### Python 环境 |
||||||
|
|
||||||
|
OpenMLDB任务组件将使用OpenMLDB Python SDK来连接OpenMLDB。所以你需要Python环境。 |
||||||
|
|
||||||
|
我们默认使用`python3`，你可以通过配置`PYTHON_HOME`来设置自己的Python环境。 |
||||||
|
|
||||||
|
请确保已通过`pip install openmldb`，在worker server的主机中安装了OpenMLDB Python SDK。 |
||||||
|
After Width: | Height: | Size: 150 KiB |
|
After Width: | Height: | Size: 89 KiB |
|
After Width: | Height: | Size: 88 KiB |
@ -0,0 +1,45 @@ |
|||||||
|
<?xml version="1.0" encoding="UTF-8"?> |
||||||
|
<!-- |
||||||
|
~ Licensed to the Apache Software Foundation (ASF) under one or more |
||||||
|
~ contributor license agreements. See the NOTICE file distributed with |
||||||
|
~ this work for additional information regarding copyright ownership. |
||||||
|
~ The ASF licenses this file to You under the Apache License, Version 2.0 |
||||||
|
~ (the "License"); you may not use this file except in compliance with |
||||||
|
~ the License. You may obtain a copy of the License at |
||||||
|
~ |
||||||
|
~ http://www.apache.org/licenses/LICENSE-2.0 |
||||||
|
~ |
||||||
|
~ Unless required by applicable law or agreed to in writing, software |
||||||
|
~ distributed under the License is distributed on an "AS IS" BASIS, |
||||||
|
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||||
|
~ See the License for the specific language governing permissions and |
||||||
|
~ limitations under the License. |
||||||
|
--> |
||||||
|
<project xmlns="http://maven.apache.org/POM/4.0.0" |
||||||
|
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" |
||||||
|
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> |
||||||
|
<parent> |
||||||
|
<artifactId>dolphinscheduler-task-plugin</artifactId> |
||||||
|
<groupId>org.apache.dolphinscheduler</groupId> |
||||||
|
<version>dev-SNAPSHOT</version> |
||||||
|
</parent> |
||||||
|
<modelVersion>4.0.0</modelVersion> |
||||||
|
|
||||||
|
<artifactId>dolphinscheduler-task-openmldb</artifactId> |
||||||
|
<packaging>jar</packaging> |
||||||
|
|
||||||
|
<dependencies> |
||||||
|
<dependency> |
||||||
|
<groupId>org.apache.dolphinscheduler</groupId> |
||||||
|
<artifactId>dolphinscheduler-spi</artifactId> |
||||||
|
</dependency> |
||||||
|
<dependency> |
||||||
|
<groupId>org.apache.dolphinscheduler</groupId> |
||||||
|
<artifactId>dolphinscheduler-task-api</artifactId> |
||||||
|
</dependency> |
||||||
|
<dependency> |
||||||
|
<groupId>org.apache.dolphinscheduler</groupId> |
||||||
|
<artifactId>dolphinscheduler-task-python</artifactId> |
||||||
|
</dependency> |
||||||
|
</dependencies> |
||||||
|
</project> |
||||||
@ -0,0 +1,90 @@ |
|||||||
|
/* |
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||||
|
* contributor license agreements. See the NOTICE file distributed with |
||||||
|
* this work for additional information regarding copyright ownership. |
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||||
|
* (the "License"); you may not use this file except in compliance with |
||||||
|
* the License. You may obtain a copy of the License at |
||||||
|
* |
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
* |
||||||
|
* Unless required by applicable law or agreed to in writing, software |
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS, |
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||||
|
* See the License for the specific language governing permissions and |
||||||
|
* limitations under the License. |
||||||
|
*/ |
||||||
|
|
||||||
|
package org.apache.dolphinscheduler.plugin.task.openmldb; |
||||||
|
|
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.model.ResourceInfo; |
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters; |
||||||
|
import org.apache.dolphinscheduler.spi.utils.StringUtils; |
||||||
|
|
||||||
|
import java.util.List; |
||||||
|
|
||||||
|
public class OpenmldbParameters extends AbstractParameters { |
||||||
|
|
||||||
|
private String zk; |
||||||
|
private String zkPath; |
||||||
|
private String executeMode; |
||||||
|
/** |
||||||
|
* origin sql script |
||||||
|
*/ |
||||||
|
private String sql; |
||||||
|
|
||||||
|
/** |
||||||
|
* resource list |
||||||
|
*/ |
||||||
|
private List<ResourceInfo> resourceList; |
||||||
|
|
||||||
|
public String getZk() { |
||||||
|
return zk; |
||||||
|
} |
||||||
|
|
||||||
|
public void setZk(String zk) { |
||||||
|
this.zk = zk; |
||||||
|
} |
||||||
|
|
||||||
|
public String getZkPath() { |
||||||
|
return zkPath; |
||||||
|
} |
||||||
|
|
||||||
|
public void setZkPath(String zkPath) { |
||||||
|
this.zkPath = zkPath; |
||||||
|
} |
||||||
|
|
||||||
|
public String getExecuteMode() { |
||||||
|
return executeMode; |
||||||
|
} |
||||||
|
|
||||||
|
public void setExecuteMode(String executeMode) { |
||||||
|
this.executeMode = executeMode; |
||||||
|
} |
||||||
|
|
||||||
|
public String getSql() { |
||||||
|
return sql; |
||||||
|
} |
||||||
|
|
||||||
|
public void setSql(String sql) { |
||||||
|
this.sql = sql; |
||||||
|
} |
||||||
|
|
||||||
|
public List<ResourceInfo> getResourceList() { |
||||||
|
return resourceList; |
||||||
|
} |
||||||
|
|
||||||
|
public void setResourceList(List<ResourceInfo> resourceList) { |
||||||
|
this.resourceList = resourceList; |
||||||
|
} |
||||||
|
|
||||||
|
@Override |
||||||
|
public boolean checkParameters() { |
||||||
|
return StringUtils.isNotEmpty(zk) && StringUtils.isNotEmpty(zkPath) && StringUtils.isNotEmpty(sql); |
||||||
|
} |
||||||
|
|
||||||
|
@Override |
||||||
|
public List<ResourceInfo> getResourceFilesList() { |
||||||
|
return this.resourceList; |
||||||
|
} |
||||||
|
} |
||||||
@ -0,0 +1,176 @@ |
|||||||
|
/* |
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||||
|
* contributor license agreements. See the NOTICE file distributed with |
||||||
|
* this work for additional information regarding copyright ownership. |
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||||
|
* (the "License"); you may not use this file except in compliance with |
||||||
|
* the License. You may obtain a copy of the License at |
||||||
|
* |
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
* |
||||||
|
* Unless required by applicable law or agreed to in writing, software |
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS, |
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||||
|
* See the License for the specific language governing permissions and |
||||||
|
* limitations under the License. |
||||||
|
*/ |
||||||
|
|
||||||
|
package org.apache.dolphinscheduler.plugin.task.openmldb; |
||||||
|
|
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.TaskException; |
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext; |
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.model.Property; |
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters; |
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.parser.ParamUtils; |
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.parser.ParameterUtils; |
||||||
|
import org.apache.dolphinscheduler.plugin.task.python.PythonTask; |
||||||
|
import org.apache.dolphinscheduler.spi.utils.JSONUtils; |
||||||
|
import org.apache.dolphinscheduler.spi.utils.StringUtils; |
||||||
|
|
||||||
|
import java.nio.file.Paths; |
||||||
|
import java.util.Locale; |
||||||
|
import java.util.Map; |
||||||
|
import java.util.regex.Matcher; |
||||||
|
import java.util.regex.Pattern; |
||||||
|
|
||||||
|
import com.google.common.base.Preconditions; |
||||||
|
|
||||||
|
/** |
||||||
|
* openmldb task |
||||||
|
*/ |
||||||
|
public class OpenmldbTask extends PythonTask { |
||||||
|
|
||||||
|
/** |
||||||
|
* openmldb parameters |
||||||
|
*/ |
||||||
|
private OpenmldbParameters openmldbParameters; |
||||||
|
|
||||||
|
/** |
||||||
|
* python process(openmldb only supports version 3 by default) |
||||||
|
*/ |
||||||
|
private static final String OPENMLDB_PYTHON = "python3"; |
||||||
|
private static final Pattern PYTHON_PATH_PATTERN = Pattern.compile("/bin/python[\\d.]*$"); |
||||||
|
|
||||||
|
/** |
||||||
|
* constructor |
||||||
|
* |
||||||
|
* @param taskRequest taskRequest |
||||||
|
*/ |
||||||
|
public OpenmldbTask(TaskExecutionContext taskRequest) { |
||||||
|
super(taskRequest); |
||||||
|
} |
||||||
|
|
||||||
|
@Override |
||||||
|
public void init() { |
||||||
|
logger.info("openmldb task params {}", taskRequest.getTaskParams()); |
||||||
|
|
||||||
|
openmldbParameters = JSONUtils.parseObject(taskRequest.getTaskParams(), OpenmldbParameters.class); |
||||||
|
|
||||||
|
if (openmldbParameters == null || !openmldbParameters.checkParameters()) { |
||||||
|
throw new TaskException("openmldb task params is not valid"); |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
@Override |
||||||
|
@Deprecated |
||||||
|
public String getPreScript() { |
||||||
|
return ""; |
||||||
|
} |
||||||
|
|
||||||
|
@Override |
||||||
|
public AbstractParameters getParameters() { |
||||||
|
return openmldbParameters; |
||||||
|
} |
||||||
|
|
||||||
|
/** |
||||||
|
* build python command file path |
||||||
|
* |
||||||
|
* @return python command file path |
||||||
|
*/ |
||||||
|
@Override |
||||||
|
protected String buildPythonCommandFilePath() { |
||||||
|
return String.format("%s/openmldb_%s.py", taskRequest.getExecutePath(), taskRequest.getTaskAppId()); |
||||||
|
} |
||||||
|
|
||||||
|
/** |
||||||
|
* build python script content from sql |
||||||
|
* |
||||||
|
* @return raw python script |
||||||
|
*/ |
||||||
|
@Override |
||||||
|
protected String buildPythonScriptContent() { |
||||||
|
logger.info("raw sql script : {}", openmldbParameters.getSql()); |
||||||
|
|
||||||
|
String rawSQLScript = openmldbParameters.getSql().replaceAll("[\\r]?\\n", "\n"); |
||||||
|
Map<String, Property> paramsMap = mergeParamsWithContext(openmldbParameters); |
||||||
|
rawSQLScript = ParameterUtils.convertParameterPlaceholders(rawSQLScript, ParamUtils.convert(paramsMap)); |
||||||
|
|
||||||
|
// convert sql to python script
|
||||||
|
String pythonScript = buildPythonScriptsFromSql(rawSQLScript); |
||||||
|
logger.info("rendered python script : {}", pythonScript); |
||||||
|
return pythonScript; |
||||||
|
} |
||||||
|
|
||||||
|
private String buildPythonScriptsFromSql(String rawSqlScript) { |
||||||
|
// imports
|
||||||
|
StringBuilder builder = new StringBuilder("import openmldb\nimport sqlalchemy as db\n"); |
||||||
|
|
||||||
|
// connect to openmldb
|
||||||
|
builder.append(String.format("engine = db.create_engine('openmldb:///?zk=%s&zkPath=%s')\n", |
||||||
|
openmldbParameters.getZk(), openmldbParameters.getZkPath())); |
||||||
|
builder.append("con = engine.connect()\n"); |
||||||
|
|
||||||
|
// execute mode
|
||||||
|
String executeMode = openmldbParameters.getExecuteMode().toLowerCase(Locale.ROOT); |
||||||
|
builder.append("con.execute(\"set @@execute_mode='").append(executeMode).append("';\")\n"); |
||||||
|
// offline job should be sync, and set job_timeout to 30min(==server.channel_keep_alive_time).
|
||||||
|
// You can set it longer in sqls.
|
||||||
|
if (executeMode.equals("offline")) { |
||||||
|
builder.append("con.execute(\"set @@sync_job=true\")\n"); |
||||||
|
builder.append("con.execute(\"set @@job_timeout=1800000\")\n"); |
||||||
|
} |
||||||
|
|
||||||
|
// split sql to list
|
||||||
|
// skip the sql only has space characters
|
||||||
|
Pattern pattern = Pattern.compile("\\S"); |
||||||
|
for (String sql : rawSqlScript.split(";")) { |
||||||
|
if (pattern.matcher(sql).find()) { |
||||||
|
sql = sql.replaceAll("\\n", "\\\\n"); |
||||||
|
builder.append("con.execute(\"").append(sql).append("\")\n"); |
||||||
|
} |
||||||
|
} |
||||||
|
return builder.toString(); |
||||||
|
} |
||||||
|
|
||||||
|
/** |
||||||
|
* Build the python task command. |
||||||
|
* If user have set the 'PYTHON_HOME' environment, we will use the 'PYTHON_HOME', |
||||||
|
* if not, we will default use python. |
||||||
|
* |
||||||
|
* @param pythonFile Python file, cannot be empty. |
||||||
|
* @return Python execute command, e.g. 'python test.py'. |
||||||
|
*/ |
||||||
|
@Override |
||||||
|
protected String buildPythonExecuteCommand(String pythonFile) { |
||||||
|
Preconditions.checkNotNull(pythonFile, "Python file cannot be null"); |
||||||
|
return getPythonCommand() + " " + pythonFile; |
||||||
|
} |
||||||
|
|
||||||
|
private String getPythonCommand() { |
||||||
|
String pythonHome = System.getenv(PYTHON_HOME); |
||||||
|
return getPythonCommand(pythonHome); |
||||||
|
} |
||||||
|
|
||||||
|
private String getPythonCommand(String pythonHome) { |
||||||
|
if (StringUtils.isEmpty(pythonHome)) { |
||||||
|
return OPENMLDB_PYTHON; |
||||||
|
} |
||||||
|
// If your python home is "xx/bin/python[xx]", you are forced to use python3
|
||||||
|
String pythonBinPath = "/bin/" + OPENMLDB_PYTHON; |
||||||
|
Matcher matcher = PYTHON_PATH_PATTERN.matcher(pythonHome); |
||||||
|
if (matcher.find()) { |
||||||
|
return matcher.replaceAll(pythonBinPath); |
||||||
|
} |
||||||
|
return Paths.get(pythonHome, pythonBinPath).toString(); |
||||||
|
} |
||||||
|
} |
||||||
@ -0,0 +1,47 @@ |
|||||||
|
/* |
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||||
|
* contributor license agreements. See the NOTICE file distributed with |
||||||
|
* this work for additional information regarding copyright ownership. |
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||||
|
* (the "License"); you may not use this file except in compliance with |
||||||
|
* the License. You may obtain a copy of the License at |
||||||
|
* |
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
* |
||||||
|
* Unless required by applicable law or agreed to in writing, software |
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS, |
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||||
|
* See the License for the specific language governing permissions and |
||||||
|
* limitations under the License. |
||||||
|
*/ |
||||||
|
|
||||||
|
package org.apache.dolphinscheduler.plugin.task.openmldb; |
||||||
|
|
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.TaskChannel; |
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext; |
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters; |
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.parameters.ParametersNode; |
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.parameters.resource.ResourceParametersHelper; |
||||||
|
import org.apache.dolphinscheduler.spi.utils.JSONUtils; |
||||||
|
|
||||||
|
public class OpenmldbTaskChannel implements TaskChannel { |
||||||
|
@Override |
||||||
|
public void cancelApplication(boolean status) { |
||||||
|
|
||||||
|
} |
||||||
|
|
||||||
|
@Override |
||||||
|
public OpenmldbTask createTask(TaskExecutionContext taskRequest) { |
||||||
|
return new OpenmldbTask(taskRequest); |
||||||
|
} |
||||||
|
|
||||||
|
@Override |
||||||
|
public AbstractParameters parseParameters(ParametersNode parametersNode) { |
||||||
|
return JSONUtils.parseObject(parametersNode.getTaskParams(), OpenmldbParameters.class); |
||||||
|
} |
||||||
|
|
||||||
|
@Override |
||||||
|
public ResourceParametersHelper getResources(String parameters) { |
||||||
|
return null; |
||||||
|
} |
||||||
|
} |
||||||
@ -0,0 +1,44 @@ |
|||||||
|
/* |
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||||
|
* contributor license agreements. See the NOTICE file distributed with |
||||||
|
* this work for additional information regarding copyright ownership. |
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||||
|
* (the "License"); you may not use this file except in compliance with |
||||||
|
* the License. You may obtain a copy of the License at |
||||||
|
* |
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
* |
||||||
|
* Unless required by applicable law or agreed to in writing, software |
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS, |
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||||
|
* See the License for the specific language governing permissions and |
||||||
|
* limitations under the License. |
||||||
|
*/ |
||||||
|
|
||||||
|
package org.apache.dolphinscheduler.plugin.task.openmldb; |
||||||
|
|
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.TaskChannel; |
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.TaskChannelFactory; |
||||||
|
import org.apache.dolphinscheduler.spi.params.base.PluginParams; |
||||||
|
|
||||||
|
import java.util.List; |
||||||
|
|
||||||
|
import com.google.auto.service.AutoService; |
||||||
|
|
||||||
|
@AutoService(TaskChannelFactory.class) |
||||||
|
public class OpenmldbTaskChannelFactory implements TaskChannelFactory { |
||||||
|
@Override |
||||||
|
public TaskChannel create() { |
||||||
|
return new OpenmldbTaskChannel(); |
||||||
|
} |
||||||
|
|
||||||
|
@Override |
||||||
|
public String getName() { |
||||||
|
return "OPENMLDB"; |
||||||
|
} |
||||||
|
|
||||||
|
@Override |
||||||
|
public List<PluginParams> getParams() { |
||||||
|
return null; |
||||||
|
} |
||||||
|
} |
||||||
@ -0,0 +1,89 @@ |
|||||||
|
/* |
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||||
|
* contributor license agreements. See the NOTICE file distributed with |
||||||
|
* this work for additional information regarding copyright ownership. |
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||||
|
* (the "License"); you may not use this file except in compliance with |
||||||
|
* the License. You may obtain a copy of the License at |
||||||
|
* |
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
* |
||||||
|
* Unless required by applicable law or agreed to in writing, software |
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS, |
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||||
|
* See the License for the specific language governing permissions and |
||||||
|
* limitations under the License. |
||||||
|
*/ |
||||||
|
|
||||||
|
package org.apache.dolphinscheduler.plugin.task.openmldb; |
||||||
|
|
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext; |
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.model.Property; |
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters; |
||||||
|
|
||||||
|
import java.util.HashMap; |
||||||
|
import java.util.Map; |
||||||
|
|
||||||
|
import org.junit.Assert; |
||||||
|
import org.junit.Test; |
||||||
|
import org.powermock.reflect.Whitebox; |
||||||
|
|
||||||
|
public class OpenmldbTaskTest { |
||||||
|
static class MockOpenmldbTask extends OpenmldbTask { |
||||||
|
/** |
||||||
|
* constructor |
||||||
|
* |
||||||
|
* @param taskRequest taskRequest |
||||||
|
*/ |
||||||
|
public MockOpenmldbTask(TaskExecutionContext taskRequest) { |
||||||
|
super(taskRequest); |
||||||
|
} |
||||||
|
|
||||||
|
@Override |
||||||
|
protected Map<String, Property> mergeParamsWithContext(AbstractParameters parameters) { |
||||||
|
return new HashMap<>(); |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
private OpenmldbTask createOpenmldbTask() { |
||||||
|
return new MockOpenmldbTask(null); |
||||||
|
} |
||||||
|
|
||||||
|
@Test |
||||||
|
public void buildPythonExecuteCommand() throws Exception { |
||||||
|
OpenmldbTask openmldbTask = createOpenmldbTask(); |
||||||
|
String pythonFile = "test.py"; |
||||||
|
String result1 = openmldbTask.buildPythonExecuteCommand(pythonFile); |
||||||
|
Assert.assertEquals("python3 test.py", result1); |
||||||
|
} |
||||||
|
|
||||||
|
@Test |
||||||
|
public void buildSQLWithComment() throws Exception { |
||||||
|
OpenmldbTask openmldbTask = createOpenmldbTask(); |
||||||
|
OpenmldbParameters openmldbParameters = new OpenmldbParameters(); |
||||||
|
openmldbParameters.setExecuteMode("offline"); |
||||||
|
String rawSQLScript = "select * from users\r\n" |
||||||
|
+ "-- some comment\n" |
||||||
|
+ "inner join order on users.order_id = order.id; \n\n;" |
||||||
|
+ "select * from users;"; |
||||||
|
openmldbParameters.setSql(rawSQLScript); |
||||||
|
Whitebox.setInternalState(openmldbTask, "openmldbParameters", openmldbParameters); |
||||||
|
OpenmldbParameters internal = (OpenmldbParameters) openmldbTask.getParameters(); |
||||||
|
Assert.assertNotNull(internal); |
||||||
|
Assert.assertEquals(internal.getExecuteMode(), "offline"); |
||||||
|
|
||||||
|
String result1 = openmldbTask.buildPythonScriptContent(); |
||||||
|
Assert.assertEquals("import openmldb\n" |
||||||
|
+ "import sqlalchemy as db\n" |
||||||
|
+ "engine = db.create_engine('openmldb:///?zk=null&zkPath=null')\n" |
||||||
|
+ "con = engine.connect()\n" |
||||||
|
+ "con.execute(\"set @@execute_mode='offline';\")\n" |
||||||
|
+ "con.execute(\"set @@sync_job=true\")\n" |
||||||
|
+ "con.execute(\"set @@job_timeout=1800000\")\n" |
||||||
|
+ "con.execute(\"select * from users\\n-- some comment\\ninner join order on users.order_id = " |
||||||
|
+ "order.id\")\n" |
||||||
|
+ "con.execute(\"select * from users\")\n" |
||||||
|
, result1); |
||||||
|
} |
||||||
|
|
||||||
|
} |
||||||
|
After Width: | Height: | Size: 88 KiB |
|
After Width: | Height: | Size: 10 KiB |
@ -0,0 +1,87 @@ |
|||||||
|
/* |
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||||
|
* contributor license agreements. See the NOTICE file distributed with |
||||||
|
* this work for additional information regarding copyright ownership. |
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||||
|
* (the "License"); you may not use this file except in compliance with |
||||||
|
* the License. You may obtain a copy of the License at |
||||||
|
* |
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
* |
||||||
|
* Unless required by applicable law or agreed to in writing, software |
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS, |
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||||
|
* See the License for the specific language governing permissions and |
||||||
|
* limitations under the License. |
||||||
|
*/ |
||||||
|
import { useI18n } from 'vue-i18n' |
||||||
|
import { useCustomParams, useResources } from '.' |
||||||
|
import type { IJsonItem } from '../types' |
||||||
|
|
||||||
|
export function useOpenmldb(model: { [field: string]: any }): IJsonItem[] { |
||||||
|
const { t } = useI18n() |
||||||
|
const options = [ |
||||||
|
{ |
||||||
|
label: t('project.node.openmldb_execute_mode_offline'), |
||||||
|
value: 'offline' |
||||||
|
}, |
||||||
|
{ |
||||||
|
label: t('project.node.openmldb_execute_mode_online'), |
||||||
|
value: 'online' |
||||||
|
} |
||||||
|
] |
||||||
|
return [ |
||||||
|
{ |
||||||
|
type: 'input', |
||||||
|
field: 'zk', |
||||||
|
name: t('project.node.openmldb_zk_address'), |
||||||
|
props: { |
||||||
|
placeholder: t('project.node.openmldb_zk_address_tips') |
||||||
|
}, |
||||||
|
validate: { |
||||||
|
trigger: ['input', 'blur'], |
||||||
|
required: true, |
||||||
|
validator(validate: any, value: string) { |
||||||
|
if (!value) { |
||||||
|
return new Error(t('project.node.openmldb_zk_address_tips')) |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
}, |
||||||
|
{ |
||||||
|
type: 'input', |
||||||
|
field: 'zkPath', |
||||||
|
name: t('project.node.openmldb_zk_path'), |
||||||
|
props: { |
||||||
|
placeholder: t('project.node.openmldb_zk_path_tips') |
||||||
|
}, |
||||||
|
validate: { |
||||||
|
trigger: ['input', 'blur'], |
||||||
|
required: true, |
||||||
|
validator(validate: any, value: string) { |
||||||
|
if (!value) { |
||||||
|
return new Error(t('project.node.openmldb_zk_path_tips')) |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
}, |
||||||
|
{ |
||||||
|
type: 'radio', |
||||||
|
field: 'executeMode', |
||||||
|
name: t('project.node.openmldb_execute_mode'), |
||||||
|
options: options |
||||||
|
}, |
||||||
|
{ |
||||||
|
type: 'editor', |
||||||
|
field: 'sql', |
||||||
|
name: t('project.node.sql_statement'), |
||||||
|
validate: { |
||||||
|
trigger: ['input', 'trigger'], |
||||||
|
required: true, |
||||||
|
message: t('project.node.sql_empty_tips') |
||||||
|
} |
||||||
|
}, |
||||||
|
useResources(), |
||||||
|
...useCustomParams({ model, field: 'localParams', isSimple: false }) |
||||||
|
] |
||||||
|
} |
||||||
@ -0,0 +1,85 @@ |
|||||||
|
/* |
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||||
|
* contributor license agreements. See the NOTICE file distributed with |
||||||
|
* this work for additional information regarding copyright ownership. |
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||||
|
* (the "License"); you may not use this file except in compliance with |
||||||
|
* the License. You may obtain a copy of the License at |
||||||
|
* |
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
* |
||||||
|
* Unless required by applicable law or agreed to in writing, software |
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS, |
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||||
|
* See the License for the specific language governing permissions and |
||||||
|
* limitations under the License. |
||||||
|
*/ |
||||||
|
|
||||||
|
import { reactive } from 'vue' |
||||||
|
import * as Fields from '../fields/index' |
||||||
|
import type { IJsonItem, INodeData } from '../types' |
||||||
|
import { ITaskData } from '../types' |
||||||
|
|
||||||
|
export function useOpenmldb({ |
||||||
|
projectCode, |
||||||
|
from = 0, |
||||||
|
readonly, |
||||||
|
data |
||||||
|
}: { |
||||||
|
projectCode: number |
||||||
|
from?: number |
||||||
|
readonly?: boolean |
||||||
|
data?: ITaskData |
||||||
|
}) { |
||||||
|
const model = reactive({ |
||||||
|
name: '', |
||||||
|
taskType: 'OPENMLDB', |
||||||
|
flag: 'YES', |
||||||
|
description: '', |
||||||
|
timeoutFlag: false, |
||||||
|
timeoutNotifyStrategy: ['WARN'], |
||||||
|
localParams: [], |
||||||
|
environmentCode: null, |
||||||
|
failRetryInterval: 1, |
||||||
|
failRetryTimes: 0, |
||||||
|
workerGroup: 'default', |
||||||
|
delayTime: 0, |
||||||
|
timeout: 30, |
||||||
|
zk: '', |
||||||
|
zkPath: '', |
||||||
|
executeMode: 'offline' |
||||||
|
} as INodeData) |
||||||
|
|
||||||
|
let extra: IJsonItem[] = [] |
||||||
|
if (from === 1) { |
||||||
|
extra = [ |
||||||
|
Fields.useTaskType(model, readonly), |
||||||
|
Fields.useProcessName({ |
||||||
|
model, |
||||||
|
projectCode, |
||||||
|
isCreate: !data?.id, |
||||||
|
from, |
||||||
|
processName: data?.processName |
||||||
|
}) |
||||||
|
] |
||||||
|
} |
||||||
|
|
||||||
|
return { |
||||||
|
json: [ |
||||||
|
Fields.useName(from), |
||||||
|
...extra, |
||||||
|
Fields.useRunFlag(), |
||||||
|
Fields.useDescription(), |
||||||
|
Fields.useTaskPriority(), |
||||||
|
Fields.useWorkerGroup(), |
||||||
|
Fields.useEnvironmentName(model, !model.id), |
||||||
|
...Fields.useTaskGroup(model, projectCode), |
||||||
|
...Fields.useFailed(), |
||||||
|
Fields.useDelayTime(model), |
||||||
|
...Fields.useTimeoutAlarm(model), |
||||||
|
...Fields.useOpenmldb(model), |
||||||
|
Fields.usePreTasks() |
||||||
|
] as IJsonItem[], |
||||||
|
model |
||||||
|
} |
||||||
|
} |
||||||
Loading…
Reference in new issue