JieguangZhou
2 years ago
committed by
GitHub
30 changed files with 1211 additions and 2 deletions
@ -0,0 +1,125 @@ |
|||||||
|
# DVC Node |
||||||
|
|
||||||
|
## Overview |
||||||
|
|
||||||
|
[DVC (Data Version Control)](https://dvc.org) is an excellent open-source version control system for machine learning projects. |
||||||
|
|
||||||
|
The DVC plugin is used to use the data version management function of DVC on DolphinScheduler, helping users to carry out data version management easily. |
||||||
|
|
||||||
|
The plugin provides the following three functions: |
||||||
|
|
||||||
|
- Init DVC: Initialize the Git repository as a DVC repository and bind the address where the data is stored to store the actual data. |
||||||
|
- Upload: Add or update specific data to the repository and record the version tag. |
||||||
|
- Download: Download a specific version of data from the repository. |
||||||
|
|
||||||
|
## Create Task |
||||||
|
|
||||||
|
- Click `Project -> Management-Project -> Name-Workflow Definition`, and click the "Create Workflow" button to enter the |
||||||
|
DAG editing page. |
||||||
|
- Drag from the toolbar <img src="../../../../img/tasks/icons/dvc.png" width="15"/> task node to canvas. |
||||||
|
|
||||||
|
## Task Example |
||||||
|
|
||||||
|
First, introduce some general parameters of DolphinScheduler: |
||||||
|
|
||||||
|
- **Node name**: The node name in a workflow definition is unique. |
||||||
|
- **Run flag**: Identifies whether this node schedules normally, if it does not need to execute, select |
||||||
|
the `prohibition execution`. |
||||||
|
- **Descriptive information**: Describe the function of the node. |
||||||
|
- **Task priority**: When the number of worker threads is insufficient, execute in the order of priority from high |
||||||
|
to low, and tasks with the same priority will execute in a first-in first-out order. |
||||||
|
- **Worker grouping**: Assign tasks to the machines of the worker group to execute. If `Default` is selected, |
||||||
|
randomly select a worker machine for execution. |
||||||
|
- **Environment Name**: Configure the environment name in which run the script. |
||||||
|
- **Times of failed retry attempts**: The number of times the task failed to resubmit. |
||||||
|
- **Failed retry interval**: The time interval (unit minute) for resubmitting the task after a failed task. |
||||||
|
- **Delayed execution time**: The time (unit minute) that a task delays in execution. |
||||||
|
- **Timeout alarm**: Check the timeout alarm and timeout failure. When the task runs exceed the "timeout", an alarm |
||||||
|
email will send and the task execution will fail. |
||||||
|
- **Predecessor task**: Selecting a predecessor task for the current task, will set the selected predecessor task as |
||||||
|
upstream of the current task. |
||||||
|
|
||||||
|
Here are some specific parameters for the DVC plugin: |
||||||
|
|
||||||
|
- **DVC Task Type** ：Upload, Download or Init DVC。 |
||||||
|
- **DVC Repository** ：The DVC repository address associated with the task execution. |
||||||
|
|
||||||
|
### Init DVC |
||||||
|
|
||||||
|
Initialize the Git repository as a DVC repository and add a new data remote to save data. |
||||||
|
|
||||||
|
After the project is initialized, it is still a Git repository, but with DVC features added. |
||||||
|
|
||||||
|
The data is not actually stored in a Git repository, but somewhere else, and DVC keeps track of the version and address of the data and handles this relationship. |
||||||
|
|
||||||
|
 |
||||||
|
|
||||||
|
**Task Parameter** |
||||||
|
|
||||||
|
- **Remote Store Url** ：The actual data is stored at the address. You can learn about the supported storage types from the [DVC supported storage types](https://dvc.org/doc/command-reference/remote/add#supported-storage-types) . |
||||||
|
|
||||||
|
The example above shows that: |
||||||
|
Initialize repository `git@github.com:<YOUR-NAME-OR-ORG>/dvc-data-repository-example.git` as a DVC project and bind the remote storage address to `~/dvc` |
||||||
|
|
||||||
|
### Upload |
||||||
|
|
||||||
|
Used to upload and update data and record version numbers. |
||||||
|
|
||||||
|
 |
||||||
|
|
||||||
|
**Task Parameter** |
||||||
|
|
||||||
|
- **Data Path in DVC Repository** ：The data will be uploaded to this path in the repository. |
||||||
|
- **Data Path In Worker** ：Data path to be uploaded. |
||||||
|
- **Version** ：After the data is uploaded, the version tag for the data will be saved in `git tag`. |
||||||
|
- **Version Message** ：Version Message. |
||||||
|
|
||||||
|
The example above shows that: |
||||||
|
|
||||||
|
Upload data `/home/data/iris` to the root directory of repository `git@github.com:<YOUR-NAME-OR-ORG>/dvc-data-repository-example.git`. The file or folder of data is named `iris`. |
||||||
|
|
||||||
|
Then run `git tag "iris_1.0" -m "init iris data"`. Record the version tag `iris_1.0` and the version message `inir iris data`. |
||||||
|
|
||||||
|
### Download |
||||||
|
|
||||||
|
Used to download data for a specific version. |
||||||
|
|
||||||
|
 |
||||||
|
|
||||||
|
**Task Parameter** |
||||||
|
|
||||||
|
- **Data Path in DVC Repository** ：The path to the data to download in the DVC repository. |
||||||
|
- **Data Path In Worker** ：Path for saving data after the file is downloaded to the local. |
||||||
|
- **Version** ：The version of the data to download. |
||||||
|
|
||||||
|
The example above shows that: |
||||||
|
|
||||||
|
Download the data for iris data at version `iris_1.0` in repository `git@github.com:<YOUR-NAME-OR-ORG>/dvc-data-repository-example.git` to the `~/dvc_test/iris` |
||||||
|
|
||||||
|
## Environment to prepare |
||||||
|
|
||||||
|
### Install DVC |
||||||
|
|
||||||
|
Make sure you have installed DVC, if not, you can run `pip install dvc` command to install it. |
||||||
|
|
||||||
|
Get the 'dvc' path and configure the environment variables. |
||||||
|
|
||||||
|
The conda environment is used as an example: |
||||||
|
|
||||||
|
Install python PIP on Conda and configure conda's environment variables so that the component can correctly find the 'DVC' command |
||||||
|
|
||||||
|
```shell |
||||||
|
which dvc |
||||||
|
# >> ~/anaconda3/bin/dvc |
||||||
|
``` |
||||||
|
|
||||||
|
You need to enter the admin account to configure a conda environment variable（Please |
||||||
|
install [anaconda](https://docs.continuum.io/anaconda/install/) |
||||||
|
or [miniconda](https://docs.conda.io/en/latest/miniconda.html#installing ) in advance). |
||||||
|
|
||||||
|
 |
||||||
|
|
||||||
|
Note During the configuration task, select the conda environment created above. Otherwise, the program cannot find the |
||||||
|
Conda environment. |
||||||
|
|
||||||
|
 |
||||||
@ -0,0 +1,110 @@ |
|||||||
|
# DVC节点 |
||||||
|
|
||||||
|
## 综述 |
||||||
|
|
||||||
|
[DVC（Data Version Control）](https://dvc.org) 是一个MLops领域一个优秀的开机器学习版本管理系统。 |
||||||
|
|
||||||
|
DVC 组件用于在DS上使用DVC的数据版本管理功能，帮助用户简易地进行数据的版本管理。组件提供如下三个功能： |
||||||
|
|
||||||
|
- Init DVC: 将git仓库初始化为DVC仓库，并绑定存储数据的地址用于存储实际的数据。 |
||||||
|
- Upload: 将特定数据添加或者更新到仓库中，并记录版本号。 |
||||||
|
- Download: 从仓库中下载特定版本的数据。 |
||||||
|
|
||||||
|
## 创建任务 |
||||||
|
|
||||||
|
- 点击项目管理-项目名称-工作流定义，点击“创建工作流”按钮，进入 DAG 编辑页面； |
||||||
|
- 拖动工具栏的 <img src="../../../../img/tasks/icons/dvc.png" width="15"/> 任务节点到画板中。 |
||||||
|
|
||||||
|
## 任务样例 |
||||||
|
|
||||||
|
首先介绍一些DS通用参数 |
||||||
|
|
||||||
|
- **节点名称** ：设置任务的名称。一个工作流定义中的节点名称是唯一的。 |
||||||
|
- **运行标志** ：标识这个节点是否能正常调度,如果不需要执行，可以打开禁止执行开关。 |
||||||
|
- **描述** ：描述该节点的功能。 |
||||||
|
- **任务优先级** ：worker 线程数不足时，根据优先级从高到低依次执行，优先级一样时根据先进先出原则执行。 |
||||||
|
- **Worker 分组** ：任务分配给 worker 组的机器执行，选择 Default，会随机选择一台 worker 机执行。 |
||||||
|
- **环境名称** ：配置运行脚本的环境。 |
||||||
|
- **失败重试次数** ：任务失败重新提交的次数。 |
||||||
|
- **失败重试间隔** ：任务失败重新提交任务的时间间隔，以分钟为单位。 |
||||||
|
- **延迟执行时间** ：任务延迟执行的时间，以分钟为单位。 |
||||||
|
- **超时告警** ：勾选超时告警、超时失败，当任务超过"超时时长"后，会发送告警邮件并且任务执行失败。 |
||||||
|
- **前置任务** ：选择当前任务的前置任务，会将被选择的前置任务设置为当前任务的上游。 |
||||||
|
|
||||||
|
以下是一些DVC 组件的常用参数 |
||||||
|
|
||||||
|
- **DVC任务类型** ：可以选择 Upload、Download、Init DVC。 |
||||||
|
- **DVC仓库** ：任务执行时关联的仓库地址。 |
||||||
|
|
||||||
|
### Init DVC |
||||||
|
|
||||||
|
|
||||||
|
将git仓库初始化为DVC仓库, 并绑定数据储存的地方。 |
||||||
|
|
||||||
|
项目初始化后，仍然为git仓库，不过添加了DVC的特性。 |
||||||
|
|
||||||
|
实际上数据并不保存在git仓库，而是存储在另外的地方，DVC会跟踪数据的版本和地址，并处理好这个关系。 |
||||||
|
|
||||||
|
 |
||||||
|
|
||||||
|
**任务参数** |
||||||
|
|
||||||
|
- **数据存储地址** |
||||||
|
：实际的数据保存的地址，支持的类型可见 [DVC supported storage types](https://dvc.org/doc/command-reference/remote/add#supported-storage-types) |
||||||
|
。 |
||||||
|
|
||||||
|
如上述例子表示： 将仓库 `git@github.com:<YOUR-NAME-OR-ORG>/dvc-data-repository-example.git` 初始化为DVC项目，并绑定远程储存地址为 `~/dvc` |
||||||
|
|
||||||
|
### Upload |
||||||
|
|
||||||
|
用于上传和更新数据，并记录版本号。 |
||||||
|
|
||||||
|
 |
||||||
|
|
||||||
|
**任务参数** |
||||||
|
|
||||||
|
- **DVC仓库中的数据路径** ：上传的数据保存到仓库的地址。 |
||||||
|
- **Worker中数据路径** ：需要上传的数据的地址。 |
||||||
|
- **数据版本** ：上传数据后，为该版本数据打上的版本号，会保存到 git tag 里面。 |
||||||
|
- **数据版本信息** ：本次上传需要备注的信息。 |
||||||
|
|
||||||
|
如上述例子表示： 将数据 `/home/data/iris` 上传到仓库 `git@github.com:<YOUR-NAME-OR-ORG>/dvc-data-repository-example.git` |
||||||
|
的根目录下，数据的文件/文件夹名字为`iris`。 然后执行 `git tag "iris_1.0" -m "init iris data"`。 记录版本号 `iris_1.0`和 版本信息 'inir iris data' |
||||||
|
|
||||||
|
### Download |
||||||
|
|
||||||
|
用于下载特定版本的数据。 |
||||||
|
|
||||||
|
 |
||||||
|
|
||||||
|
**任务参数** |
||||||
|
|
||||||
|
- **DVC仓库中的数据路径** ：需要下载数据在仓库中的路径。 |
||||||
|
- **Worker中数据路径** ：数据下载到本地后的保存地址。 |
||||||
|
- **数据版本** ：需要下载的数据的版本。 |
||||||
|
|
||||||
|
如上述例子表示： 将仓库 `git@github.com:xxxx/dvc-data-repository-example.git` 版本为 `iris_1.0` 的 iris 的数据下载到 `~/dvc_test/iris` |
||||||
|
|
||||||
|
## 环境准备 |
||||||
|
|
||||||
|
### dvc 安装 |
||||||
|
|
||||||
|
确保你已经安装DVC可以使用`pip install dvc`进行安装。 |
||||||
|
|
||||||
|
获取dvc地址, 并配置环境变量 |
||||||
|
|
||||||
|
下面以 conda 上的 python pip 安装为例子，配置 conda 的环境变量，使得组件能正确找到`dvc`命令 |
||||||
|
|
||||||
|
```shell |
||||||
|
which dvc |
||||||
|
# >> ~/anaconda3/bin/dvc |
||||||
|
``` |
||||||
|
|
||||||
|
你需要进入admin账户配置一个conda环境变量（请提前[安装anaconda](https://docs.continuum.io/anaconda/install/) |
||||||
|
或者[安装miniconda](https://docs.conda.io/en/latest/miniconda.html#installing) )。 |
||||||
|
|
||||||
|
 |
||||||
|
|
||||||
|
后续注意配置任务时，环境选择上面创建的conda环境，否则程序会找不到conda环境。 |
||||||
|
|
||||||
|
 |
||||||
|
After Width: | Height: | Size: 24 KiB |
|
After Width: | Height: | Size: 119 KiB |
|
After Width: | Height: | Size: 49 KiB |
|
After Width: | Height: | Size: 16 KiB |
|
After Width: | Height: | Size: 28 KiB |
|
After Width: | Height: | Size: 7.5 KiB |
@ -0,0 +1,46 @@ |
|||||||
|
<?xml version="1.0" encoding="UTF-8"?> |
||||||
|
<!-- |
||||||
|
~ Licensed to the Apache Software Foundation (ASF) under one or more |
||||||
|
~ contributor license agreements. See the NOTICE file distributed with |
||||||
|
~ this work for additional information regarding copyright ownership. |
||||||
|
~ The ASF licenses this file to You under the Apache License, Version 2.0 |
||||||
|
~ (the "License"); you may not use this file except in compliance with |
||||||
|
~ the License. You may obtain a copy of the License at |
||||||
|
~ |
||||||
|
~ http://www.apache.org/licenses/LICENSE-2.0 |
||||||
|
~ |
||||||
|
~ Unless required by applicable law or agreed to in writing, software |
||||||
|
~ distributed under the License is distributed on an "AS IS" BASIS, |
||||||
|
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||||
|
~ See the License for the specific language governing permissions and |
||||||
|
~ limitations under the License. |
||||||
|
--> |
||||||
|
<project xmlns="http://maven.apache.org/POM/4.0.0" |
||||||
|
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" |
||||||
|
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> |
||||||
|
<parent> |
||||||
|
<artifactId>dolphinscheduler-task-plugin</artifactId> |
||||||
|
<groupId>org.apache.dolphinscheduler</groupId> |
||||||
|
<version>dev-SNAPSHOT</version> |
||||||
|
</parent> |
||||||
|
<modelVersion>4.0.0</modelVersion> |
||||||
|
|
||||||
|
<artifactId>dolphinscheduler-task-dvc</artifactId> |
||||||
|
<packaging>jar</packaging> |
||||||
|
|
||||||
|
<dependencies> |
||||||
|
<dependency> |
||||||
|
<groupId>org.apache.dolphinscheduler</groupId> |
||||||
|
<artifactId>dolphinscheduler-spi</artifactId> |
||||||
|
<scope>provided</scope> |
||||||
|
</dependency> |
||||||
|
<dependency> |
||||||
|
<groupId>org.apache.dolphinscheduler</groupId> |
||||||
|
<artifactId>dolphinscheduler-task-api</artifactId> |
||||||
|
</dependency> |
||||||
|
<dependency> |
||||||
|
<groupId>org.apache.commons</groupId> |
||||||
|
<artifactId>commons-collections4</artifactId> |
||||||
|
</dependency> |
||||||
|
</dependencies> |
||||||
|
</project> |
||||||
@ -0,0 +1,56 @@ |
|||||||
|
/* |
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||||
|
* contributor license agreements. See the NOTICE file distributed with |
||||||
|
* this work for additional information regarding copyright ownership. |
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||||
|
* (the "License"); you may not use this file except in compliance with |
||||||
|
* the License. You may obtain a copy of the License at |
||||||
|
* |
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
* |
||||||
|
* Unless required by applicable law or agreed to in writing, software |
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS, |
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||||
|
* See the License for the specific language governing permissions and |
||||||
|
* limitations under the License. |
||||||
|
*/ |
||||||
|
|
||||||
|
package org.apache.dolphinscheduler.plugin.task.dvc; |
||||||
|
|
||||||
|
public class DvcConstants { |
||||||
|
private DvcConstants() { |
||||||
|
throw new IllegalStateException("Utility class"); |
||||||
|
} |
||||||
|
|
||||||
|
public static final String CHECK_AND_SET_DVC_REPO = "which dvc || { echo \"dvc does not exist\"; exit 1; }; DVC_REPO=%s"; |
||||||
|
|
||||||
|
public static final String SET_DATA_PATH = "DVC_DATA_PATH=%s"; |
||||||
|
|
||||||
|
public static final String SET_DATA_LOCATION = "DVC_DATA_LOCATION=%s"; |
||||||
|
|
||||||
|
public static final String SET_VERSION = "DVC_VERSION=%s"; |
||||||
|
|
||||||
|
public static final String SET_MESSAGE = "DVC_MESSAGE=\"%s\""; |
||||||
|
|
||||||
|
public static final String GIT_CLONE_DVC_REPO = "git clone $DVC_REPO dvc-repository; cd dvc-repository; pwd"; |
||||||
|
|
||||||
|
public static final String DVC_AUTOSTAGE = "dvc config core.autostage true --local || exit 1"; |
||||||
|
|
||||||
|
public static final String DVC_ADD_DATA = "dvc add $DVC_DATA_PATH -v -o $DVC_DATA_LOCATION --to-remote || exit 1"; |
||||||
|
|
||||||
|
public static final String GIT_UPDATE_FOR_UPDATE_DATA = "git commit -am \"$DVC_MESSAGE\"\n" + |
||||||
|
"git tag \"$DVC_VERSION\" -m \"$DVC_MESSAGE\"\n" + |
||||||
|
"git push --all\n" + |
||||||
|
"git push --tags"; |
||||||
|
|
||||||
|
public static final String DVC_DOWNLOAD = "dvc get $DVC_REPO $DVC_DATA_LOCATION -o $DVC_DATA_PATH -v --rev $DVC_VERSION"; |
||||||
|
|
||||||
|
|
||||||
|
public static final String DVC_INIT = "dvc init || exit 1"; |
||||||
|
|
||||||
|
public static final String DVC_ADD_REMOTE = "dvc remote add origin %s -d"; |
||||||
|
|
||||||
|
public static final String GIT_UPDATE_FOR_INIT_DVC = "git commit -am \"init dvc project and add remote\"; git push"; |
||||||
|
|
||||||
|
} |
||||||
|
|
||||||
@ -0,0 +1,105 @@ |
|||||||
|
/* |
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||||
|
* contributor license agreements. See the NOTICE file distributed with |
||||||
|
* this work for additional information regarding copyright ownership. |
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||||
|
* (the "License"); you may not use this file except in compliance with |
||||||
|
* the License. You may obtain a copy of the License at |
||||||
|
* |
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
* |
||||||
|
* Unless required by applicable law or agreed to in writing, software |
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS, |
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||||
|
* See the License for the specific language governing permissions and |
||||||
|
* limitations under the License. |
||||||
|
*/ |
||||||
|
|
||||||
|
package org.apache.dolphinscheduler.plugin.task.dvc; |
||||||
|
|
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters; |
||||||
|
|
||||||
|
public class DvcParameters extends AbstractParameters { |
||||||
|
|
||||||
|
/** |
||||||
|
* common parameters |
||||||
|
*/ |
||||||
|
|
||||||
|
private TaskTypeEnum dvcTaskType; |
||||||
|
|
||||||
|
private String dvcRepository; |
||||||
|
|
||||||
|
private String dvcVersion; |
||||||
|
|
||||||
|
private String dvcDataLocation; |
||||||
|
|
||||||
|
private String dvcMessage; |
||||||
|
|
||||||
|
private String dvcLoadSaveDataPath; |
||||||
|
|
||||||
|
private String dvcStoreUrl; |
||||||
|
|
||||||
|
public void setDvcTaskType(TaskTypeEnum dvcTaskType) { |
||||||
|
this.dvcTaskType = dvcTaskType; |
||||||
|
} |
||||||
|
|
||||||
|
public TaskTypeEnum getDvcTaskType() { |
||||||
|
return dvcTaskType; |
||||||
|
} |
||||||
|
|
||||||
|
public void setDvcRepository(String dvcRepository) { |
||||||
|
this.dvcRepository = dvcRepository; |
||||||
|
} |
||||||
|
|
||||||
|
public String getDvcRepository() { |
||||||
|
return dvcRepository; |
||||||
|
} |
||||||
|
|
||||||
|
public void setDvcVersion(String dvcVersion) { |
||||||
|
this.dvcVersion = dvcVersion; |
||||||
|
} |
||||||
|
|
||||||
|
public String getDvcVersion() { |
||||||
|
return dvcVersion; |
||||||
|
} |
||||||
|
|
||||||
|
public void setDvcDataLocation(String dvcDataLocation) { |
||||||
|
this.dvcDataLocation = dvcDataLocation; |
||||||
|
} |
||||||
|
|
||||||
|
public String getDvcDataLocation() { |
||||||
|
return dvcDataLocation; |
||||||
|
} |
||||||
|
|
||||||
|
public void setDvcMessage(String dvcMessage) { |
||||||
|
this.dvcMessage = dvcMessage; |
||||||
|
} |
||||||
|
|
||||||
|
public String getDvcMessage() { |
||||||
|
return dvcMessage; |
||||||
|
} |
||||||
|
|
||||||
|
public void setDvcLoadSaveDataPath(String dvcLoadSaveDataPath) { |
||||||
|
this.dvcLoadSaveDataPath = dvcLoadSaveDataPath; |
||||||
|
} |
||||||
|
|
||||||
|
public String getDvcLoadSaveDataPath() { |
||||||
|
return dvcLoadSaveDataPath; |
||||||
|
} |
||||||
|
|
||||||
|
public void setDvcStoreUrl(String dvcStoreUrl) { |
||||||
|
this.dvcStoreUrl = dvcStoreUrl; |
||||||
|
} |
||||||
|
|
||||||
|
public String getDvcStoreUrl() { |
||||||
|
return dvcStoreUrl; |
||||||
|
} |
||||||
|
|
||||||
|
@Override |
||||||
|
public boolean checkParameters() { |
||||||
|
Boolean checkResult = true; |
||||||
|
return checkResult; |
||||||
|
} |
||||||
|
|
||||||
|
} |
||||||
|
|
||||||
@ -0,0 +1,163 @@ |
|||||||
|
/* |
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||||
|
* contributor license agreements. See the NOTICE file distributed with |
||||||
|
* this work for additional information regarding copyright ownership. |
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||||
|
* (the "License"); you may not use this file except in compliance with |
||||||
|
* the License. You may obtain a copy of the License at |
||||||
|
* |
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
* |
||||||
|
* Unless required by applicable law or agreed to in writing, software |
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS, |
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||||
|
* See the License for the specific language governing permissions and |
||||||
|
* limitations under the License. |
||||||
|
*/ |
||||||
|
|
||||||
|
package org.apache.dolphinscheduler.plugin.task.dvc; |
||||||
|
|
||||||
|
import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.EXIT_CODE_FAILURE; |
||||||
|
|
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.AbstractTaskExecutor; |
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.ShellCommandExecutor; |
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext; |
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.model.TaskResponse; |
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters; |
||||||
|
import org.apache.dolphinscheduler.spi.utils.JSONUtils; |
||||||
|
|
||||||
|
import java.util.ArrayList; |
||||||
|
import java.util.List; |
||||||
|
|
||||||
|
/** |
||||||
|
* shell task |
||||||
|
*/ |
||||||
|
public class DvcTask extends AbstractTaskExecutor { |
||||||
|
|
||||||
|
/** |
||||||
|
* dvc parameters |
||||||
|
*/ |
||||||
|
private DvcParameters parameters; |
||||||
|
|
||||||
|
/** |
||||||
|
* shell command executor |
||||||
|
*/ |
||||||
|
private ShellCommandExecutor shellCommandExecutor; |
||||||
|
|
||||||
|
/** |
||||||
|
* taskExecutionContext |
||||||
|
*/ |
||||||
|
private TaskExecutionContext taskExecutionContext; |
||||||
|
|
||||||
|
/** |
||||||
|
* constructor |
||||||
|
* |
||||||
|
* @param taskExecutionContext taskExecutionContext |
||||||
|
*/ |
||||||
|
public DvcTask(TaskExecutionContext taskExecutionContext) { |
||||||
|
super(taskExecutionContext); |
||||||
|
|
||||||
|
this.taskExecutionContext = taskExecutionContext; |
||||||
|
this.shellCommandExecutor = new ShellCommandExecutor(this::logHandle, taskExecutionContext, logger); |
||||||
|
} |
||||||
|
|
||||||
|
@Override |
||||||
|
public void init() { |
||||||
|
logger.info("dvc task params {}", taskExecutionContext.getTaskParams()); |
||||||
|
|
||||||
|
parameters = JSONUtils.parseObject(taskExecutionContext.getTaskParams(), DvcParameters.class); |
||||||
|
|
||||||
|
if (!parameters.checkParameters()) { |
||||||
|
throw new RuntimeException("dvc task params is not valid"); |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
@Override |
||||||
|
public void handle() throws Exception { |
||||||
|
try { |
||||||
|
// construct process
|
||||||
|
String command = buildCommand(); |
||||||
|
TaskResponse commandExecuteResult = shellCommandExecutor.run(command); |
||||||
|
setExitStatusCode(commandExecuteResult.getExitStatusCode()); |
||||||
|
setAppIds(commandExecuteResult.getAppIds()); |
||||||
|
setProcessId(commandExecuteResult.getProcessId()); |
||||||
|
parameters.dealOutParam(shellCommandExecutor.getVarPool()); |
||||||
|
} catch (Exception e) { |
||||||
|
logger.error("dvc task error", e); |
||||||
|
setExitStatusCode(EXIT_CODE_FAILURE); |
||||||
|
throw e; |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
@Override |
||||||
|
public void cancelApplication(boolean cancelApplication) throws Exception { |
||||||
|
// cancel process
|
||||||
|
shellCommandExecutor.cancelApplication(); |
||||||
|
} |
||||||
|
|
||||||
|
public String buildCommand() { |
||||||
|
String command = ""; |
||||||
|
TaskTypeEnum taskType = parameters.getDvcTaskType(); |
||||||
|
if (taskType == TaskTypeEnum.UPLOAD) { |
||||||
|
command = buildUploadCommond(); |
||||||
|
}else if (taskType == TaskTypeEnum.DOWNLOAD){ |
||||||
|
command = buildDownCommond(); |
||||||
|
}else if (taskType == TaskTypeEnum.INIT){ |
||||||
|
command = buildInitDvcCommond(); |
||||||
|
} |
||||||
|
logger.info("Run DVC task with command: \n{}", command); |
||||||
|
return command; |
||||||
|
} |
||||||
|
|
||||||
|
private String buildUploadCommond() { |
||||||
|
List<String> args = new ArrayList<>(); |
||||||
|
args.add(String.format(DvcConstants.CHECK_AND_SET_DVC_REPO, parameters.getDvcRepository())); |
||||||
|
args.add(String.format(DvcConstants.SET_DATA_PATH, parameters.getDvcLoadSaveDataPath())); |
||||||
|
args.add(String.format(DvcConstants.SET_DATA_LOCATION, parameters.getDvcDataLocation())); |
||||||
|
args.add(String.format(DvcConstants.SET_VERSION, parameters.getDvcVersion())); |
||||||
|
args.add(String.format(DvcConstants.SET_MESSAGE, parameters.getDvcMessage())); |
||||||
|
args.add(DvcConstants.GIT_CLONE_DVC_REPO); |
||||||
|
args.add(DvcConstants.DVC_AUTOSTAGE); |
||||||
|
args.add(DvcConstants.DVC_ADD_DATA); |
||||||
|
args.add(DvcConstants.GIT_UPDATE_FOR_UPDATE_DATA); |
||||||
|
|
||||||
|
String command = String.join("\n", args); |
||||||
|
return command; |
||||||
|
|
||||||
|
} |
||||||
|
|
||||||
|
private String buildDownCommond() { |
||||||
|
List<String> args = new ArrayList<>(); |
||||||
|
args.add(String.format(DvcConstants.CHECK_AND_SET_DVC_REPO, parameters.getDvcRepository())); |
||||||
|
args.add(String.format(DvcConstants.SET_DATA_PATH, parameters.getDvcLoadSaveDataPath())); |
||||||
|
args.add(String.format(DvcConstants.SET_DATA_LOCATION, parameters.getDvcDataLocation())); |
||||||
|
args.add(String.format(DvcConstants.SET_VERSION, parameters.getDvcVersion())); |
||||||
|
args.add(DvcConstants.DVC_DOWNLOAD); |
||||||
|
|
||||||
|
String command = String.join("\n", args); |
||||||
|
return command; |
||||||
|
|
||||||
|
} |
||||||
|
|
||||||
|
private String buildInitDvcCommond() { |
||||||
|
List<String> args = new ArrayList<>(); |
||||||
|
args.add(String.format(DvcConstants.CHECK_AND_SET_DVC_REPO, parameters.getDvcRepository())); |
||||||
|
args.add(DvcConstants.GIT_CLONE_DVC_REPO); |
||||||
|
args.add(DvcConstants.DVC_INIT); |
||||||
|
args.add(String.format(DvcConstants.DVC_ADD_REMOTE, parameters.getDvcStoreUrl())); |
||||||
|
args.add(DvcConstants.GIT_UPDATE_FOR_INIT_DVC); |
||||||
|
|
||||||
|
String command = String.join("\n", args); |
||||||
|
return command; |
||||||
|
|
||||||
|
} |
||||||
|
|
||||||
|
|
||||||
|
@Override |
||||||
|
public AbstractParameters getParameters() { |
||||||
|
return parameters; |
||||||
|
} |
||||||
|
|
||||||
|
|
||||||
|
} |
||||||
|
|
||||||
@ -0,0 +1,51 @@ |
|||||||
|
/* |
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||||
|
* contributor license agreements. See the NOTICE file distributed with |
||||||
|
* this work for additional information regarding copyright ownership. |
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||||
|
* (the "License"); you may not use this file except in compliance with |
||||||
|
* the License. You may obtain a copy of the License at |
||||||
|
* |
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
* |
||||||
|
* Unless required by applicable law or agreed to in writing, software |
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS, |
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||||
|
* See the License for the specific language governing permissions and |
||||||
|
* limitations under the License. |
||||||
|
*/ |
||||||
|
|
||||||
|
package org.apache.dolphinscheduler.plugin.task.dvc; |
||||||
|
|
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.TaskChannel; |
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext; |
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters; |
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.parameters.ParametersNode; |
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.parameters.resource.ResourceParametersHelper; |
||||||
|
import org.apache.dolphinscheduler.spi.utils.JSONUtils; |
||||||
|
|
||||||
|
|
||||||
|
public class DvcTaskChannel implements TaskChannel { |
||||||
|
|
||||||
|
@Override |
||||||
|
public void cancelApplication(boolean status) { |
||||||
|
|
||||||
|
} |
||||||
|
|
||||||
|
@Override |
||||||
|
public DvcTask createTask(TaskExecutionContext taskRequest) { |
||||||
|
return new DvcTask(taskRequest); |
||||||
|
} |
||||||
|
|
||||||
|
@Override |
||||||
|
public AbstractParameters parseParameters(ParametersNode parametersNode) { |
||||||
|
return JSONUtils.parseObject(parametersNode.getTaskParams(), DvcParameters.class); |
||||||
|
} |
||||||
|
|
||||||
|
@Override |
||||||
|
public ResourceParametersHelper getResources(String parameters) { |
||||||
|
return null; |
||||||
|
} |
||||||
|
|
||||||
|
} |
||||||
|
|
||||||
@ -0,0 +1,58 @@ |
|||||||
|
/* |
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||||
|
* contributor license agreements. See the NOTICE file distributed with |
||||||
|
* this work for additional information regarding copyright ownership. |
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||||
|
* (the "License"); you may not use this file except in compliance with |
||||||
|
* the License. You may obtain a copy of the License at |
||||||
|
* |
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
* |
||||||
|
* Unless required by applicable law or agreed to in writing, software |
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS, |
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||||
|
* See the License for the specific language governing permissions and |
||||||
|
* limitations under the License. |
||||||
|
*/ |
||||||
|
|
||||||
|
package org.apache.dolphinscheduler.plugin.task.dvc; |
||||||
|
|
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.TaskChannel; |
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.TaskChannelFactory; |
||||||
|
import org.apache.dolphinscheduler.spi.params.base.ParamsOptions; |
||||||
|
import org.apache.dolphinscheduler.spi.params.base.PluginParams; |
||||||
|
import org.apache.dolphinscheduler.spi.params.base.Validate; |
||||||
|
import org.apache.dolphinscheduler.spi.params.input.InputParam; |
||||||
|
import org.apache.dolphinscheduler.spi.params.radio.RadioParam; |
||||||
|
|
||||||
|
import java.util.ArrayList; |
||||||
|
import java.util.List; |
||||||
|
|
||||||
|
import com.google.auto.service.AutoService; |
||||||
|
|
||||||
|
@AutoService(TaskChannelFactory.class) |
||||||
|
public class DvcTaskChannelFactory implements TaskChannelFactory { |
||||||
|
@Override |
||||||
|
public TaskChannel create() { |
||||||
|
return new DvcTaskChannel(); |
||||||
|
} |
||||||
|
|
||||||
|
@Override |
||||||
|
public String getName() { |
||||||
|
return "DVC"; |
||||||
|
} |
||||||
|
|
||||||
|
@Override |
||||||
|
public List<PluginParams> getParams() { |
||||||
|
List<PluginParams> paramsList = new ArrayList<>(); |
||||||
|
|
||||||
|
InputParam nodeName = InputParam.newBuilder("name", "$t('Node name')").addValidate(Validate.newBuilder().setRequired(true).build()).build(); |
||||||
|
|
||||||
|
RadioParam runFlag = RadioParam.newBuilder("runFlag", "RUN_FLAG").addParamsOptions(new ParamsOptions("NORMAL", "NORMAL", false)).addParamsOptions(new ParamsOptions("FORBIDDEN", "FORBIDDEN", false)).build(); |
||||||
|
|
||||||
|
paramsList.add(nodeName); |
||||||
|
paramsList.add(runFlag); |
||||||
|
return paramsList; |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
@ -0,0 +1,30 @@ |
|||||||
|
/* |
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||||
|
* contributor license agreements. See the NOTICE file distributed with |
||||||
|
* this work for additional information regarding copyright ownership. |
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||||
|
* (the "License"); you may not use this file except in compliance with |
||||||
|
* the License. You may obtain a copy of the License at |
||||||
|
* |
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
* |
||||||
|
* Unless required by applicable law or agreed to in writing, software |
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS, |
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||||
|
* See the License for the specific language governing permissions and |
||||||
|
* limitations under the License. |
||||||
|
*/ |
||||||
|
|
||||||
|
package org.apache.dolphinscheduler.plugin.task.dvc; |
||||||
|
|
||||||
|
import com.fasterxml.jackson.annotation.JsonProperty; |
||||||
|
|
||||||
|
public enum TaskTypeEnum { |
||||||
|
|
||||||
|
@JsonProperty("Upload") |
||||||
|
UPLOAD, |
||||||
|
@JsonProperty("Download") |
||||||
|
DOWNLOAD, |
||||||
|
@JsonProperty("Init DVC") |
||||||
|
INIT |
||||||
|
} |
||||||
@ -0,0 +1,153 @@ |
|||||||
|
/* |
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||||
|
* contributor license agreements. See the NOTICE file distributed with |
||||||
|
* this work for additional information regarding copyright ownership. |
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||||
|
* (the "License"); you may not use this file except in compliance with |
||||||
|
* the License. You may obtain a copy of the License at |
||||||
|
* |
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
* |
||||||
|
* Unless required by applicable law or agreed to in writing, software |
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS, |
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||||
|
* See the License for the specific language governing permissions and |
||||||
|
* limitations under the License. |
||||||
|
*/ |
||||||
|
|
||||||
|
package org.apache.dolphinscheduler.plugin.task.dvc; |
||||||
|
|
||||||
|
import java.util.Date; |
||||||
|
import java.util.UUID; |
||||||
|
|
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext; |
||||||
|
import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContextCacheManager; |
||||||
|
import org.junit.Assert; |
||||||
|
import org.junit.Before; |
||||||
|
import org.junit.Test; |
||||||
|
import org.mockito.Mockito; |
||||||
|
import org.powermock.api.mockito.PowerMockito; |
||||||
|
import org.slf4j.Logger; |
||||||
|
import org.slf4j.LoggerFactory; |
||||||
|
import org.apache.dolphinscheduler.spi.utils.PropertyUtils; |
||||||
|
|
||||||
|
import org.powermock.core.classloader.annotations.PrepareForTest; |
||||||
|
import org.powermock.core.classloader.annotations.PowerMockIgnore; |
||||||
|
import org.junit.runner.RunWith; |
||||||
|
import org.powermock.modules.junit4.PowerMockRunner; |
||||||
|
import org.powermock.core.classloader.annotations.SuppressStaticInitializationFor; |
||||||
|
import org.apache.dolphinscheduler.spi.utils.JSONUtils; |
||||||
|
|
||||||
|
|
||||||
|
@RunWith(PowerMockRunner.class) |
||||||
|
@PrepareForTest({ |
||||||
|
JSONUtils.class, |
||||||
|
PropertyUtils.class, |
||||||
|
}) |
||||||
|
@PowerMockIgnore({"javax.*"}) |
||||||
|
@SuppressStaticInitializationFor("org.apache.dolphinscheduler.spi.utils.PropertyUtils") |
||||||
|
public class DvcTaskTest { |
||||||
|
|
||||||
|
@Before |
||||||
|
public void before() throws Exception { |
||||||
|
PowerMockito.mockStatic(PropertyUtils.class); |
||||||
|
} |
||||||
|
|
||||||
|
public TaskExecutionContext createContext(DvcParameters dvcParameters) { |
||||||
|
String parameters = JSONUtils.toJsonString(dvcParameters); |
||||||
|
TaskExecutionContext taskExecutionContext = Mockito.mock(TaskExecutionContext.class); |
||||||
|
Mockito.when(taskExecutionContext.getTaskParams()).thenReturn(parameters); |
||||||
|
Mockito.when(taskExecutionContext.getTaskLogName()).thenReturn("DvcTest"); |
||||||
|
Mockito.when(taskExecutionContext.getExecutePath()).thenReturn("/tmp/dolphinscheduler_dvc_test"); |
||||||
|
Mockito.when(taskExecutionContext.getTaskAppId()).thenReturn(UUID.randomUUID().toString()); |
||||||
|
Mockito.when(taskExecutionContext.getStartTime()).thenReturn(new Date()); |
||||||
|
Mockito.when(taskExecutionContext.getTaskTimeout()).thenReturn(10000); |
||||||
|
Mockito.when(taskExecutionContext.getLogPath()).thenReturn("/tmp/dolphinscheduler_dvc_test/log"); |
||||||
|
Mockito.when(taskExecutionContext.getEnvironmentConfig()).thenReturn("export PATH=$HOME/anaconda3/bin:$PATH"); |
||||||
|
|
||||||
|
String userName = System.getenv().get("USER"); |
||||||
|
Mockito.when(taskExecutionContext.getTenantCode()).thenReturn(userName); |
||||||
|
|
||||||
|
TaskExecutionContextCacheManager.cacheTaskExecutionContext(taskExecutionContext); |
||||||
|
return taskExecutionContext; |
||||||
|
} |
||||||
|
|
||||||
|
private DvcTask initTask(DvcParameters parameters) { |
||||||
|
TaskExecutionContext taskExecutionContext = createContext(parameters); |
||||||
|
DvcTask dvcTask = new DvcTask(taskExecutionContext); |
||||||
|
dvcTask.init(); |
||||||
|
dvcTask.getParameters().setVarPool(taskExecutionContext.getVarPool()); |
||||||
|
return dvcTask; |
||||||
|
|
||||||
|
} |
||||||
|
|
||||||
|
@Test |
||||||
|
public void testDvcUpload() throws Exception{ |
||||||
|
DvcTask dvcTask = initTask(createUploadParameters()); |
||||||
|
Assert.assertEquals(dvcTask.buildCommand(), |
||||||
|
"which dvc || { echo \"dvc does not exist\"; exit 1; }; DVC_REPO=git@github.com:<YOUR-NAME-OR-ORG>/dvc-data-repository-example\n" + |
||||||
|
"DVC_DATA_PATH=/home/<YOUR-NAME-OR-ORG>/test\n" + |
||||||
|
"DVC_DATA_LOCATION=test\n" + |
||||||
|
"DVC_VERSION=iris_v2.3.1\n" + |
||||||
|
"DVC_MESSAGE=\"add test iris data\"\n" + |
||||||
|
"git clone $DVC_REPO dvc-repository; cd dvc-repository; pwd\n" + |
||||||
|
"dvc config core.autostage true --local || exit 1\n" + |
||||||
|
"dvc add $DVC_DATA_PATH -v -o $DVC_DATA_LOCATION --to-remote || exit 1\n" + |
||||||
|
"git commit -am \"$DVC_MESSAGE\"\n" + |
||||||
|
"git tag \"$DVC_VERSION\" -m \"$DVC_MESSAGE\"\n" + |
||||||
|
"git push --all\n" + |
||||||
|
"git push --tags"); |
||||||
|
|
||||||
|
} |
||||||
|
|
||||||
|
@Test |
||||||
|
public void testDvcDownload() throws Exception{ |
||||||
|
DvcTask dvcTask = initTask(createDownloadParameters()); |
||||||
|
Assert.assertEquals(dvcTask.buildCommand(), |
||||||
|
"which dvc || { echo \"dvc does not exist\"; exit 1; }; DVC_REPO=git@github.com:<YOUR-NAME-OR-ORG>/dvc-data-repository-example\n" + |
||||||
|
"DVC_DATA_PATH=data\n" + |
||||||
|
"DVC_DATA_LOCATION=iris\n" + |
||||||
|
"DVC_VERSION=iris_v2.3.1\n" + |
||||||
|
"dvc get $DVC_REPO $DVC_DATA_LOCATION -o $DVC_DATA_PATH -v --rev $DVC_VERSION"); |
||||||
|
} |
||||||
|
|
||||||
|
@Test |
||||||
|
public void testInitDvc() throws Exception{ |
||||||
|
DvcTask dvcTask = initTask(createInitDvcParameters()); |
||||||
|
Assert.assertEquals(dvcTask.buildCommand(), |
||||||
|
"which dvc || { echo \"dvc does not exist\"; exit 1; }; DVC_REPO=git@github.com:<YOUR-NAME-OR-ORG>/dvc-data-repository-example\n" + |
||||||
|
"git clone $DVC_REPO dvc-repository; cd dvc-repository; pwd\n" + |
||||||
|
"dvc init || exit 1\n" + |
||||||
|
"dvc remote add origin ~/.dvc_test -d\n" + |
||||||
|
"git commit -am \"init dvc project and add remote\"; git push"); |
||||||
|
} |
||||||
|
|
||||||
|
private DvcParameters createUploadParameters() { |
||||||
|
DvcParameters parameters = new DvcParameters(); |
||||||
|
parameters.setDvcTaskType(TaskTypeEnum.UPLOAD); |
||||||
|
parameters.setDvcRepository("git@github.com:<YOUR-NAME-OR-ORG>/dvc-data-repository-example"); |
||||||
|
parameters.setDvcLoadSaveDataPath("/home/<YOUR-NAME-OR-ORG>/test"); |
||||||
|
parameters.setDvcDataLocation("test"); |
||||||
|
parameters.setDvcVersion("iris_v2.3.1"); |
||||||
|
parameters.setDvcMessage("add test iris data"); |
||||||
|
return parameters; |
||||||
|
} |
||||||
|
|
||||||
|
private DvcParameters createDownloadParameters() { |
||||||
|
DvcParameters parameters = new DvcParameters(); |
||||||
|
parameters.setDvcTaskType(TaskTypeEnum.DOWNLOAD); |
||||||
|
parameters.setDvcRepository("git@github.com:<YOUR-NAME-OR-ORG>/dvc-data-repository-example"); |
||||||
|
parameters.setDvcLoadSaveDataPath("data"); |
||||||
|
parameters.setDvcDataLocation("iris"); |
||||||
|
parameters.setDvcVersion("iris_v2.3.1"); |
||||||
|
return parameters; |
||||||
|
} |
||||||
|
|
||||||
|
private DvcParameters createInitDvcParameters() { |
||||||
|
DvcParameters parameters = new DvcParameters(); |
||||||
|
parameters.setDvcTaskType(TaskTypeEnum.INIT); |
||||||
|
parameters.setDvcRepository("git@github.com:<YOUR-NAME-OR-ORG>/dvc-data-repository-example"); |
||||||
|
parameters.setDvcStoreUrl("~/.dvc_test"); |
||||||
|
return parameters; |
||||||
|
} |
||||||
|
} |
||||||
|
After Width: | Height: | Size: 7.5 KiB |
|
After Width: | Height: | Size: 17 KiB |
@ -0,0 +1,171 @@ |
|||||||
|
/* |
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||||
|
* contributor license agreements. See the NOTICE file distributed with |
||||||
|
* this work for additional information regarding copyright ownership. |
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||||
|
* (the "License"); you may not use this file except in compliance with |
||||||
|
* the License. You may obtain a copy of the License at |
||||||
|
* |
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
* |
||||||
|
* Unless required by applicable law or agreed to in writing, software |
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS, |
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||||
|
* See the License for the specific language governing permissions and |
||||||
|
* limitations under the License. |
||||||
|
*/ |
||||||
|
import { useI18n } from 'vue-i18n' |
||||||
|
import type { IJsonItem } from '../types' |
||||||
|
import { watch, ref } from 'vue' |
||||||
|
|
||||||
|
export const DVC_TASK_TYPE = [ |
||||||
|
{ |
||||||
|
label: 'Upload', |
||||||
|
value: 'Upload' |
||||||
|
}, |
||||||
|
{ |
||||||
|
label: 'Download', |
||||||
|
value: 'Download' |
||||||
|
}, |
||||||
|
{ |
||||||
|
label: 'Init DVC', |
||||||
|
value: 'Init DVC' |
||||||
|
} |
||||||
|
] |
||||||
|
|
||||||
|
export function useDvc(model: { [field: string]: any }): IJsonItem[] { |
||||||
|
const { t } = useI18n() |
||||||
|
|
||||||
|
const dvcLoadSaveDataPathSpan = ref(0) |
||||||
|
const dvcDataLocationSpan = ref(0) |
||||||
|
const dvcVersionSpan = ref(0) |
||||||
|
const dvcMessageSpan = ref(0) |
||||||
|
const dvcStoreUrlSpan = ref(0) |
||||||
|
|
||||||
|
const setFlag = () => { |
||||||
|
model.isUpload = model.dvcTaskType === 'Upload' |
||||||
|
model.isDownload = model.dvcTaskType === 'Download' |
||||||
|
model.isInit = model.dvcTaskType === 'Init DVC' |
||||||
|
} |
||||||
|
|
||||||
|
const resetData = () => { |
||||||
|
dvcLoadSaveDataPathSpan.value = model.isUpload || model.isDownload ? 24 : 0 |
||||||
|
dvcDataLocationSpan.value = model.isUpload || model.isDownload ? 24 : 0 |
||||||
|
dvcVersionSpan.value = model.isUpload || model.isDownload ? 24 : 0 |
||||||
|
dvcMessageSpan.value = model.isUpload ? 24 : 0 |
||||||
|
dvcStoreUrlSpan.value = model.isInit ? 24 : 0 |
||||||
|
} |
||||||
|
|
||||||
|
watch( |
||||||
|
() => [model.dvcTaskType], |
||||||
|
() => { |
||||||
|
setFlag() |
||||||
|
resetData() |
||||||
|
} |
||||||
|
) |
||||||
|
setFlag() |
||||||
|
resetData() |
||||||
|
|
||||||
|
return [ |
||||||
|
{ |
||||||
|
type: 'select', |
||||||
|
field: 'dvcTaskType', |
||||||
|
name: t('project.node.dvc_task_type'), |
||||||
|
span: 12, |
||||||
|
options: DVC_TASK_TYPE |
||||||
|
}, |
||||||
|
{ |
||||||
|
type: 'input', |
||||||
|
field: 'dvcRepository', |
||||||
|
name: t('project.node.dvc_repository'), |
||||||
|
span: 24, |
||||||
|
props: { |
||||||
|
placeholder: t('project.node.dvc_repository_tips') |
||||||
|
}, |
||||||
|
validate: { |
||||||
|
trigger: ['input', 'blur'], |
||||||
|
required: true, |
||||||
|
message: t('project.node.dvc_empty_tips') |
||||||
|
} |
||||||
|
}, |
||||||
|
{ |
||||||
|
type: 'input', |
||||||
|
field: 'dvcDataLocation', |
||||||
|
name: t('project.node.dvc_data_location'), |
||||||
|
props: { placeholder: 'dvcDataLocation' }, |
||||||
|
span: dvcDataLocationSpan, |
||||||
|
validate: { |
||||||
|
trigger: ['input', 'blur'], |
||||||
|
required: true, |
||||||
|
validator(validate: any, value: string) { |
||||||
|
if ((model.isUpload || model.isDownload) && !value) { |
||||||
|
return new Error(t('project.node.dvc_empty_tips')) |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
}, |
||||||
|
{ |
||||||
|
type: 'input', |
||||||
|
field: 'dvcLoadSaveDataPath', |
||||||
|
name: t('project.node.dvc_load_save_data_path'), |
||||||
|
span: dvcLoadSaveDataPathSpan, |
||||||
|
validate: { |
||||||
|
trigger: ['input', 'blur'], |
||||||
|
required: true, |
||||||
|
validator(validate: any, value: string) { |
||||||
|
if ((model.isUpload || model.isDownload) && !value) { |
||||||
|
return new Error(t('project.node.dvc_empty_tips')) |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
}, |
||||||
|
{ |
||||||
|
type: 'input', |
||||||
|
field: 'dvcVersion', |
||||||
|
name: t('project.node.dvc_version'), |
||||||
|
span: dvcVersionSpan, |
||||||
|
props: { |
||||||
|
placeholder: t('project.node.dvc_version_tips') |
||||||
|
}, |
||||||
|
validate: { |
||||||
|
trigger: ['input', 'blur'], |
||||||
|
required: true, |
||||||
|
validator(validate: any, value: string) { |
||||||
|
if ((model.isUpload || model.isDownload) && !value) { |
||||||
|
return new Error(t('project.node.dvc_empty_tips')) |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
}, |
||||||
|
{ |
||||||
|
type: 'input', |
||||||
|
field: 'dvcMessage', |
||||||
|
name: t('project.node.dvc_message'), |
||||||
|
span: dvcMessageSpan, |
||||||
|
validate: { |
||||||
|
trigger: ['input', 'blur'], |
||||||
|
required: true, |
||||||
|
validator(validate: any, value: string) { |
||||||
|
if (model.isUpload && !value) { |
||||||
|
return new Error(t('project.node.dvc_empty_tips')) |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
}, |
||||||
|
{ |
||||||
|
type: 'input', |
||||||
|
field: 'dvcStoreUrl', |
||||||
|
name: t('project.node.dvc_store_url'), |
||||||
|
span: dvcStoreUrlSpan, |
||||||
|
validate: { |
||||||
|
trigger: ['input', 'blur'], |
||||||
|
required: true, |
||||||
|
validator(validate: any, value: string) { |
||||||
|
if (!model.isInit && value) { |
||||||
|
return new Error(t('project.node.dvc_empty_tips')) |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
] |
||||||
|
} |
||||||
@ -0,0 +1,82 @@ |
|||||||
|
/* |
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||||
|
* contributor license agreements. See the NOTICE file distributed with |
||||||
|
* this work for additional information regarding copyright ownership. |
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||||
|
* (the "License"); you may not use this file except in compliance with |
||||||
|
* the License. You may obtain a copy of the License at |
||||||
|
* |
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
* |
||||||
|
* Unless required by applicable law or agreed to in writing, software |
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS, |
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||||
|
* See the License for the specific language governing permissions and |
||||||
|
* limitations under the License. |
||||||
|
*/ |
||||||
|
|
||||||
|
import { reactive } from 'vue' |
||||||
|
import * as Fields from '../fields/index' |
||||||
|
import type { IJsonItem, INodeData, ITaskData } from '../types' |
||||||
|
|
||||||
|
export function useDvc({ |
||||||
|
projectCode, |
||||||
|
from = 0, |
||||||
|
readonly, |
||||||
|
data |
||||||
|
}: { |
||||||
|
projectCode: number |
||||||
|
from?: number |
||||||
|
readonly?: boolean |
||||||
|
data?: ITaskData |
||||||
|
}) { |
||||||
|
const model = reactive({ |
||||||
|
name: '', |
||||||
|
taskType: 'MLFLOW', |
||||||
|
flag: 'YES', |
||||||
|
description: '', |
||||||
|
timeoutFlag: false, |
||||||
|
localParams: [], |
||||||
|
environmentCode: null, |
||||||
|
failRetryInterval: 1, |
||||||
|
failRetryTimes: 0, |
||||||
|
workerGroup: 'default', |
||||||
|
delayTime: 0, |
||||||
|
timeout: 30, |
||||||
|
timeoutNotifyStrategy: ['WARN'], |
||||||
|
dvcTaskType: 'Upload', |
||||||
|
} as INodeData) |
||||||
|
|
||||||
|
let extra: IJsonItem[] = [] |
||||||
|
if (from === 1) { |
||||||
|
extra = [ |
||||||
|
Fields.useTaskType(model, readonly), |
||||||
|
Fields.useProcessName({ |
||||||
|
model, |
||||||
|
projectCode, |
||||||
|
isCreate: !data?.id, |
||||||
|
from, |
||||||
|
processName: data?.processName |
||||||
|
}) |
||||||
|
] |
||||||
|
} |
||||||
|
|
||||||
|
return { |
||||||
|
json: [ |
||||||
|
Fields.useName(from), |
||||||
|
...extra, |
||||||
|
Fields.useRunFlag(), |
||||||
|
Fields.useDescription(), |
||||||
|
Fields.useTaskPriority(), |
||||||
|
Fields.useWorkerGroup(), |
||||||
|
Fields.useEnvironmentName(model, !model.id), |
||||||
|
...Fields.useTaskGroup(model, projectCode), |
||||||
|
...Fields.useFailed(), |
||||||
|
Fields.useDelayTime(model), |
||||||
|
...Fields.useTimeoutAlarm(model), |
||||||
|
...Fields.useDvc(model), |
||||||
|
Fields.usePreTasks() |
||||||
|
] as IJsonItem[], |
||||||
|
model |
||||||
|
} |
||||||
|
} |
||||||
Loading…
Reference in new issue