Browse Source
* add data transfer between tasks * add delete DATA_TRANSFER API * convert Result to DeleteDataTransferResponse * add api UT * fix final * fix doc3.2.0-release
JieguangZhou
2 years ago
committed by
GitHub
21 changed files with 1135 additions and 1 deletions
@ -0,0 +1,102 @@
|
||||
# FILE Parameter |
||||
|
||||
Use the file parameter to pass files (or folders, hereinafter referred to as **files**) in the working directory of the upstream task to the downstream task in the same workflow instance. The following scenarios may be used |
||||
|
||||
- In the ETL task, pass the data files processed by multiple upstream tasks to a specific downstream task. |
||||
- In the machine learning scenario, pass the data set file of the upstream data preparation task to the downstream model training task. |
||||
|
||||
## Usage |
||||
|
||||
### Configure file parameter |
||||
|
||||
File parameter configuration method: click the plus sign on the right side of "Custom Parameters" on the task definition page to configure. |
||||
|
||||
### Output file to downstream task |
||||
|
||||
**Four options of custom parameters are:** |
||||
|
||||
- Parameter name: the identifier used when passing tasks, such as `KEY1` and `KEY2` in the figure below |
||||
- Direction: OUT, which means outputting the file to the downstream task |
||||
- Parameter type: FILE, indicating file parameter |
||||
- Parameter value: output file path, such as `data` and `data/test2/text.txt` in the figure below |
||||
|
||||
The configuration in the figure below indicates that the `output` task passes two file data to the downstream task, respectively: |
||||
|
||||
- Pass out the folder `data`, and mark it as `dir-data`. The downstream task can get this folder through `output.dir-data` |
||||
- Pass out the file `data/test2/text.txt`, and mark it as `file-text`. The downstream task can get this folder through `output.file-text` |
||||
|
||||
 |
||||
|
||||
### Get the file from the upstream task |
||||
|
||||
**Four options of custom parameters are:** |
||||
|
||||
- Parameter name: the position where the upstream file is saved after input, such as `input_dir` used in the figure below |
||||
- Direction: IN, which means to get the file from the upstream task |
||||
- Parameter type: FILE, indicating file parameter |
||||
- Parameter value: the identifier of the upstream file, in the format of `taskName.KEY`. For example, `output.dir-data` in the figure below, where `output` is the name of the upstream task, and `dir-data` is the file identifier output by the upstream task |
||||
|
||||
The configuration in the figure below indicates that the task gets the folder identified by `dir-data` from the upstream task `output` and saves it as `input_dir` |
||||
|
||||
 |
||||
|
||||
The configuration in the figure below indicates that the task gets the file identified by `file-text` from the upstream task `output` and saves it as `input.txt` |
||||
|
||||
 |
||||
|
||||
## Other |
||||
|
||||
### Note |
||||
|
||||
- The file transfer between upstream and downstream tasks is based on the resource center as a transfer, and the data is saved in the `DATA_TRANSFER` directory of the resource center. Therefore, **the resource center function must be enabled**, please refer to [Resource Center Configuration Details](../resource/configuration.md) for details, otherwise the file parameter function cannot be used. |
||||
- The file naming rule is `DATA_TRANSFER/DATE/ProcessDefineCode/ProcessDefineVersion_ProcessInstanceID/TaskName_TaskInstanceID_FileName` |
||||
- If the transferred file data is a folder, it will be packaged into a compressed file with a suffix of `.zip` and uploaded. The downstream task will unzip and save it in the corresponding directory after receiving it |
||||
- If you need to delete the file data, you can delete the corresponding folder in the `DATA_TRANSFER` directory of the resource center. If you delete the date subdirectory directly, all the file data under that date will be deleted. You can also use the [Open API interface](../open-api.md) (`resources/data-transfer`) to delete the corresponding file data (delete data N days ago). |
||||
- If there is a task chain task1->task2->tas3, then the downstream task task3 can also get the file data of task1 |
||||
- Support one-to-many transmission and many-to-one transmission |
||||
- If you frequently transfer a large number of files, it is obvious that the system IO performance will be affected by the amount of transferred data |
||||
|
||||
### Example |
||||
|
||||
You can save the following YAML file locally and then execute `pydolphinscheduler yaml -f data-transfer.yaml` to run the Demo. |
||||
|
||||
```yaml |
||||
# Define the workflow |
||||
workflow: |
||||
name: "data-transfer" |
||||
run: true |
||||
|
||||
# Define the tasks under the workflow |
||||
tasks: |
||||
- name: output |
||||
task_type: Shell |
||||
command: | |
||||
mkdir -p data/test1 data/test2 |
||||
echo "test1 message" >> data/test1/text.txt |
||||
echo "test2 message" >> data/test2/text.txt |
||||
tree . |
||||
local_params: |
||||
- { "prop": "dir-data", "direct": "OUT", "type": "FILE", "value": "data" } |
||||
- { "prop": "file-text", "direct": "OUT", "type": "FILE", "value": "data/test2/text.txt" } |
||||
|
||||
- name: input_dir |
||||
task_type: Shell |
||||
deps: [output] |
||||
command: | |
||||
tree . |
||||
cat input_dir/test1/text.txt |
||||
cat input_dir/test2/text.txt |
||||
local_params: |
||||
- { "prop": "input_dir", "direct": "IN", "type": "FILE", "value": "output.dir-data" } |
||||
|
||||
|
||||
- name: input_file |
||||
task_type: Shell |
||||
deps: [output] |
||||
command: | |
||||
tree . |
||||
cat input.txt |
||||
local_params: |
||||
- { "prop": "input.txt", "direct": "IN", "type": "FILE", "value": "output.file-text" } |
||||
``` |
||||
|
||||
@ -0,0 +1,101 @@
|
||||
# 文件参数 |
||||
|
||||
通过配置文件参数，在同一工作流实例中，可以将上游任务工作目录下的文件(或文件夹，下统一以**文件**代替)传递给下游任务。 如以下场景可能使用到 |
||||
|
||||
- 在ETL任务中，将多个上游任务处理好的数据文件一起传递给特定的下游任务。 |
||||
- 在机器学习场景中，将上游数据准备任务的数据集文件传递给下游模型训练任务。 |
||||
|
||||
## 使用方式 |
||||
|
||||
### 配置文件参数 |
||||
|
||||
文件参数配置方式如下：在任务定义页面，点击“自定义参数”右边的加号，即可进行配置。 |
||||
|
||||
### 输出文件给下游任务 |
||||
|
||||
**自定义参数四个选项分别为:** |
||||
|
||||
- 参数名：任务间传递时使用的标识，如下图中使用的`KEY1`和`KEY2` |
||||
- 方向：OUT, 则表示输出文件给下游任务 |
||||
- 参数类型：FILE, 表示文件参数 |
||||
- 参数值：输出的文件路径，如下图中的`data`和`data/test2/text.txt` |
||||
|
||||
下图的配置表示任务`output`向下游任务传递两个文件数据，分别为: |
||||
- 传出文件夹 `data`, 并标记为`dir-data`, 下游任务可以通过`output.dir-data`获取该文件夹 |
||||
- 传出文件 `data/test2/text.txt`, 并标记为`file-text`, 下游任务可以通过`output.file-text`获取该文件夹 |
||||
|
||||
 |
||||
|
||||
### 获取上游任务的文件 |
||||
|
||||
**自定义参数四个选项分别为:** |
||||
|
||||
- 参数名：上游文件输入后保存的位置，如下图中使用的`input_dir` |
||||
- 方向：IN, 则表示从上游任务获取文件 |
||||
- 参数类型：FILE, 表示文件参数 |
||||
- 参数值：上游文件的标识，为 `taskName.KEY` 的格式 如下图中的`output.dir-data`, 其中`output`为上游任务的名称，`dir-data`为上游任务中输出的文件标识 |
||||
|
||||
下图的配置表示任务从上游任务`output`中获取标识为`dir-data`的文件夹，并保存为`input_dir` |
||||
|
||||
 |
||||
|
||||
下图的配置表示任务从上游任务`output`中获取标识为`file-text`的文件，并保存为`input.txt` |
||||
|
||||
 |
||||
|
||||
## 其他 |
||||
|
||||
### 备注 |
||||
|
||||
- 上下游任务间的文件传递基于资源中心作为中转，数据保存在资源中心`DATA_TRANSFER`的目录下, 因此**必须开启资源中心功能**，详情请参考[资源中心配置详情](../resource/configuration.md), 否则无法使用文件参数功能。 |
||||
- 文件命名规则为 `DATA_TRANSFER/日期/工作流Code/工作流版本_工作流实例ID/任务名称_任务实例ID_文件名` |
||||
- 若传输的文件数据为文件夹，则会打包成后缀为`.zip`的压缩文件再上传，下游任务接到后会解压并保存在对应目录 |
||||
- 若需要删除文件数据，可以在资源中心的`DATA_TRANSFER`目录下删除对应文件夹即可, 如直接按照日期子目录删除，会删除该日期下所有的文件数据. 也可以使用`resources/data-transfer`[Open API 接口](../open-api.md)（删除N天前的数据）删除对应文件数据。 |
||||
- 如果存在任务链 task1->task2->tas3, 则最下游任务task3也能获取task1的文件数据 |
||||
- 支持一对多传输以及多对一传输 |
||||
- 如果频繁大量传输文件，毫无疑问会因传输的数据量影响到系统IO性能 |
||||
|
||||
### 样例 |
||||
|
||||
你可以保存以下YAML文件到本地，然后执行`pydolphinscheduler yaml -f data-transfer.yaml`即可运行Demo. |
||||
|
||||
```yaml |
||||
# Define the workflow |
||||
workflow: |
||||
name: "data-transfer" |
||||
run: true |
||||
|
||||
# Define the tasks under the workflow |
||||
tasks: |
||||
- name: output |
||||
task_type: Shell |
||||
command: | |
||||
mkdir -p data/test1 data/test2 |
||||
echo "test1 message" >> data/test1/text.txt |
||||
echo "test2 message" >> data/test2/text.txt |
||||
tree . |
||||
local_params: |
||||
- { "prop": "dir-data", "direct": "OUT", "type": "FILE", "value": "data" } |
||||
- { "prop": "file-text", "direct": "OUT", "type": "FILE", "value": "data/test2/text.txt" } |
||||
|
||||
- name: input_dir |
||||
task_type: Shell |
||||
deps: [output] |
||||
command: | |
||||
tree . |
||||
cat input_dir/test1/text.txt |
||||
cat input_dir/test2/text.txt |
||||
local_params: |
||||
- { "prop": "input_dir", "direct": "IN", "type": "FILE", "value": "output.dir-data" } |
||||
|
||||
|
||||
- name: input_file |
||||
task_type: Shell |
||||
deps: [output] |
||||
command: | |
||||
tree . |
||||
cat input.txt |
||||
local_params: |
||||
- { "prop": "input.txt", "direct": "IN", "type": "FILE", "value": "output.file-text" } |
||||
``` |
||||
|
||||
|
After Width: | Height: | Size: 72 KiB |
|
After Width: | Height: | Size: 53 KiB |
|
After Width: | Height: | Size: 121 KiB |
@ -0,0 +1,33 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.api.dto.resources; |
||||
|
||||
import org.apache.dolphinscheduler.api.utils.Result; |
||||
|
||||
import java.util.List; |
||||
|
||||
import lombok.Data; |
||||
|
||||
@Data |
||||
public class DeleteDataTransferResponse extends Result { |
||||
|
||||
private List<String> successList; |
||||
|
||||
private List<String> failedList; |
||||
|
||||
} |
||||
@ -0,0 +1,202 @@
|
||||
|
||||
Apache License |
||||
Version 2.0, January 2004 |
||||
http://www.apache.org/licenses/ |
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION |
||||
|
||||
1. Definitions. |
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction, |
||||
and distribution as defined by Sections 1 through 9 of this document. |
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by |
||||
the copyright owner that is granting the License. |
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all |
||||
other entities that control, are controlled by, or are under common |
||||
control with that entity. For the purposes of this definition, |
||||
"control" means (i) the power, direct or indirect, to cause the |
||||
direction or management of such entity, whether by contract or |
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the |
||||
outstanding shares, or (iii) beneficial ownership of such entity. |
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity |
||||
exercising permissions granted by this License. |
||||
|
||||
"Source" form shall mean the preferred form for making modifications, |
||||
including but not limited to software source code, documentation |
||||
source, and configuration files. |
||||
|
||||
"Object" form shall mean any form resulting from mechanical |
||||
transformation or translation of a Source form, including but |
||||
not limited to compiled object code, generated documentation, |
||||
and conversions to other media types. |
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or |
||||
Object form, made available under the License, as indicated by a |
||||
copyright notice that is included in or attached to the work |
||||
(an example is provided in the Appendix below). |
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object |
||||
form, that is based on (or derived from) the Work and for which the |
||||
editorial revisions, annotations, elaborations, or other modifications |
||||
represent, as a whole, an original work of authorship. For the purposes |
||||
of this License, Derivative Works shall not include works that remain |
||||
separable from, or merely link (or bind by name) to the interfaces of, |
||||
the Work and Derivative Works thereof. |
||||
|
||||
"Contribution" shall mean any work of authorship, including |
||||
the original version of the Work and any modifications or additions |
||||
to that Work or Derivative Works thereof, that is intentionally |
||||
submitted to Licensor for inclusion in the Work by the copyright owner |
||||
or by an individual or Legal Entity authorized to submit on behalf of |
||||
the copyright owner. For the purposes of this definition, "submitted" |
||||
means any form of electronic, verbal, or written communication sent |
||||
to the Licensor or its representatives, including but not limited to |
||||
communication on electronic mailing lists, source code control systems, |
||||
and issue tracking systems that are managed by, or on behalf of, the |
||||
Licensor for the purpose of discussing and improving the Work, but |
||||
excluding communication that is conspicuously marked or otherwise |
||||
designated in writing by the copyright owner as "Not a Contribution." |
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity |
||||
on behalf of whom a Contribution has been received by Licensor and |
||||
subsequently incorporated within the Work. |
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of |
||||
this License, each Contributor hereby grants to You a perpetual, |
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable |
||||
copyright license to reproduce, prepare Derivative Works of, |
||||
publicly display, publicly perform, sublicense, and distribute the |
||||
Work and such Derivative Works in Source or Object form. |
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of |
||||
this License, each Contributor hereby grants to You a perpetual, |
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable |
||||
(except as stated in this section) patent license to make, have made, |
||||
use, offer to sell, sell, import, and otherwise transfer the Work, |
||||
where such license applies only to those patent claims licensable |
||||
by such Contributor that are necessarily infringed by their |
||||
Contribution(s) alone or by combination of their Contribution(s) |
||||
with the Work to which such Contribution(s) was submitted. If You |
||||
institute patent litigation against any entity (including a |
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work |
||||
or a Contribution incorporated within the Work constitutes direct |
||||
or contributory patent infringement, then any patent licenses |
||||
granted to You under this License for that Work shall terminate |
||||
as of the date such litigation is filed. |
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the |
||||
Work or Derivative Works thereof in any medium, with or without |
||||
modifications, and in Source or Object form, provided that You |
||||
meet the following conditions: |
||||
|
||||
(a) You must give any other recipients of the Work or |
||||
Derivative Works a copy of this License; and |
||||
|
||||
(b) You must cause any modified files to carry prominent notices |
||||
stating that You changed the files; and |
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works |
||||
that You distribute, all copyright, patent, trademark, and |
||||
attribution notices from the Source form of the Work, |
||||
excluding those notices that do not pertain to any part of |
||||
the Derivative Works; and |
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its |
||||
distribution, then any Derivative Works that You distribute must |
||||
include a readable copy of the attribution notices contained |
||||
within such NOTICE file, excluding those notices that do not |
||||
pertain to any part of the Derivative Works, in at least one |
||||
of the following places: within a NOTICE text file distributed |
||||
as part of the Derivative Works; within the Source form or |
||||
documentation, if provided along with the Derivative Works; or, |
||||
within a display generated by the Derivative Works, if and |
||||
wherever such third-party notices normally appear. The contents |
||||
of the NOTICE file are for informational purposes only and |
||||
do not modify the License. You may add Your own attribution |
||||
notices within Derivative Works that You distribute, alongside |
||||
or as an addendum to the NOTICE text from the Work, provided |
||||
that such additional attribution notices cannot be construed |
||||
as modifying the License. |
||||
|
||||
You may add Your own copyright statement to Your modifications and |
||||
may provide additional or different license terms and conditions |
||||
for use, reproduction, or distribution of Your modifications, or |
||||
for any such Derivative Works as a whole, provided Your use, |
||||
reproduction, and distribution of the Work otherwise complies with |
||||
the conditions stated in this License. |
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise, |
||||
any Contribution intentionally submitted for inclusion in the Work |
||||
by You to the Licensor shall be under the terms and conditions of |
||||
this License, without any additional terms or conditions. |
||||
Notwithstanding the above, nothing herein shall supersede or modify |
||||
the terms of any separate license agreement you may have executed |
||||
with Licensor regarding such Contributions. |
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade |
||||
names, trademarks, service marks, or product names of the Licensor, |
||||
except as required for reasonable and customary use in describing the |
||||
origin of the Work and reproducing the content of the NOTICE file. |
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or |
||||
agreed to in writing, Licensor provides the Work (and each |
||||
Contributor provides its Contributions) on an "AS IS" BASIS, |
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or |
||||
implied, including, without limitation, any warranties or conditions |
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A |
||||
PARTICULAR PURPOSE. You are solely responsible for determining the |
||||
appropriateness of using or redistributing the Work and assume any |
||||
risks associated with Your exercise of permissions under this License. |
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory, |
||||
whether in tort (including negligence), contract, or otherwise, |
||||
unless required by applicable law (such as deliberate and grossly |
||||
negligent acts) or agreed to in writing, shall any Contributor be |
||||
liable to You for damages, including any direct, indirect, special, |
||||
incidental, or consequential damages of any character arising as a |
||||
result of this License or out of the use or inability to use the |
||||
Work (including but not limited to damages for loss of goodwill, |
||||
work stoppage, computer failure or malfunction, or any and all |
||||
other commercial damages or losses), even if such Contributor |
||||
has been advised of the possibility of such damages. |
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing |
||||
the Work or Derivative Works thereof, You may choose to offer, |
||||
and charge a fee for, acceptance of support, warranty, indemnity, |
||||
or other liability obligations and/or rights consistent with this |
||||
License. However, in accepting such obligations, You may act only |
||||
on Your own behalf and on Your sole responsibility, not on behalf |
||||
of any other Contributor, and only if You agree to indemnify, |
||||
defend, and hold each Contributor harmless for any liability |
||||
incurred by, or claims asserted against, such Contributor by reason |
||||
of your accepting any such warranty or additional liability. |
||||
|
||||
END OF TERMS AND CONDITIONS |
||||
|
||||
APPENDIX: How to apply the Apache License to your work. |
||||
|
||||
To apply the Apache License to your work, attach the following |
||||
boilerplate notice, with the fields enclosed by brackets "[]" |
||||
replaced with your own identifying information. (Don't include |
||||
the brackets!) The text should be enclosed in the appropriate |
||||
comment syntax for the file format. We also recommend that a |
||||
file or class name and description of purpose be included on the |
||||
same "printed page" as the copyright notice for easier |
||||
identification within third-party archives. |
||||
|
||||
Copyright 2012 ZeroTurnaround LLC. |
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); |
||||
you may not use this file except in compliance with the License. |
||||
You may obtain a copy of the License at |
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0 |
||||
|
||||
Unless required by applicable law or agreed to in writing, software |
||||
distributed under the License is distributed on an "AS IS" BASIS, |
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
See the License for the specific language governing permissions and |
||||
limitations under the License. |
||||
@ -0,0 +1,270 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.server.worker.utils; |
||||
|
||||
import org.apache.dolphinscheduler.common.utils.DateUtils; |
||||
import org.apache.dolphinscheduler.common.utils.JSONUtils; |
||||
import org.apache.dolphinscheduler.plugin.task.api.TaskConstants; |
||||
import org.apache.dolphinscheduler.plugin.task.api.TaskException; |
||||
import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext; |
||||
import org.apache.dolphinscheduler.plugin.task.api.enums.DataType; |
||||
import org.apache.dolphinscheduler.plugin.task.api.enums.Direct; |
||||
import org.apache.dolphinscheduler.plugin.task.api.model.Property; |
||||
import org.apache.dolphinscheduler.service.storage.StorageOperate; |
||||
|
||||
import org.apache.commons.lang3.StringUtils; |
||||
|
||||
import java.io.File; |
||||
import java.io.IOException; |
||||
import java.time.format.DateTimeFormatter; |
||||
import java.util.ArrayList; |
||||
import java.util.List; |
||||
import java.util.Map; |
||||
import java.util.stream.Collectors; |
||||
|
||||
import org.slf4j.Logger; |
||||
import org.slf4j.LoggerFactory; |
||||
import org.zeroturnaround.zip.ZipUtil; |
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode; |
||||
|
||||
public class TaskFilesTransferUtils { |
||||
|
||||
protected final static Logger logger = LoggerFactory |
||||
.getLogger(String.format(TaskConstants.TASK_LOG_LOGGER_NAME_FORMAT, TaskFilesTransferUtils.class)); |
||||
|
||||
// tmp path in local path for transfer
|
||||
final static String DOWNLOAD_TMP = ".DT_TMP"; |
||||
|
||||
// suffix of the package file
|
||||
final static String PACK_SUFFIX = "_ds_pack.zip"; |
||||
|
||||
// root path in resource storage
|
||||
final static String RESOURCE_TAG = "DATA_TRANSFER"; |
||||
|
||||
private TaskFilesTransferUtils() { |
||||
throw new IllegalStateException("Utility class"); |
||||
} |
||||
|
||||
/** |
||||
* upload output files to resource storage |
||||
* |
||||
* @param taskExecutionContext is the context of task |
||||
* @param storageOperate is the storage operate |
||||
* @throws TaskException TaskException |
||||
*/ |
||||
public static void uploadOutputFiles(TaskExecutionContext taskExecutionContext, |
||||
StorageOperate storageOperate) throws TaskException { |
||||
List<Property> varPools = getVarPools(taskExecutionContext); |
||||
// get map of varPools for quick search
|
||||
Map<String, Property> varPoolsMap = varPools.stream().collect(Collectors.toMap(Property::getProp, x -> x)); |
||||
|
||||
// get OUTPUT FILE parameters
|
||||
List<Property> localParamsProperty = getFileLocalParams(taskExecutionContext, Direct.OUT); |
||||
|
||||
if (localParamsProperty.isEmpty()) { |
||||
return; |
||||
} |
||||
|
||||
logger.info("Upload output files ..."); |
||||
for (Property property : localParamsProperty) { |
||||
// get local file path
|
||||
String srcPath = |
||||
packIfDir(String.format("%s/%s", taskExecutionContext.getExecutePath(), property.getValue())); |
||||
// get remote file path
|
||||
String resourcePath = getResourcePath(taskExecutionContext, new File(srcPath).getName()); |
||||
try { |
||||
// upload file to storage
|
||||
String resourceWholePath = |
||||
storageOperate.getResourceFileName(taskExecutionContext.getTenantCode(), resourcePath); |
||||
logger.info("{} --- Local:{} to Remote:{}", property, srcPath, resourceWholePath); |
||||
storageOperate.upload(taskExecutionContext.getTenantCode(), srcPath, resourceWholePath, false, true); |
||||
} catch (IOException ex) { |
||||
throw new TaskException("Upload file to storage error", ex); |
||||
} |
||||
|
||||
// update varPool
|
||||
Property oriProperty; |
||||
// if the property is not in varPool, add it
|
||||
if (varPoolsMap.containsKey(property.getProp())) { |
||||
oriProperty = varPoolsMap.get(property.getProp()); |
||||
} else { |
||||
oriProperty = new Property(property.getProp(), Direct.OUT, DataType.FILE, property.getValue()); |
||||
varPools.add(oriProperty); |
||||
} |
||||
oriProperty.setProp(String.format("%s.%s", taskExecutionContext.getTaskName(), oriProperty.getProp())); |
||||
oriProperty.setValue(resourcePath); |
||||
} |
||||
taskExecutionContext.setVarPool(JSONUtils.toJsonString(varPools)); |
||||
} |
||||
|
||||
/** |
||||
* download upstream files from storage |
||||
* only download files which are defined in the task parameters |
||||
* |
||||
* @param taskExecutionContext is the context of task |
||||
* @param storageOperate is the storage operate |
||||
* @throws TaskException task exception |
||||
*/ |
||||
public static void downloadUpstreamFiles(TaskExecutionContext taskExecutionContext, StorageOperate storageOperate) { |
||||
List<Property> varPools = getVarPools(taskExecutionContext); |
||||
// get map of varPools for quick search
|
||||
Map<String, Property> varPoolsMap = varPools.stream().collect(Collectors.toMap(Property::getProp, x -> x)); |
||||
|
||||
// get "IN FILE" parameters
|
||||
List<Property> localParamsProperty = getFileLocalParams(taskExecutionContext, Direct.IN); |
||||
|
||||
if (localParamsProperty.isEmpty()) { |
||||
return; |
||||
} |
||||
|
||||
String executePath = taskExecutionContext.getExecutePath(); |
||||
// data path to download packaged data
|
||||
String downloadTmpPath = String.format("%s/%s", executePath, DOWNLOAD_TMP); |
||||
|
||||
logger.info("Download upstream files..."); |
||||
for (Property property : localParamsProperty) { |
||||
Property inVarPool = varPoolsMap.get(property.getValue()); |
||||
if (inVarPool == null) { |
||||
logger.error("{} not in {}", property.getValue(), varPoolsMap.keySet()); |
||||
throw new TaskException(String.format("Can not find upstream file using %s, please check the key", |
||||
property.getValue())); |
||||
} |
||||
|
||||
String resourcePath = inVarPool.getValue(); |
||||
String targetPath = String.format("%s/%s", executePath, property.getProp()); |
||||
|
||||
String downloadPath; |
||||
// If the data is packaged, download it to a special directory (DOWNLOAD_TMP) and unpack it to the
|
||||
// targetPath
|
||||
boolean isPack = resourcePath.endsWith(PACK_SUFFIX); |
||||
if (isPack) { |
||||
downloadPath = String.format("%s/%s", downloadTmpPath, new File(resourcePath).getName()); |
||||
} else { |
||||
downloadPath = targetPath; |
||||
} |
||||
|
||||
try { |
||||
String resourceWholePath = |
||||
storageOperate.getResourceFileName(taskExecutionContext.getTenantCode(), resourcePath); |
||||
logger.info("{} --- Remote:{} to Local:{}", property, resourceWholePath, downloadPath); |
||||
storageOperate.download(taskExecutionContext.getTenantCode(), resourceWholePath, downloadPath, false, |
||||
true); |
||||
} catch (IOException ex) { |
||||
throw new TaskException("Download file from storage error", ex); |
||||
} |
||||
|
||||
// unpack if the data is packaged
|
||||
if (isPack) { |
||||
File downloadFile = new File(downloadPath); |
||||
logger.info("Unpack {} to {}", downloadPath, targetPath); |
||||
ZipUtil.unpack(downloadFile, new File(targetPath)); |
||||
} |
||||
} |
||||
|
||||
// delete DownloadTmp Folder if DownloadTmpPath exists
|
||||
try { |
||||
org.apache.commons.io.FileUtils.deleteDirectory(new File(downloadTmpPath)); |
||||
} catch (IOException e) { |
||||
logger.error("Delete DownloadTmpPath {} failed, this will not affect the task status", downloadTmpPath, e); |
||||
} |
||||
} |
||||
|
||||
/** |
||||
* get local parameters property which type is FILE and direction is equal to direct |
||||
* |
||||
* @param taskExecutionContext is the context of task |
||||
* @param direct may be Direct.IN or Direct.OUT. |
||||
* @return List<Property> |
||||
*/ |
||||
public static List<Property> getFileLocalParams(TaskExecutionContext taskExecutionContext, Direct direct) { |
||||
List<Property> localParamsProperty = new ArrayList<>(); |
||||
JsonNode taskParams = JSONUtils.parseObject(taskExecutionContext.getTaskParams()); |
||||
for (JsonNode localParam : taskParams.get("localParams")) { |
||||
Property property = JSONUtils.parseObject(localParam.toString(), Property.class); |
||||
|
||||
if (property.getDirect().equals(direct) && property.getType().equals(DataType.FILE)) { |
||||
localParamsProperty.add(property); |
||||
} |
||||
} |
||||
return localParamsProperty; |
||||
} |
||||
|
||||
/** |
||||
* get Resource path for manage files in storage |
||||
* |
||||
* @param taskExecutionContext is the context of task |
||||
* @param fileName is the file name |
||||
* @return resource path, RESOURCE_TAG/DATE/ProcessDefineCode/ProcessDefineVersion_ProcessInstanceID/TaskName_TaskInstanceID_FileName |
||||
*/ |
||||
public static String getResourcePath(TaskExecutionContext taskExecutionContext, String fileName) { |
||||
String date = |
||||
DateUtils.formatTimeStamp(taskExecutionContext.getEndTime(), DateTimeFormatter.ofPattern("yyyyMMdd")); |
||||
// get resource Folder: RESOURCE_TAG/DATE/ProcessDefineCode/ProcessDefineVersion_ProcessInstanceID
|
||||
String resourceFolder = String.format("%s/%s/%d/%d_%d", RESOURCE_TAG, date, |
||||
taskExecutionContext.getProcessDefineCode(), taskExecutionContext.getProcessDefineVersion(), |
||||
taskExecutionContext.getProcessInstanceId()); |
||||
// get resource fileL: resourceFolder/TaskName_TaskInstanceID_FileName
|
||||
return String.format("%s/%s_%s_%s", resourceFolder, taskExecutionContext.getTaskName().replace(" ", "_"), |
||||
taskExecutionContext.getTaskInstanceId(), fileName); |
||||
} |
||||
|
||||
/** |
||||
* get varPool from taskExecutionContext |
||||
* |
||||
* @param taskExecutionContext is the context of task |
||||
* @return List<Property> |
||||
*/ |
||||
public static List<Property> getVarPools(TaskExecutionContext taskExecutionContext) { |
||||
List<Property> varPools = new ArrayList<>(); |
||||
|
||||
// get varPool
|
||||
String varPoolString = taskExecutionContext.getVarPool(); |
||||
if (StringUtils.isEmpty(varPoolString)) { |
||||
return varPools; |
||||
} |
||||
// parse varPool
|
||||
for (JsonNode varPoolData : JSONUtils.parseArray(varPoolString)) { |
||||
Property property = JSONUtils.parseObject(varPoolData.toString(), Property.class); |
||||
varPools.add(property); |
||||
} |
||||
return varPools; |
||||
} |
||||
|
||||
/** |
||||
* If the path is a directory, pack it and return the path of the package
|
||||
* |
||||
* @param path is the input path, may be a file or a directory |
||||
* @return new path |
||||
*/ |
||||
public static String packIfDir(String path) throws TaskException { |
||||
File file = new File(path); |
||||
if (!file.exists()) { |
||||
throw new TaskException(String.format("%s dose not exists", path)); |
||||
} |
||||
String newPath; |
||||
if (file.isDirectory()) { |
||||
newPath = file.getPath() + PACK_SUFFIX; |
||||
logger.info("Pack {} to {}", path, newPath); |
||||
ZipUtil.pack(file, new File(newPath)); |
||||
} else { |
||||
newPath = path; |
||||
} |
||||
return newPath; |
||||
} |
||||
} |
||||
@ -0,0 +1,255 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.server.worker.utils; |
||||
|
||||
import org.apache.dolphinscheduler.common.utils.DateUtils; |
||||
import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext; |
||||
import org.apache.dolphinscheduler.plugin.task.api.enums.DataType; |
||||
import org.apache.dolphinscheduler.plugin.task.api.enums.Direct; |
||||
import org.apache.dolphinscheduler.plugin.task.api.model.Property; |
||||
import org.apache.dolphinscheduler.service.storage.StorageOperate; |
||||
|
||||
import org.apache.curator.shaded.com.google.common.io.Files; |
||||
|
||||
import java.io.File; |
||||
import java.io.IOException; |
||||
import java.time.format.DateTimeFormatter; |
||||
import java.util.List; |
||||
|
||||
import org.junit.jupiter.api.Assertions; |
||||
import org.junit.jupiter.api.BeforeEach; |
||||
import org.junit.jupiter.api.Test; |
||||
import org.mockito.Mockito; |
||||
import org.zeroturnaround.zip.ZipUtil; |
||||
|
||||
public class TaskFilesTransferUtilsTest { |
||||
|
||||
private final long processDefineCode = 123; |
||||
private final int processDefineVersion = 456; |
||||
private final int processInstanceId = 678; |
||||
private final int taskInstanceId = 789; |
||||
private final String taskName = "test"; |
||||
|
||||
private final String tenantCode = "ubuntu"; |
||||
|
||||
private long endTime; |
||||
|
||||
private String exceptTemplate; |
||||
|
||||
@BeforeEach |
||||
void init() { |
||||
endTime = System.currentTimeMillis(); |
||||
String date = DateUtils.formatTimeStamp(endTime, DateTimeFormatter.ofPattern("yyyyMMdd")); |
||||
exceptTemplate = String.format("%s/%s/%d/%d_%d/%s_%d", |
||||
TaskFilesTransferUtils.RESOURCE_TAG, |
||||
date, |
||||
processDefineCode, |
||||
processDefineVersion, |
||||
processInstanceId, |
||||
taskName, |
||||
taskInstanceId); |
||||
} |
||||
|
||||
@Test |
||||
void testUploadOutputFiles() throws IOException { |
||||
File executePath = Files.createTempDir(); |
||||
File folderPath = new File(executePath, "data"); |
||||
File file = new File(folderPath.getPath() + "/test.txt"); |
||||
if (!(folderPath.mkdirs() && file.createNewFile())) { |
||||
return; |
||||
} |
||||
String varPool = "[" + |
||||
String.format("{\"prop\":\"folder\",\"direct\":\"OUT\",\"type\":\"FILE\",\"value\":\"%s\"},", |
||||
folderPath.getName()) |
||||
+ |
||||
String.format(" {\"prop\":\"file\",\"direct\":\"OUT\",\"type\":\"FILE\",\"value\":\"%s/%s\"},", |
||||
folderPath.getName(), file.getName()) |
||||
+ |
||||
"{\"prop\":\"a\",\"direct\":\"OUT\",\"type\":\"VARCHAR\",\"value\":\"a\"}," + |
||||
"{\"prop\":\"b\",\"direct\":\"OUT\",\"type\":\"VARCHAR\",\"value\":\"b\"}" + |
||||
"]"; |
||||
String taskParams = String.format("{\"localParams\": %s}", varPool); |
||||
TaskExecutionContext taskExecutionContext = TaskExecutionContext.builder() |
||||
.varPool(varPool) |
||||
.taskParams(taskParams) |
||||
.processInstanceId(processInstanceId) |
||||
.processDefineVersion(processDefineVersion) |
||||
.processDefineCode(processDefineCode) |
||||
.taskInstanceId(taskInstanceId) |
||||
.taskName(taskName) |
||||
.tenantCode(tenantCode) |
||||
.executePath(executePath.toString()) |
||||
.endTime(endTime) |
||||
.build(); |
||||
|
||||
List<Property> oriProperties = TaskFilesTransferUtils.getVarPools(taskExecutionContext); |
||||
|
||||
StorageOperate storageOperate = Mockito.mock(StorageOperate.class); |
||||
TaskFilesTransferUtils.uploadOutputFiles(taskExecutionContext, storageOperate); |
||||
System.out.println(taskExecutionContext.getVarPool()); |
||||
|
||||
String exceptFolder = |
||||
String.format("%s_%s", exceptTemplate, folderPath.getName() + TaskFilesTransferUtils.PACK_SUFFIX); |
||||
String exceptFile = String.format("%s_%s", exceptTemplate, file.getName()); |
||||
|
||||
List<Property> properties = TaskFilesTransferUtils.getVarPools(taskExecutionContext); |
||||
Assertions.assertEquals(4, properties.size()); |
||||
|
||||
Assertions.assertEquals(String.format("%s.%s", taskName, "folder"), properties.get(0).getProp()); |
||||
Assertions.assertEquals(exceptFolder, properties.get(0).getValue()); |
||||
|
||||
Assertions.assertEquals(String.format("%s.%s", taskName, "file"), properties.get(1).getProp()); |
||||
Assertions.assertEquals(exceptFile, properties.get(1).getValue()); |
||||
|
||||
Assertions.assertEquals(oriProperties.get(2).getProp(), properties.get(2).getProp()); |
||||
Assertions.assertEquals(oriProperties.get(3).getValue(), properties.get(3).getValue()); |
||||
|
||||
} |
||||
|
||||
@Test |
||||
void testDownloadUpstreamFiles() { |
||||
File executePath = Files.createTempDir(); |
||||
String folderPath = exceptTemplate + "_folder" + TaskFilesTransferUtils.PACK_SUFFIX; |
||||
String filePath = exceptTemplate + "_file"; |
||||
String varPool = "[" + |
||||
String.format( |
||||
"{\"prop\":\"task1.folder\",\"direct\":\"IN\",\"type\":\"FILE\",\"value\":\"%s\"},", folderPath) |
||||
+ |
||||
String.format(" {\"prop\":\"task2.file\",\"direct\":\"IN\",\"type\":\"FILE\",\"value\":\"%s\"},", |
||||
filePath) |
||||
+ |
||||
"{\"prop\":\"a\",\"direct\":\"OUT\",\"type\":\"VARCHAR\",\"value\":\"a\"}," + |
||||
"{\"prop\":\"b\",\"direct\":\"OUT\",\"type\":\"VARCHAR\",\"value\":\"b\"}" + |
||||
"]"; |
||||
String varPoolParams = "[" + |
||||
"{\"prop\":\"folder\",\"direct\":\"IN\",\"type\":\"FILE\",\"value\":\"task1.folder\"}," + |
||||
" {\"prop\":\"file\",\"direct\":\"IN\",\"type\":\"FILE\",\"value\":\"task2.file\"}" + |
||||
"]"; |
||||
String taskParams = String.format("{\"localParams\": %s}", varPoolParams); |
||||
TaskExecutionContext taskExecutionContext = TaskExecutionContext.builder() |
||||
.varPool(varPool) |
||||
.taskParams(taskParams) |
||||
.processInstanceId(processInstanceId) |
||||
.processDefineVersion(processDefineVersion) |
||||
.processDefineCode(processDefineCode) |
||||
.taskInstanceId(taskInstanceId) |
||||
.taskName(taskName) |
||||
.tenantCode(tenantCode) |
||||
.executePath(executePath.toString()) |
||||
.endTime(endTime) |
||||
.build(); |
||||
|
||||
StorageOperate storageOperate = Mockito.mock(StorageOperate.class); |
||||
Mockito.mockStatic(ZipUtil.class); |
||||
Assertions.assertDoesNotThrow( |
||||
() -> TaskFilesTransferUtils.downloadUpstreamFiles(taskExecutionContext, storageOperate)); |
||||
} |
||||
|
||||
@Test |
||||
void testGetFileLocalParams() { |
||||
String taskParams = "{\"localParams\":[" + |
||||
"{\"prop\":\"inputFile\",\"direct\":\"IN\",\"type\":\"FILE\",\"value\":\"task1.data\"}," + |
||||
"{\"prop\":\"outputFile\",\"direct\":\"OUT\",\"type\":\"FILE\",\"value\":\"data\"}," + |
||||
"{\"prop\":\"a\",\"direct\":\"IN\",\"type\":\"VARCHAR\",\"value\":\"a\"}," + |
||||
"{\"prop\":\"b\",\"direct\":\"OUT\",\"type\":\"VARCHAR\",\"value\":\"b\"}" + |
||||
"]}"; |
||||
TaskExecutionContext taskExecutionContext = Mockito.mock(TaskExecutionContext.class); |
||||
Mockito.when(taskExecutionContext.getTaskParams()).thenReturn(taskParams); |
||||
|
||||
List<Property> fileLocalParamsIn = TaskFilesTransferUtils.getFileLocalParams(taskExecutionContext, Direct.IN); |
||||
Assertions.assertEquals(1, fileLocalParamsIn.size()); |
||||
Assertions.assertEquals("inputFile", fileLocalParamsIn.get(0).getProp()); |
||||
Assertions.assertEquals("task1.data", fileLocalParamsIn.get(0).getValue()); |
||||
|
||||
List<Property> fileLocalParamsOut = TaskFilesTransferUtils.getFileLocalParams(taskExecutionContext, Direct.OUT); |
||||
Assertions.assertEquals(1, fileLocalParamsOut.size()); |
||||
Assertions.assertEquals("outputFile", fileLocalParamsOut.get(0).getProp()); |
||||
Assertions.assertEquals("data", fileLocalParamsOut.get(0).getValue()); |
||||
|
||||
} |
||||
|
||||
@Test |
||||
void testGetResourcePath() { |
||||
String fileName = "test.txt"; |
||||
TaskExecutionContext taskExecutionContext = Mockito.mock(TaskExecutionContext.class); |
||||
|
||||
Mockito.when(taskExecutionContext.getEndTime()).thenReturn(endTime); |
||||
|
||||
Mockito.when(taskExecutionContext.getProcessDefineCode()).thenReturn(processDefineCode); |
||||
Mockito.when(taskExecutionContext.getProcessDefineVersion()).thenReturn(processDefineVersion); |
||||
Mockito.when(taskExecutionContext.getProcessInstanceId()).thenReturn(processInstanceId); |
||||
Mockito.when(taskExecutionContext.getTaskInstanceId()).thenReturn(taskInstanceId); |
||||
Mockito.when(taskExecutionContext.getTaskName()).thenReturn(taskName); |
||||
|
||||
String except = String.format("%s_%s", exceptTemplate, fileName); |
||||
Assertions.assertEquals(except, TaskFilesTransferUtils.getResourcePath(taskExecutionContext, fileName)); |
||||
|
||||
} |
||||
|
||||
@Test |
||||
void testGetVarPools() { |
||||
String varPoolsString = "[" + |
||||
"{\"prop\":\"input\",\"direct\":\"IN\",\"type\":\"FILE\",\"value\":\"task1.output\"}" + |
||||
",{\"prop\":\"a\",\"direct\":\"IN\",\"type\":\"VARCHAR\",\"value\":\"${a}\"}" + |
||||
"]"; |
||||
TaskExecutionContext taskExecutionContext = Mockito.mock(TaskExecutionContext.class); |
||||
Mockito.when(taskExecutionContext.getVarPool()).thenReturn(varPoolsString); |
||||
|
||||
List<Property> varPools = TaskFilesTransferUtils.getVarPools(taskExecutionContext); |
||||
Assertions.assertEquals(2, varPools.size()); |
||||
|
||||
Property varPool0 = varPools.get(0); |
||||
Assertions.assertEquals("input", varPool0.getProp()); |
||||
Assertions.assertEquals(Direct.IN, varPool0.getDirect()); |
||||
Assertions.assertEquals(DataType.FILE, varPool0.getType()); |
||||
Assertions.assertEquals("task1.output", varPool0.getValue()); |
||||
|
||||
Property varPool1 = varPools.get(1); |
||||
Assertions.assertEquals("a", varPool1.getProp()); |
||||
Assertions.assertEquals(Direct.IN, varPool1.getDirect()); |
||||
Assertions.assertEquals(DataType.VARCHAR, varPool1.getType()); |
||||
Assertions.assertEquals("${a}", varPool1.getValue()); |
||||
|
||||
Mockito.when(taskExecutionContext.getVarPool()).thenReturn("[]"); |
||||
List<Property> varPoolsEmpty = TaskFilesTransferUtils.getVarPools(taskExecutionContext); |
||||
Assertions.assertEquals(0, varPoolsEmpty.size()); |
||||
|
||||
Mockito.when(taskExecutionContext.getVarPool()).thenReturn(null); |
||||
List<Property> varPoolsNull = TaskFilesTransferUtils.getVarPools(taskExecutionContext); |
||||
Assertions.assertEquals(0, varPoolsNull.size()); |
||||
|
||||
} |
||||
|
||||
@Test |
||||
void testPackIfDir() throws Exception { |
||||
File folderPath = Files.createTempDir(); |
||||
File file1 = new File(folderPath.getPath() + "/test.txt"); |
||||
File file2 = new File(folderPath.getPath() + "/test.zip"); |
||||
boolean isSuccess1 = file1.createNewFile(); |
||||
boolean isSuccess2 = file2.createNewFile(); |
||||
|
||||
Assertions.assertTrue(isSuccess1); |
||||
Assertions.assertTrue(isSuccess2); |
||||
|
||||
Assertions.assertEquals(file1.getPath(), TaskFilesTransferUtils.packIfDir(file1.getPath())); |
||||
Assertions.assertEquals(file2.getPath(), TaskFilesTransferUtils.packIfDir(file2.getPath())); |
||||
|
||||
String expectFolderPackPath = folderPath.getPath() + TaskFilesTransferUtils.PACK_SUFFIX; |
||||
Assertions.assertEquals(expectFolderPackPath, TaskFilesTransferUtils.packIfDir(folderPath.getPath())); |
||||
} |
||||
} |
||||
Loading…
Reference in new issue