[Feature][task]add azure datafactory task type (#13417)

* init data factory

* add task-type-config.yaml

* add data factory ui

* submit doc

* submit license

---------

Co-authored-by: devosend <devosend@gmail.com>

docs/docs/en/guide/task/datafactory.md

@@ -0,0 +1,70 @@
+# Azure DataFactory Node
+
+## Overview
+
+In the world of big data, raw, unorganized data is often stored in relational, non-relational, and other storage systems.[Azure DataFactory](https://learn.microsoft.com/en-us/azure/data-factory/introduction) is a managed cloud service that's built for these complex hybrid extract-transform-load (ETL), extract-load-transform (ELT), and data integration projects.
+
+DolphinScheduler DataFactory functions:
+
+- Create Azure DataFactory task enable DolphinScheduler to schedule DataFactory pipelines, keep query pipeline execution status until finish the execution.
+
+## Prerequisite
+
+- **ResourceGroup**: Azure owns a resource group
+- **DataFactory**: Azure owns a data factory under the resource group
+- **Pipeline**: Azure owns pipelines under the assigned resource group and data factory
+- **Application**: Azure owns an application has permission to visit data factory, enables using SDK to call data factory functions
+- **ApplicationClientSecret**: To apply for client secret under Application `Certificates & secrets`
+- **AZURE-CLI**: Install Azure auth application AZURE-CLI on machine, please refer to [Install the Azure CLI on Linux](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux)
+
+### Application Permission Setting
+
+First, visit the `Subcription` page and choose `Access control (IAM)`, then click `Add role assignment` to the authorization page.
+![Subcription-IAM](../../../../img/tasks/demo/datafactory_auth1.png)
+After that, select `Contributor` role which satisfy functions calls in data factory. Then click `Members` page, and click `Select members`.
+Search application name or application `Object ID` to assign `Contributor` role to application.
+![Subcription-Role](../../../../img/tasks/demo/datafactory_auth2.png)
+
+## Configurations
+
+Config Azure configurations, modify `azure` related configurations in `common.properties`
+- **resource.azure.client.id**: Azure Application Application (client) ID
+- **resource.azure.client.secret**: Azure Application client secret under `Certificates & secrets`
+- **resource.azure.subId**: Data factory subscription ID
+- **resource.azure.tenant.id**: Azure Active Directory tenant ID
+
+```yaml
+# The Azure client ID (Azure Application (client) ID)
+resource.azure.client.id=minioadmin
+# The Azure client secret in the Azure application
+resource.azure.client.secret=minioadmin
+# The Azure data factory subscription ID
+resource.azure.subId=minioadmin
+# The Azure tenant ID in the Azure Active Directory
+resource.azure.tenant.id=minioadmin
+
+```
+
+## Create Task
+
+- Click `Project -> Management-Project -> Name-Workflow Definition`, and click the "Create Workflow" button to enter the
+  DAG editing page.
+- Drag from the toolbar <img src="../../../../img/tasks/icons/datafactory.png" width="15"/> task node to canvas.
+
+## Task Example
+
+[//]: # (TODO: use the commented anchor below once our website template supports this syntax)
+[//]: # (- Please refer to [DolphinScheduler Task Parameters Appendix]&#40;appendix.md#default-task-parameters&#41; `Default Task Parameters` section for default parameters.)
+
+- Please refer to [DolphinScheduler Task Parameters Appendix](appendix.md) `Default Task Parameters` section for default parameters.
+
+Here are some specific parameters for the DataFactory plugin:
+
+- **factoryName**: data factory name
+- **resourceGroupName**: resource group name of the data factory
+- **pipelineName**: pipeline name under the resource group and data factory
+
+The following shows the task plugin example:
+
+![data-factory](../../../../img/tasks/demo/datafactory.png)
+

docs/docs/zh/guide/task/datafactory.md

@@ -0,0 +1,68 @@
+# Azure DataFactory 节点
+
+## 综述
+
+在大数据环境中，原始、散乱的数据通常存储在关系、非关系和其他存储系统中。[Azure DataFactory](https://learn.microsoft.com/en-us/azure/data-factory/introduction) 是为这些复杂的混合提取-转换-加载 (ETL)、提取-加载-转换 (ELT) 和数据集成项目而构建的托管云服务。
+
+DolphinScheduler DataFactory 组件的功能:
+
+- 创建 Azure DataFactory 任务可以调度DataFactory中的pipeline，持续获取执行状态，直至任务执行完成。
+
+## 前置条件
+
+- **资源组**: 拥有一个资源组
+- **数据工厂**: 拥有一个资源组下的数据工厂
+- **Pipeline**: 拥有前两者对应的pipeline
+- **应用**: 拥有一个有数据工厂访问权限的应用，才能赋予SDK调用数据工厂
+- **应用客户端密钥**: 在应用的`Certificates & secrets`申请客户端密钥
+- **AZURE-CLI**: 在机器上安装了AZURE的鉴权应用AZURE-CLI，请参考[在LINUX上安装AZURE-CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux)
+
+### 应用权限设置
+
+首先打开当前`Subcription`页面，点击`Access control (IAM)`，再点击`Add role assignment`进入授权页面。
+![Subcription-IAM](../../../../img/tasks/demo/datafactory_auth1.png)
+首先选择`Contributor`角色足够满足调用数据工厂。然后选择`Members`页面，再选择`Select members`，检索APP名称或APP的`Object ID`并添加，从给指定APP添加权限.
+![Subcription-Role](../../../../img/tasks/demo/datafactory_auth2.png)
+
+## 环境配置
+
+需要进行Azure的一些配置，修改`common.properties`中的`azure`相关配置信息
+- **resource.azure.client.id**: Azure Application应用的Application (client) ID
+- **resource.azure.client.secret**: Azure Application应用的`Certificates & secrets`下的客户端密钥
+- **resource.azure.subId**: 数据工厂的subscription ID
+- **resource.azure.tenant.id**: Azure Active Directory下的tenant ID
+
+```yaml
+# The Azure client ID (Azure Application (client) ID)
+resource.azure.client.id=minioadmin
+# The Azure client secret in the Azure application
+resource.azure.client.secret=minioadmin
+# The Azure data factory subscription ID
+resource.azure.subId=minioadmin
+# The Azure tenant ID in the Azure Active Directory
+resource.azure.tenant.id=minioadmin
+
+```
+
+## 创建任务
+
+- 点击项目管理-项目名称-工作流定义，点击“创建工作流”按钮，进入 DAG 编辑页面；
+- 拖动工具栏的 <img src="../../../../img/tasks/icons/datafactory.png" width="15"/> 任务节点到画板中。
+
+## 任务样例
+
+[//]: # (TODO: use the commented anchor below once our website template supports this syntax)
+[//]: # (- 默认参数说明请参考[DolphinScheduler任务参数附录]&#40;appendix.md#默认任务参数&#41;`默认任务参数`一栏。)
+
+- 默认参数说明请参考[DolphinScheduler任务参数附录](appendix.md)`默认任务参数`一栏。
+
+### 独有参数
+
+- **factoryName**: 数据工厂名称
+- **resourceGroupName**: 与数据工厂相关的资源组名称
+- **pipelineName**: 选取数据工厂以及资源组下的对应pipeline名称
+
+组件图示如下：
+
+![data-factory](../../../../img/tasks/demo/datafactory.png)
+

dolphinscheduler-api/pom.xml

@@ -198,6 +198,14 @@
             <groupId>org.springdoc</groupId>
             <artifactId>springdoc-openapi-ui</artifactId>
         </dependency>
+        <dependency>
+            <groupId>com.azure.resourcemanager</groupId>
+            <artifactId>azure-resourcemanager</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>com.azure.resourcemanager</groupId>
+            <artifactId>azure-resourcemanager-datafactory</artifactId>
+        </dependency>
     </dependencies>
     <build>
         <testResources>

dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/CloudController.java

@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.api.controller;
+
+import static org.apache.dolphinscheduler.api.enums.Status.LIST_AZURE_DATA_FACTORY_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.LIST_AZURE_DATA_FACTORY_PIPELINE_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.LIST_AZURE_RESOURCE_GROUP_ERROR;
+
+import org.apache.dolphinscheduler.api.aspect.AccessLogAnnotation;
+import org.apache.dolphinscheduler.api.enums.Status;
+import org.apache.dolphinscheduler.api.exceptions.ApiException;
+import org.apache.dolphinscheduler.api.service.CloudService;
+import org.apache.dolphinscheduler.api.utils.Result;
+import org.apache.dolphinscheduler.common.constants.Constants;
+import org.apache.dolphinscheduler.dao.entity.User;
+
+import java.util.List;
+
+import javax.annotation.Resource;
+
+import org.springframework.http.HttpStatus;
+import org.springframework.web.bind.annotation.GetMapping;
+import org.springframework.web.bind.annotation.RequestAttribute;
+import org.springframework.web.bind.annotation.RequestMapping;
+import org.springframework.web.bind.annotation.RequestParam;
+import org.springframework.web.bind.annotation.ResponseStatus;
+import org.springframework.web.bind.annotation.RestController;
+
+import io.swagger.v3.oas.annotations.Operation;
+import io.swagger.v3.oas.annotations.Parameter;
+import io.swagger.v3.oas.annotations.tags.Tag;
+
+/**
+ * cloud controller
+ */
+@Tag(name = "CLOUD_TAG")
+@RestController
+@RequestMapping("/cloud")
+public class CloudController extends BaseController {
+
+    @Resource
+    private CloudService cloudService;
+
+    /**
+     * get datafactory list
+     *
+     * @param loginUser login user
+     * @return datafactory name list
+     */
+    @Operation(summary = "listDataFactory", description = "LIST_DATA_FACTORY")
+    @GetMapping(value = "/azure/datafactory/factories")
+    @ResponseStatus(HttpStatus.OK)
+    @ApiException(LIST_AZURE_DATA_FACTORY_ERROR)
+    @AccessLogAnnotation(ignoreRequestArgs = "loginUser")
+    public Result listDataFactory(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser) {
+        List<String> factoryNames = cloudService.listDataFactory(loginUser);
+        return success(Status.SUCCESS.getMsg(), factoryNames);
+    }
+
+    /**
+     * get resourceGroup list
+     *
+     * @param loginUser login user
+     * @return resourceGroup list
+     */
+    @Operation(summary = "listResourceGroup", description = "LIST_RESOURCE_GROUP")
+    @GetMapping(value = "/azure/datafactory/resourceGroups")
+    @ResponseStatus(HttpStatus.OK)
+    @ApiException(LIST_AZURE_RESOURCE_GROUP_ERROR)
+    @AccessLogAnnotation(ignoreRequestArgs = "loginUser")
+    public Result listResourceGroup(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser) {
+        List<String> resourceGroupNames = cloudService.listResourceGroup(loginUser);
+        return success(Status.SUCCESS.getMsg(), resourceGroupNames);
+    }
+
+    /**
+     * get resourceGroup list
+     *
+     * @param loginUser login user
+     * @return resourceGroup list
+     */
+    @Operation(summary = "listPipeline", description = "LIST_PIPELINE")
+    @GetMapping(value = "/azure/datafactory/pipelines")
+    @ResponseStatus(HttpStatus.OK)
+    @ApiException(LIST_AZURE_DATA_FACTORY_PIPELINE_ERROR)
+    @AccessLogAnnotation(ignoreRequestArgs = "loginUser")
+    public Result listPipeline(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
+                               @RequestParam("factoryName") String factoryName,
+                               @RequestParam("resourceGroupName") String resourceGroupName) {
+        List<String> pipelineNames = cloudService.listPipeline(loginUser, factoryName, resourceGroupName);
+        return success(Status.SUCCESS.getMsg(), pipelineNames);
+    }
+}

dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/enums/Status.java

@@ -265,6 +265,9 @@ public enum Status {
             "请先保存后再执行"),
     START_NODE_NOT_EXIST_IN_LAST_PROCESS(10207, "this node {0} does not exist in the latest process definition",
             "该节点 {0} 不存在于最新的流程定义中"),
+    LIST_AZURE_DATA_FACTORY_ERROR(10208, "list azure data factory error", "查询AZURE数据工厂列表错误"),
+    LIST_AZURE_RESOURCE_GROUP_ERROR(10209, "list azure resource group error", "查询AZURE资源组列表错误"),
+    LIST_AZURE_DATA_FACTORY_PIPELINE_ERROR(10210, "list azure data factory pipeline error", "查询AZURE数据工厂pipeline列表错误"),
 
     UDF_FUNCTION_NOT_EXIST(20001, "UDF function not found", "UDF函数不存在"),
     UDF_FUNCTION_EXISTS(20002, "UDF function already exists", "UDF函数已存在"),

dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/CloudService.java

@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.api.service;
+
+import org.apache.dolphinscheduler.dao.entity.User;
+
+import java.util.List;
+
+public interface CloudService {
+
+    List<String> listDataFactory(User loginUser);
+
+    List<String> listResourceGroup(User loginUser);
+
+    List<String> listPipeline(User loginUser, String factoryName, String resourceGroupName);
+}

dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/CloudServiceImpl.java

@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.api.service.impl;
+
+import org.apache.dolphinscheduler.api.service.CloudService;
+import org.apache.dolphinscheduler.common.utils.PropertyUtils;
+import org.apache.dolphinscheduler.dao.entity.User;
+import org.apache.dolphinscheduler.plugin.task.api.TaskConstants;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.springframework.stereotype.Service;
+
+import com.azure.core.http.rest.PagedIterable;
+import com.azure.core.management.AzureEnvironment;
+import com.azure.core.management.profile.AzureProfile;
+import com.azure.identity.ClientSecretCredential;
+import com.azure.identity.ClientSecretCredentialBuilder;
+import com.azure.resourcemanager.AzureResourceManager;
+import com.azure.resourcemanager.datafactory.DataFactoryManager;
+import com.azure.resourcemanager.datafactory.models.Factories;
+import com.azure.resourcemanager.datafactory.models.PipelineResource;
+import com.azure.resourcemanager.resources.models.ResourceGroups;
+
+@Service
+public class CloudServiceImpl extends BaseServiceImpl implements CloudService {
+
+    private static final String AZURE_CLIENT_ID = PropertyUtils.getString(TaskConstants.AZURE_CLIENT_ID);
+    private static final String AZURE_CLIENT_SECRET = PropertyUtils.getString(TaskConstants.AZURE_CLIENT_SECRET);
+    private static final String AZURE_ACCESS_SUB_ID = PropertyUtils.getString(TaskConstants.AZURE_ACCESS_SUB_ID);
+    private static final String AZURE_SECRET_TENANT_ID = PropertyUtils.getString(TaskConstants.AZURE_SECRET_TENANT_ID);
+    private static final AzureProfile profile =
+            new AzureProfile(AZURE_SECRET_TENANT_ID, AZURE_ACCESS_SUB_ID, AzureEnvironment.AZURE);
+    private static final ClientSecretCredential clientSecretCredential = new ClientSecretCredentialBuilder()
+            .clientId(AZURE_CLIENT_ID)
+            .clientSecret(AZURE_CLIENT_SECRET)
+            .tenantId(AZURE_SECRET_TENANT_ID)
+            .authorityHost(profile.getEnvironment().getActiveDirectoryEndpoint())
+            .build();
+    private static final DataFactoryManager manager;
+    private static final AzureResourceManager azure;
+
+    static {
+        manager = DataFactoryManager.authenticate(clientSecretCredential, profile);
+        azure = AzureResourceManager.authenticate(clientSecretCredential, profile).withDefaultSubscription();
+    }
+
+    @Override
+    public List<String> listDataFactory(User loginUser) {
+        Factories factories = manager.factories();
+        List<String> names = new ArrayList<>();
+        factories.list().stream().forEach(e -> names.add(e.name()));
+        return names;
+    }
+
+    @Override
+    public List<String> listResourceGroup(User loginUser) {
+        ResourceGroups resourceGroups = azure.resourceGroups();
+        List<String> names = new ArrayList<>();
+        resourceGroups.list().stream().forEach(e -> names.add(e.name()));
+        return names;
+    }
+
+    @Override
+    public List<String> listPipeline(User loginUser, String factoryName, String resourceGroupName) {
+        PagedIterable<PipelineResource> pipelineResources =
+                manager.pipelines().listByFactory(resourceGroupName, factoryName);
+        List<String> names = new ArrayList<>();
+        pipelineResources.stream().forEach(e -> names.add(e.name()));
+        return names;
+    }
+}

dolphinscheduler-api/src/main/resources/task-type-config.yaml

@@ -33,6 +33,7 @@ task:
     - 'EMR'
     - 'K8S'
     - 'DMS'
+    - 'DATA_FACTORY'
   logic:
     - 'SUB_PROCESS'
     - 'DEPENDENT'

dolphinscheduler-bom/pom.xml

@@ -103,6 +103,8 @@
         <azure-identity.version>1.7.1</azure-identity.version>
         <redshift-jdbc42.version>2.1.0.9</redshift-jdbc42.version>
         <aws-java-sdk-redshift.version>1.12.300</aws-java-sdk-redshift.version>
+        <azure-resourcemanager.version>2.21.0</azure-resourcemanager.version>
+        <azure-resourcemanager-datafactory.version>1.0.0-beta.19</azure-resourcemanager-datafactory.version>
     </properties>
 
     <dependencyManagement>
@@ -798,6 +800,16 @@
                 <artifactId>azure-identity</artifactId>
                 <version>${azure-identity.version}</version>
             </dependency>
+            <dependency>
+                <groupId>com.azure.resourcemanager</groupId>
+                <artifactId>azure-resourcemanager</artifactId>
+                <version>${azure-resourcemanager.version}</version>
+            </dependency>
+            <dependency>
+                <groupId>com.azure.resourcemanager</groupId>
+                <artifactId>azure-resourcemanager-datafactory</artifactId>
+                <version>${azure-resourcemanager-datafactory.version}</version>
+            </dependency>
             <dependency>
                 <groupId>com.amazon.redshift</groupId>
                 <artifactId>redshift-jdbc42</artifactId>

dolphinscheduler-common/src/main/resources/common.properties

@@ -26,6 +26,17 @@ resource.storage.type=NONE
 # resource store on HDFS/S3 path, resource file will store to this base path, self configuration, please make sure the directory exists on hdfs and have read write permissions. "/dolphinscheduler" is recommended
 resource.storage.upload.base.path=/dolphinscheduler
 
+# The Azure client ID (Azure Application (client) ID)
+resource.azure.client.id=minioadmin
+# The Azure client secret in the Azure application
+resource.azure.client.secret=minioadmin
+# The Azure data factory subscription ID
+resource.azure.subId=minioadmin
+# The Azure tenant id in the Azure Active Directory
+resource.azure.tenant.id=minioadmin
+# The query interval
+resource.query.interval=10000
+
 # The AWS access key. if resource.storage.type=S3 or use EMR-Task, This configuration is required
 resource.aws.access.key.id=minioadmin
 # The AWS secret access key. if resource.storage.type=S3 or use EMR-Task, This configuration is required

dolphinscheduler-dist/release-docs/LICENSE

@@ -517,6 +517,8 @@ The text of each license is also included at licenses/LICENSE-[project].txt.
     reactor-netty-core 1.0.22: https://mvnrepository.com/artifact/io.projectreactor.netty/reactor-netty-core/1.0.22, Apache 2.0
     reactor-netty-http 1.0.22: https://mvnrepository.com/artifact/io.projectreactor.netty/reactor-netty-http/1.0.22, Apache 2.0
     DmJdbcDriver18 8.1.2.79: https://mvnrepository.com/artifact/com.dameng/DmJdbcDriver18/8.1.2.79, Apache 2.0
+    nimbus-jose-jwt 9.22: https://mvnrepository.com/artifact/com.nimbusds/nimbus-jose-jwt/9.22, Apache 2.0
+    woodstox-core 6.4.0: https://mvnrepository.com/artifact/com.fasterxml.woodstox/woodstox-core/6.4.0, Apache 2.0
 
 jna-platform
 
@@ -605,12 +607,42 @@ The text of each license is also included at licenses/LICENSE-[project].txt.
     bcpkix-jdk15on 1.68: https://mvnrepository.com/artifact/org.bouncycastle/bcpkix-jdk15on, MIT
     bcprov-jdk15on 1.68: https://mvnrepository.com/artifact/org.bouncycastle/bcprov-jdk15on, MIT
     reactive-streams 1.0.4: https://mvnrepository.com/artifact/org.reactivestreams/reactive-streams/1.0.4, MIT
-    webjars-locator-core 0.50 https://mvnrepository.com/artifact/org.webjars/webjars-locator-core/0.50, MIT
+    webjars-locator-core 0.50: https://mvnrepository.com/artifact/org.webjars/webjars-locator-core/0.50, MIT
-    azure-core 1.34.0 https://mvnrepository.com/artifact/com.azure/azure-core/1.34.0, MIT
+    azure-core 1.34.0: https://mvnrepository.com/artifact/com.azure/azure-core/1.34.0, MIT
-    azure-core-http-netty 1.12.7 https://mvnrepository.com/artifact/com.azure/azure-core-http-netty/1.12.7, MIT
+    azure-core-http-netty 1.12.7: https://mvnrepository.com/artifact/com.azure/azure-core-http-netty/1.12.7, MIT
-    azure-identity 1.7.1 https://mvnrepository.com/artifact/com.azure/azure-identity/1.7.1, MIT
+    azure-identity 1.7.1: https://mvnrepository.com/artifact/com.azure/azure-identity/1.7.1, MIT
-    msal4j 1.13.3 https://mvnrepository.com/artifact/com.microsoft.azure/msal4j/1.13.3, MIT
+    msal4j 1.13.3: https://mvnrepository.com/artifact/com.microsoft.azure/msal4j/1.13.3, MIT
-    msal4j-persistence-extension 1.1.0 https://mvnrepository.com/artifact/com.microsoft.azure/msal4j-persistence-extension/1.1.0, MIT
+    msal4j-persistence-extension 1.1.0: https://mvnrepository.com/artifact/com.microsoft.azure/msal4j-persistence-extension/1.1.0, MIT
+    azure-resourcemanager 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager/2.21.0, MIT
+    azure-resourcemanager-appplatform 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-appplatform/2.21.0, MIT
+    azure-resourcemanager-appservice 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-appservice/2.21.0, MIT
+    azure-resourcemanager-authorization 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-authorization/2.21.0, MIT
+    azure-resourcemanager-cdn 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-cdn/2.21.0, MIT
+    azure-resourcemanager-compute 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-compute/2.21.0, MIT
+    azure-resourcemanager-containerinstance 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-containerinstance/2.21.0, MIT
+    azure-resourcemanager-containerregistry 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-containerregistry/2.21.0, MIT
+    azure-resourcemanager-containerservice 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-containerservice/2.21.0, MIT
+    azure-resourcemanager-cosmos 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-cosmos/2.21.0, MIT
+    azure-resourcemanager-datafactory 1.0.0-beta.19 https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-datafactory/1.0.0-beta.19, MIT
+    azure-resourcemanager-dns 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-dns/2.21.0, MIT
+    azure-resourcemanager-eventhubs 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-eventhubs/2.21.0, MIT
+    azure-resourcemanager-keyvault 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-keyvault/2.21.0, MIT
+    azure-resourcemanager-monitor 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-monitor/2.21.0, MIT
+    azure-resourcemanager-msi 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-msi/2.21.0, MIT
+    azure-resourcemanager-network 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-network/2.21.0, MIT
+    azure-resourcemanager-privatedns 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-privatedns/2.21.0, MIT
+    azure-resourcemanager-redis 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-redis/2.21.0, MIT
+    azure-resourcemanager-resources 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-resources/2.21.0, MIT
+    azure-resourcemanager-search 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-search/2.21.0, MIT
+    azure-resourcemanager-servicebus 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-servicebus/2.21.0, MIT
+    azure-resourcemanager-sql 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-sql/2.21.0, MIT
+    azure-resourcemanager-storage 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-storage/2.21.0, MIT
+    azure-resourcemanager-trafficmanager 2.21.0: https://mvnrepository.com/artifact/com.azure.resourcemanager/azure-resourcemanager-trafficmanager/2.21.0, MIT
+    azure-security-keyvault-keys 4.5.2: https://mvnrepository.com/artifact/com.azure/azure-security-keyvault-keys/4.5.2, MIT
+    azure-security-keyvault-secrets 4.5.2: https://mvnrepository.com/artifact/com.azure/azure-security-keyvault-secrets/4.5.2, MIT
+    azure-storage-common 12.19.1: https://mvnrepository.com/artifact/com.azure/azure-storage-common/12.19.1, MIT
+    azure-storage-file-share 12.16.1: https://mvnrepository.com/artifact/com.azure/azure-storage-file-share/12.16.1, MIT
+    azure-core-management 1.9.0: https://mvnrepository.com/artifact/com.azure/azure-core-management/1.9.0, MIT
 
 ========================================================================
 MPL 1.1 licenses

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-core-management.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-appplatform.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-appservice.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-authorization.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-cdn.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-compute.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-containerinstance.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-containerregistry.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-containerservice.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-cosmos.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-datafactory.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-dns.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-eventhubs.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-keyvault.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-monitor.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-msi.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-network.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-privatedns.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-redis.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-resources.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-search.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-servicebus.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-sql.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-storage.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager-trafficmanager.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-resourcemanager.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-security-keyvault-keys.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-security-keyvault-secrets.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-storage-common.txt

@@ -0,0 +1,14 @@
+LICENSE-msal4j.txtCopyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-dist/release-docs/licenses/LICENSE-azure-storage-file-share.txt

@@ -0,0 +1,14 @@
+Copyright(c) 2016 Microsoft Corporation
+All rights reserved.
+
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files(the "Software"), 
+to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
+and / or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions :
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
+IN THE SOFTWARE.

dolphinscheduler-e2e/dolphinscheduler-e2e-case/src/test/resources/docker/file-manage/common.properties

@@ -26,6 +26,17 @@ resource.storage.type=S3
 # resource store on HDFS/S3 path, resource file will store to this base path, self configuration, please make sure the directory exists on hdfs and have read write permissions. "/dolphinscheduler" is recommended
 resource.storage.upload.base.path=/dolphinscheduler
 
+# The Azure client ID (Azure Application (client) ID)
+resource.azure.client.id=minioadmin
+# The Azure client secret in the Azure application
+resource.azure.client.secret=minioadmin
+# The Azure data factory subscription ID
+resource.azure.subId=minioadmin
+# The Azure tenant id in the Azure Active Directory
+resource.azure.tenant.id=minioadmin
+# The query interval
+resource.query.interval=10000
+
 # The AWS access key. if resource.storage.type=S3 or use EMR-Task, This configuration is required
 resource.aws.access.key.id=accessKey123
 # The AWS secret access key. if resource.storage.type=S3 or use EMR-Task, This configuration is required

dolphinscheduler-task-plugin/dolphinscheduler-task-all/pom.xml

@@ -212,6 +212,11 @@
             <artifactId>dolphinscheduler-task-linkis</artifactId>
             <version>${project.version}</version>
         </dependency>
+        <dependency>
+            <groupId>org.apache.dolphinscheduler</groupId>
+            <artifactId>dolphinscheduler-task-datafactory</artifactId>
+            <version>${project.version}</version>
+        </dependency>
     </dependencies>
 
 </project>

dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/TaskConstants.java

@@ -443,6 +443,15 @@ public class TaskConstants {
 
     public static final String TASK_TYPE_STREAM = "STREAM";
 
+    /**
+     * azure config
+     */
+    public static final String AZURE_CLIENT_ID = "resource.azure.client.id";
+    public static final String AZURE_CLIENT_SECRET = "resource.azure.client.secret";
+    public static final String AZURE_ACCESS_SUB_ID = "resource.azure.subId";
+    public static final String AZURE_SECRET_TENANT_ID = "resource.azure.tenant.id";
+    public static final String QUERY_INTERVAL = "resource.query.interval";
+
     /**
      * aws config
      */

dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/pom.xml

@@ -0,0 +1,57 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one or more
+  ~ contributor license agreements.  See the NOTICE file distributed with
+  ~ this work for additional information regarding copyright ownership.
+  ~ The ASF licenses this file to You under the Apache License, Version 2.0
+  ~ (the "License"); you may not use this file except in compliance with
+  ~ the License.  You may obtain a copy of the License at
+  ~
+  ~     http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing, software
+  ~ distributed under the License is distributed on an "AS IS" BASIS,
+  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  -->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+    <parent>
+        <groupId>org.apache.dolphinscheduler</groupId>
+        <artifactId>dolphinscheduler-task-plugin</artifactId>
+        <version>dev-SNAPSHOT</version>
+    </parent>
+
+    <artifactId>dolphinscheduler-task-datafactory</artifactId>
+    <packaging>jar</packaging>
+
+    <dependencies>
+        <dependency>
+            <groupId>org.apache.dolphinscheduler</groupId>
+            <artifactId>dolphinscheduler-spi</artifactId>
+            <scope>provided</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.dolphinscheduler</groupId>
+            <artifactId>dolphinscheduler-task-api</artifactId>
+            <scope>provided</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.dolphinscheduler</groupId>
+            <artifactId>dolphinscheduler-common</artifactId>
+            <scope>provided</scope>
+        </dependency>
+        <dependency>
+            <groupId>com.azure.resourcemanager</groupId>
+            <artifactId>azure-resourcemanager-datafactory</artifactId>
+            <version>1.0.0-beta.19</version>
+        </dependency>
+        <dependency>
+            <groupId>com.azure</groupId>
+            <artifactId>azure-identity</artifactId>
+        </dependency>
+    </dependencies>
+
+</project>

dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryHook.java

@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.datafactory;
+
+import org.apache.dolphinscheduler.common.utils.PropertyUtils;
+import org.apache.dolphinscheduler.plugin.task.api.TaskConstants;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.util.Arrays;
+import java.util.List;
+
+import lombok.Data;
+import lombok.SneakyThrows;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.azure.core.management.AzureEnvironment;
+import com.azure.core.management.profile.AzureProfile;
+import com.azure.identity.ClientSecretCredential;
+import com.azure.identity.ClientSecretCredentialBuilder;
+import com.azure.resourcemanager.datafactory.DataFactoryManager;
+import com.azure.resourcemanager.datafactory.models.CreateRunResponse;
+import com.azure.resourcemanager.datafactory.models.PipelineResource;
+import com.azure.resourcemanager.datafactory.models.PipelineRun;
+import com.azure.resourcemanager.datafactory.models.PipelineRuns;
+
+@Data
+public class DatafactoryHook {
+
+    public static DatafactoryStatus[] taskFinishFlags =
+            {DatafactoryStatus.Failed, DatafactoryStatus.Succeeded, DatafactoryStatus.Cancelled};
+    protected final Logger logger =
+            LoggerFactory.getLogger(String.format(TaskConstants.TASK_LOG_LOGGER_NAME_FORMAT, getClass()));
+    private final int QUERY_INTERVAL = PropertyUtils.getInt(TaskConstants.QUERY_INTERVAL, 10000);
+    private DataFactoryManager client;
+    private static AzureProfile profile;
+    private static ClientSecretCredential credential;
+    private String runId;
+
+    public DatafactoryHook() {
+        logger.info("initDatafactoryClient ......");
+        client = createClient();
+    }
+
+    protected static DataFactoryManager createClient() {
+        final String AZURE_ACCESS_SUB_ID = PropertyUtils.getString(TaskConstants.AZURE_ACCESS_SUB_ID);
+        final String AZURE_SECRET_TENANT_ID = PropertyUtils.getString(TaskConstants.AZURE_SECRET_TENANT_ID);
+        final String AZURE_CLIENT_ID = PropertyUtils.getString(TaskConstants.AZURE_CLIENT_ID);
+        final String AZURE_CLIENT_SECRET = PropertyUtils.getString(TaskConstants.AZURE_CLIENT_SECRET);
+        profile =
+                new AzureProfile(AZURE_SECRET_TENANT_ID, AZURE_ACCESS_SUB_ID, AzureEnvironment.AZURE);
+        credential = new ClientSecretCredentialBuilder()
+                .clientId(AZURE_CLIENT_ID)
+                .clientSecret(AZURE_CLIENT_SECRET)
+                .tenantId(AZURE_SECRET_TENANT_ID)
+                .authorityHost(profile.getEnvironment().getActiveDirectoryEndpoint())
+                .build();
+        return DataFactoryManager.authenticate(credential, profile);
+    }
+
+    public Boolean startDatafactoryTask(DatafactoryParameters parameters) {
+        logger.info("initDatafactoryTask ......");
+        PipelineResource pipelineResource = getPipelineResource(parameters);
+        if (pipelineResource == null) {
+            return false;
+        }
+        logger.info("startDatafactoryTask ......");
+        CreateRunResponse run = pipelineResource.createRun();
+        if (StringUtils.isEmpty(run.runId())) {
+            return false;
+        }
+        runId = run.runId();
+        parameters.setRunId(runId);
+        return true;
+    }
+
+    public Boolean cancelDatafactoryTask(DatafactoryParameters parameters) {
+        logger.info("cancelTask ......");
+        PipelineRuns pipelineRuns = client.pipelineRuns();
+        try {
+            pipelineRuns.cancel(parameters.getResourceGroupName(), parameters.getFactoryName(), runId);
+        } catch (RuntimeException e) {
+            logger.error("failed to cancel datafactory task: " + e.getMessage());
+            return false;
+        }
+        return true;
+    }
+
+    public DatafactoryStatus queryDatafactoryTaskStatus(DatafactoryParameters parameters) {
+        logger.info("queryDatafactoryTaskStatus ......");
+
+        PipelineRuns pipelineRuns = client.pipelineRuns();
+        PipelineRun pipelineRun =
+                pipelineRuns.get(parameters.getResourceGroupName(), parameters.getFactoryName(), runId);
+
+        if (pipelineRun != null) {
+            logger.info("queryDatafactoryTaskStatus ......{}", pipelineRun.status());
+            return DatafactoryStatus.valueOf(pipelineRun.status());
+        }
+        return null;
+    }
+
+    private PipelineResource getPipelineResource(DatafactoryParameters parameters) {
+        return client.pipelines().get(parameters.getResourceGroupName(), parameters.getFactoryName(),
+                parameters.getPipelineName());
+    }
+
+    @SneakyThrows
+    public Boolean queryStatus(DatafactoryParameters parameters) {
+        List<DatafactoryStatus> stopStatusSet = Arrays.asList(taskFinishFlags);
+        int maxRetry = 5;
+        while (maxRetry > 0) {
+            DatafactoryStatus status = queryDatafactoryTaskStatus(parameters);
+
+            if (status == null) {
+                maxRetry--;
+                continue;
+            }
+
+            if (stopStatusSet.contains(status)) {
+                if (status.equals(DatafactoryStatus.Succeeded)) {
+                    return true;
+                }
+                return false;
+            }
+            logger.debug("wait {}ms to recheck finish status....", QUERY_INTERVAL);
+            Thread.sleep(QUERY_INTERVAL);
+        }
+        return false;
+    }
+}

dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryParameters.java

@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.datafactory;
+
+import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
+
+import org.apache.commons.lang3.StringUtils;
+
+import lombok.Getter;
+import lombok.NoArgsConstructor;
+import lombok.Setter;
+import lombok.ToString;
+
+@Setter
+@Getter
+@NoArgsConstructor
+@ToString
+public class DatafactoryParameters extends AbstractParameters {
+
+    private String factoryName;
+    private String resourceGroupName;
+    private String pipelineName;
+    private String runId;
+
+    @Override
+    public boolean checkParameters() {
+        return StringUtils.isNotEmpty(factoryName) && StringUtils.isNotEmpty(resourceGroupName)
+                && StringUtils.isNotEmpty(pipelineName);
+    }
+}

dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryStatus.java

@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.datafactory;
+
+public enum DatafactoryStatus {
+
+    Queued,
+    InProgress,
+    Succeeded,
+    Failed,
+    Canceling,
+    Cancelled,
+    ;
+
+    /**
+     * Gets the status property: The status of a pipeline run. Possible values: Queued, InProgress, Succeeded, Failed,
+     * Canceling, Cancelled.
+     *
+     * @return the status value.
+     */
+    DatafactoryStatus() {
+    }
+}

dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryTask.java

@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.datafactory;
+
+import org.apache.dolphinscheduler.common.utils.JSONUtils;
+import org.apache.dolphinscheduler.plugin.task.api.AbstractRemoteTask;
+import org.apache.dolphinscheduler.plugin.task.api.TaskConstants;
+import org.apache.dolphinscheduler.plugin.task.api.TaskException;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.util.Collections;
+import java.util.List;
+
+import lombok.Getter;
+import lombok.Setter;
+
+@Setter
+@Getter
+public class DatafactoryTask extends AbstractRemoteTask {
+
+    private final TaskExecutionContext taskExecutionContext;
+    private DatafactoryParameters parameters;
+    private DatafactoryHook hook;
+
+    public DatafactoryTask(TaskExecutionContext taskExecutionContext) {
+        super(taskExecutionContext);
+        this.taskExecutionContext = taskExecutionContext;
+    }
+
+    @Override
+    public List<String> getApplicationIds() throws TaskException {
+        return Collections.emptyList();
+    }
+
+    @Override
+    public void init() {
+        parameters = JSONUtils.parseObject(taskExecutionContext.getTaskParams(), DatafactoryParameters.class);
+        logger.info("Initialize Datafactory task params {}", JSONUtils.toPrettyJsonString(parameters));
+        hook = new DatafactoryHook();
+    }
+
+    @Override
+    public void submitApplication() throws TaskException {
+        try {
+            // start task
+            exitStatusCode = startDatafactoryTask();
+            setExitStatusCode(exitStatusCode);
+        } catch (Exception e) {
+            setExitStatusCode(TaskConstants.EXIT_CODE_FAILURE);
+            throw new TaskException("data factory start task error", e);
+        }
+        // set runId to the appIds if start success
+        setAppIds(parameters.getRunId());
+    }
+
+    @Override
+    public void cancelApplication() throws TaskException {
+        checkApplicationId();
+        hook.cancelDatafactoryTask(parameters);
+        exitStatusCode = TaskConstants.EXIT_CODE_KILL;
+    }
+
+    @Override
+    public void trackApplicationStatus() throws TaskException {
+        checkApplicationId();
+        Boolean isFinishedSuccessfully;
+        isFinishedSuccessfully = hook.queryStatus(parameters);
+        if (!isFinishedSuccessfully) {
+            exitStatusCode = TaskConstants.EXIT_CODE_FAILURE;
+        } else {
+            exitStatusCode = TaskConstants.EXIT_CODE_SUCCESS;
+        }
+    }
+
+    /**
+     * check datafactory applicationId or get it from appId
+     */
+    private void checkApplicationId() {
+        String taskExecArn = hook.getRunId();
+        if (StringUtils.isEmpty(taskExecArn)) {
+            if (StringUtils.isEmpty(getAppIds())) {
+                throw new TaskException("datafactory runId is null, not created yet");
+            }
+            hook.setRunId(getAppIds());
+        }
+    }
+
+    public int startDatafactoryTask() {
+        Boolean isStartSuccessfully = hook.startDatafactoryTask(parameters);
+        if (!isStartSuccessfully) {
+            return TaskConstants.EXIT_CODE_FAILURE;
+        }
+        return TaskConstants.EXIT_CODE_SUCCESS;
+    }
+
+    @Override
+    public DatafactoryParameters getParameters() {
+        return parameters;
+    }
+
+}

dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryTaskChannel.java

@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.datafactory;
+
+import org.apache.dolphinscheduler.common.utils.JSONUtils;
+import org.apache.dolphinscheduler.plugin.task.api.TaskChannel;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.ParametersNode;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.resource.ResourceParametersHelper;
+
+public class DatafactoryTaskChannel implements TaskChannel {
+
+    @Override
+    public void cancelApplication(boolean status) {
+    }
+
+    @Override
+    public DatafactoryTask createTask(TaskExecutionContext taskRequest) {
+        return new DatafactoryTask(taskRequest);
+    }
+
+    @Override
+    public AbstractParameters parseParameters(ParametersNode parametersNode) {
+        return JSONUtils.parseObject(parametersNode.getTaskParams(), DatafactoryParameters.class);
+    }
+
+    @Override
+    public ResourceParametersHelper getResources(String parameters) {
+        return null;
+    }
+}

dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/main/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryTaskChannelFactory.java

@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.datafactory;
+
+import org.apache.dolphinscheduler.plugin.task.api.TaskChannel;
+import org.apache.dolphinscheduler.plugin.task.api.TaskChannelFactory;
+import org.apache.dolphinscheduler.spi.params.base.PluginParams;
+
+import java.util.Collections;
+import java.util.List;
+
+import com.google.auto.service.AutoService;
+
+@AutoService(TaskChannelFactory.class)
+public class DatafactoryTaskChannelFactory implements TaskChannelFactory {
+
+    @Override
+    public TaskChannel create() {
+        return new DatafactoryTaskChannel();
+    }
+
+    @Override
+    public String getName() {
+        return "DATA_FACTORY";
+    }
+
+    @Override
+    public List<PluginParams> getParams() {
+        return Collections.emptyList();
+    }
+}

dolphinscheduler-task-plugin/dolphinscheduler-task-datafactory/src/test/java/org/apache/dolphinscheduler/plugin/task/datafactory/DatafactoryTaskTest.java

@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.datafactory;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.mockStatic;
+import static org.mockito.Mockito.when;
+
+import org.apache.dolphinscheduler.common.utils.JSONUtils;
+import org.apache.dolphinscheduler.plugin.task.api.TaskConstants;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.mockito.Mock;
+import org.mockito.MockedStatic;
+import org.mockito.Mockito;
+import org.mockito.junit.jupiter.MockitoExtension;
+
+import com.azure.resourcemanager.datafactory.DataFactoryManager;
+import com.azure.resourcemanager.datafactory.models.CreateRunResponse;
+import com.azure.resourcemanager.datafactory.models.PipelineResource;
+import com.azure.resourcemanager.datafactory.models.Pipelines;
+
+@ExtendWith(MockitoExtension.class)
+public class DatafactoryTaskTest {
+
+    private static final String mockRunId =
+            "3c7182f4-d107-43c7-af2a-8c7b3ed1d614";
+
+    DatafactoryTask datafactoryTask;
+
+    @Mock
+    DataFactoryManager client;
+    MockedStatic<DatafactoryHook> datafactoryHookMockedStatic;
+
+    @BeforeEach
+    public void before() throws IllegalAccessException {
+        client = mock(DataFactoryManager.class);
+        datafactoryHookMockedStatic = mockStatic(DatafactoryHook.class);
+        when(DatafactoryHook.createClient()).thenReturn(client);
+
+        DatafactoryParameters datafactoryParameters = new DatafactoryParameters();
+        datafactoryTask = initTask(datafactoryParameters);
+    }
+
+    @Test
+    public void testConvertParam() {
+        DatafactoryParameters parameters = castParam();
+        Assertions.assertEquals("whale1", parameters.getFactoryName());
+        Assertions.assertEquals("factory1", parameters.getResourceGroupName());
+        Assertions.assertEquals("CopyPipeline_ps8", parameters.getPipelineName());
+        datafactoryHookMockedStatic.close();
+    }
+
+    @Test
+    public void testCheckCreateTask() {
+        DatafactoryHook hook = new DatafactoryHook();
+        DataFactoryManager client1 = DatafactoryHook.createClient();
+        hook.setClient(client1);
+
+        PipelineResource pipelineResource = mock(PipelineResource.class);
+        Pipelines pipelines = mock(Pipelines.class);
+        when(client1.pipelines()).thenReturn(pipelines);
+        when(pipelines.get(any(), any(), any())).thenReturn(pipelineResource);
+
+        CreateRunResponse response = mock(CreateRunResponse.class);
+        when(pipelineResource.createRun()).thenReturn(response);
+        when(response.runId()).thenReturn(mockRunId);
+        hook.startDatafactoryTask(castParam());
+        Assertions.assertEquals(mockRunId, hook.getRunId());
+        datafactoryHookMockedStatic.close();
+    }
+
+    @Test
+    public void testCancelTask() {
+        datafactoryTask.cancelApplication();
+        Assertions.assertEquals(datafactoryTask.getExitStatusCode(), TaskConstants.EXIT_CODE_KILL);
+        datafactoryHookMockedStatic.close();
+    }
+
+    private DatafactoryParameters castParam() {
+        String input =
+                "{\"localParams\":[],\"resourceList\":[],\"factoryName\":\"whale1\",\"resourceGroupName\":\"factory1\",\"pipelineName\":\"CopyPipeline_ps8\"}";
+        return JSONUtils.parseObject(input, DatafactoryParameters.class);
+    }
+
+    private DatafactoryTask initTask(DatafactoryParameters DatasyncParameters) {
+        TaskExecutionContext taskExecutionContext = createContext(DatasyncParameters);
+        DatafactoryTask task = new DatafactoryTask(taskExecutionContext);
+        task.init();
+        task.getHook().setRunId(mockRunId);
+        return task;
+    }
+
+    public TaskExecutionContext createContext(DatafactoryParameters datafactoryParameters) {
+        String parameters = JSONUtils.toJsonString(datafactoryParameters);
+        TaskExecutionContext taskExecutionContext = Mockito.mock(TaskExecutionContext.class);
+        Mockito.when(taskExecutionContext.getTaskParams()).thenReturn(parameters);
+        return taskExecutionContext;
+    }
+}

dolphinscheduler-task-plugin/pom.xml

@@ -61,6 +61,7 @@
         <module>dolphinscheduler-task-datasync</module>
         <module>dolphinscheduler-task-kubeflow</module>
         <module>dolphinscheduler-task-linkis</module>
+        <module>dolphinscheduler-task-datafactory</module>
     </modules>
 
     <dependencyManagement>

dolphinscheduler-ui/src/locales/en_US/project.ts

@@ -37,10 +37,10 @@ export default {
     confirm: 'Confirm',
     cancel: 'Cancel',
     delete_confirm: 'Delete?',
-    authorize_level:'Authorize Level',
+    authorize_level: 'Authorize Level',
     no_permission: 'No Permission',
     read_permission: 'Read Permission',
-    all_permission: 'All Permission',
+    all_permission: 'All Permission'
   },
   workflow: {
     on_line: 'Online',
@@ -215,7 +215,7 @@ export default {
     workflow_relation_no_data_result_desc:
       'There is not any workflows. Please create a workflow, and then visit this page again.',
     ready_to_block: 'Ready to block',
-    block: 'Block',
+    block: 'Block'
   },
   task: {
     on_line: 'Online',
@@ -329,7 +329,7 @@ export default {
     online: 'Online'
   },
   node: {
-    is_cache: "Cache Execution",
+    is_cache: 'Cache Execution',
     jvm_args: 'Java VM Parameters',
     jvm_args_tips: 'Please enter virtual machine parameters',
     run_type: 'Run Type',
@@ -377,7 +377,8 @@ export default {
     image: 'Image',
     image_tips: 'Please enter image',
     command: 'Command',
-    command_tips: 'Please enter the container execution command, for example: /bin/echo hello world',
+    command_tips:
+      'Please enter the container execution command, for example: /bin/echo hello world',
     min_memory_tips: 'Please enter min memory',
     state: 'State',
     branch_flow: 'Branch flow',
@@ -798,7 +799,13 @@ export default {
     pytorch_requirements: 'Requirement File',
     pytorch_conda_python_version: 'Python Version',
     pytorch_conda_python_version_tips:
-      'Please enter the version number, such as 3.6, 3.7, 3.x'
+      'Please enter the version number, such as 3.6, 3.7, 3.x',
+    factory_name: 'Factory Name',
+    resource_group_name: 'Resource Group Name',
+    pipeline_name: 'Pipeline Name',
+    factory_tips: 'Please select factory',
+    resource_group_tips: 'Please select resource group',
+    pipeline_tips: 'Please select pipeline'
   },
   menu: {
     fav: 'Favorites',
@@ -808,6 +815,6 @@ export default {
     di: 'Data Integration',
     dq: 'Data Quality',
     ml: 'Machine Learning',
-    other: 'Other',
+    other: 'Other'
   }
 }

dolphinscheduler-ui/src/locales/zh_CN/project.ts

@@ -777,7 +777,13 @@ export default {
     pytorch_python_env_tool: 'python环境管理工具',
     pytorch_requirements: '依赖文件',
     pytorch_conda_python_version: 'python版本',
-    pytorch_conda_python_version_tips: '请输入版本号，如 3.6, 3.7, 3.x等'
+    pytorch_conda_python_version_tips: '请输入版本号，如 3.6, 3.7, 3.x等',
+    factory_name: '工厂名称',
+    resource_group_name: '资源组名称',
+    pipeline_name: 'pipeline名称',
+    factory_tips: '请选择工厂',
+    resource_group_tips: '请选择资源组',
+    pipeline_tips: '请选择pipeline'
   },
   menu: {
     fav: '收藏组件',

dolphinscheduler-ui/src/service/modules/azure/index.ts

@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { axios } from '@/service/service'
+import { DataFactoryReq } from './types'
+
+export function queryDataFactoryFactories(): any {
+  return axios({
+    url: '/cloud/azure/datafactory/factories',
+    method: 'get'
+  })
+}
+
+export function queryDataFactoryResourceGroups(): any {
+  return axios({
+    url: '/cloud/azure/datafactory/resourceGroups',
+    method: 'get'
+  })
+}
+
+export function queryDataFactoryPipelines(params: DataFactoryReq): any {
+  return axios({
+    url: '/cloud/azure/datafactory/pipelines',
+    method: 'get',
+    params
+  })
+}

dolphinscheduler-ui/src/service/modules/azure/types.ts

@@ -0,0 +1,23 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+interface DataFactoryReq {
+  factoryName: string
+  resourceGroupName: string
+}
+
+export { DataFactoryReq }

dolphinscheduler-ui/src/store/project/task-type.ts

@@ -149,6 +149,10 @@ export const TASK_TYPES_MAP = {
   LINKIS: {
     alias: 'LINKIS',
     helperLinkDisable: true
+  },
+  DATA_FACTORY: {
+    alias: 'DATA_FACTORY',
+    helperLinkDisable: true
   }
 } as {
   [key in TaskType]: {

dolphinscheduler-ui/src/store/project/types.ts

@@ -56,6 +56,7 @@ type TaskType =
   | 'DATASYNC'
   | 'KUBEFLOW'
   | 'LINKIS'
+  | 'DATA_FACTORY'
 
 type ProgramType = 'JAVA' | 'SCALA' | 'PYTHON'
 

dolphinscheduler-ui/src/views/projects/task/components/node/fields/index.ts

@@ -85,3 +85,4 @@ export { useDms } from './use-dms'
 export { useDatasync } from './use-datasync'
 export { useKubeflow } from './use-kubeflow'
 export { useLinkis } from './use-linkis'
+export { useDataFactory } from './use-data-factory'

dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-data-factory.ts

@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {
+  queryDataFactoryFactories,
+  queryDataFactoryPipelines,
+  queryDataFactoryResourceGroups
+} from '@/service/modules/azure'
+import { onMounted, ref, watch } from 'vue'
+import { useI18n } from 'vue-i18n'
+import { useCustomParams } from '.'
+import type { IJsonItem } from '../types'
+
+export function useDataFactory(model: { [field: string]: any }): IJsonItem[] {
+  const { t } = useI18n()
+
+  const factoryLoading = ref(false)
+  const resourceGroupLoading = ref(false)
+  const pipelineLoading = ref(false)
+
+  const factoryOptions = ref([] as { label: string; value: number }[])
+  const resourceGroupOptions = ref([] as { label: string; value: number }[])
+  const pipelineOptions = ref([] as { label: string; value: number }[])
+
+  const getFactoryOptions = async () => {
+    if (factoryLoading.value) return
+    factoryLoading.value = true
+    const factories = await queryDataFactoryFactories()
+    factoryOptions.value = factories.map((factory: string) => ({
+      label: factory,
+      value: factory
+    }))
+    factoryLoading.value = false
+  }
+
+  const getResourceGroupName = async () => {
+    if (resourceGroupLoading.value) return
+    resourceGroupLoading.value = true
+    const groupNames = await queryDataFactoryResourceGroups()
+    resourceGroupOptions.value = groupNames.map((groupName: string) => ({
+      label: groupName,
+      value: groupName
+    }))
+    resourceGroupLoading.value = false
+  }
+
+  const getPipelineName = async (
+    factoryName: string,
+    resourceGroupName: string
+  ) => {
+    if (pipelineLoading.value) return
+    pipelineLoading.value = true
+    const pipelineNames = await queryDataFactoryPipelines({
+      factoryName,
+      resourceGroupName
+    })
+    pipelineOptions.value = pipelineNames.map((pipelineName: string) => ({
+      label: pipelineName,
+      value: pipelineName
+    }))
+    pipelineLoading.value = false
+  }
+
+  const onChange = () => {
+    model['pipelineName'] = ''
+    if (model['factoryName'] && model['resourceGroupName']) {
+      getPipelineName(model['factoryName'], model['resourceGroupName'])
+    }
+  }
+
+  watch(
+    () => model['pipelineName'],
+    () => {
+      if (model['pipelineName'] && pipelineOptions.value.length === 0) {
+        getPipelineName(model['factoryName'], model['resourceGroupName'])
+      }
+    }
+  )
+
+  onMounted(() => {
+    getFactoryOptions()
+    getResourceGroupName()
+  })
+
+  return [
+    {
+      type: 'select',
+      field: 'factoryName',
+      span: 24,
+      name: t('project.node.factory_name'),
+      options: factoryOptions,
+      props: {
+        'on-update:value': onChange,
+        loading: factoryLoading
+      },
+      validate: {
+        trigger: ['input', 'trigger'],
+        required: true,
+        message: t('project.node.factory_tips')
+      }
+    },
+    {
+      type: 'select',
+      field: 'resourceGroupName',
+      span: 24,
+      name: t('project.node.resource_group_name'),
+      options: resourceGroupOptions,
+      props: {
+        'on-update:value': onChange,
+        loading: resourceGroupLoading
+      },
+      validate: {
+        trigger: ['input', 'trigger'],
+        required: true,
+        message: t('project.node.resource_group_tips')
+      }
+    },
+    {
+      type: 'select',
+      field: 'pipelineName',
+      span: 24,
+      name: t('project.node.pipeline_name'),
+      options: pipelineOptions,
+      props: {
+        loading: pipelineLoading
+      },
+      validate: {
+        trigger: ['input', 'trigger'],
+        required: true,
+        message: t('project.node.pipeline_tips')
+      }
+    },
+    ...useCustomParams({
+      model,
+      field: 'localParams',
+      isSimple: true
+    })
+  ]
+}

dolphinscheduler-ui/src/views/projects/task/components/node/format-data.ts

@@ -456,6 +456,12 @@ export function formatParams(data: INodeData): {
     taskParams.rawScript = data.rawScript
   }
 
+  if (data.taskType === 'DATA_FACTORY') {
+    taskParams.factoryName = data.factoryName
+    taskParams.resourceGroupName = data.resourceGroupName
+    taskParams.pipelineName = data.pipelineName
+  }
+
   let timeoutNotifyStrategy = ''
   if (data.timeoutNotifyStrategy) {
     if (data.timeoutNotifyStrategy.length === 1) {

dolphinscheduler-ui/src/views/projects/task/components/node/tasks/index.ts

@@ -50,6 +50,7 @@ import { useDms } from './use-dms'
 import { useDatasync } from './use-datasync'
 import { useKubeflow } from './use-kubeflow'
 import { useLinkis } from './use-linkis'
+import { useDataFactory } from './use-data-factory'
 
 export default {
   SHELL: useShell,
@@ -86,5 +87,6 @@ export default {
   DMS: useDms,
   DATASYNC: useDatasync,
   KUBEFLOW: useKubeflow,
-  LINKIS: useLinkis
+  LINKIS: useLinkis,
+  DATA_FACTORY: useDataFactory
 }

dolphinscheduler-ui/src/views/projects/task/components/node/tasks/use-data-factory.ts

@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { reactive } from 'vue'
+import * as Fields from '../fields/index'
+import type { IJsonItem, INodeData } from '../types'
+import { ITaskData } from '../types'
+
+export function useDataFactory({
+  projectCode,
+  from = 0,
+  readonly,
+  data
+}: {
+  projectCode: number
+  from?: number
+  readonly?: boolean
+  data?: ITaskData
+}) {
+  const model = reactive({
+    name: '',
+    taskType: 'DATA_FACTORY',
+    flag: 'YES',
+    description: '',
+    timeoutFlag: false,
+    localParams: [],
+    environmentCode: null,
+    failRetryInterval: 1,
+    failRetryTimes: 0,
+    workerGroup: 'default',
+    delayTime: 0,
+    timeout: 30,
+    timeoutNotifyStrategy: ['WARN'],
+    factoryName: '',
+    resourceGroupName: '',
+    pipelineName: ''
+  } as INodeData)
+
+  return {
+    json: [
+      Fields.useName(from),
+      ...Fields.useTaskDefinition({ projectCode, from, readonly, data, model }),
+      Fields.useRunFlag(),
+      Fields.useCache(),
+      Fields.useDescription(),
+      Fields.useTaskPriority(),
+      Fields.useWorkerGroup(),
+      Fields.useEnvironmentName(model, !data?.id),
+      ...Fields.useTaskGroup(model, projectCode),
+      ...Fields.useFailed(),
+      Fields.useDelayTime(model),
+      ...Fields.useTimeoutAlarm(model),
+      ...Fields.useDataFactory(model),
+      Fields.usePreTasks()
+    ] as IJsonItem[],
+    model
+  }
+}

dolphinscheduler-ui/src/views/projects/task/components/node/types.ts

@@ -405,6 +405,9 @@ interface ITaskParams {
   cloudWatchLogGroupArn?: string
   yamlContent?: string
   paramScript?: ILocalParam[]
+  factoryName?: string
+  resourceGroupName?: string
+  pipelineName?: string
 }
 
 interface INodeData

dolphinscheduler-ui/src/views/projects/task/constants/task-type.ts

@@ -50,6 +50,7 @@ export type TaskType =
   | 'DATASYNC'
   | 'KUBEFLOW'
   | 'LINKIS'
+  | 'DATA_FACTORY'
 
 export type TaskExecuteType = 'STREAM' | 'BATCH'
 
@@ -180,6 +181,10 @@ export const TASK_TYPES_MAP = {
   LINKIS: {
     alias: 'LINKIS',
     helperLinkDisable: true
+  },
+  DATA_FACTORY: {
+    alias: 'DATA_FACTORY',
+    helperLinkDisable: true
   }
 } as {
   [key in TaskType]: {

dolphinscheduler-ui/src/views/projects/workflow/components/dag/dag.module.scss

@@ -204,6 +204,9 @@ $bgLight: #ffffff;
     &.icon-kubeflow {
       background-image: url('/images/task-icons/kubeflow.png');
     }
+    &.icon-data_factory {
+      background-image: url('/images/task-icons/data_factory.png');
+    }
   }
 
   &:hover {
@@ -311,6 +314,9 @@ $bgLight: #ffffff;
       &.icon-kubeflow {
         background-image: url('/images/task-icons/kubeflow_hover.png');
       }
+      &.icon-data_factory {
+        background-image: url('/images/task-icons/data_factory_hover.png');
+      }
     }
   }
 

tools/dependencies/known-dependencies.txt

@@ -399,3 +399,35 @@ reactor-core-3.4.22.jar
 reactor-netty-core-1.0.22.jar
 reactor-netty-http-1.0.22.jar
 DmJdbcDriver18-8.1.2.79.jar
+azure-resourcemanager-2.21.0.jar
+azure-resourcemanager-appplatform-2.21.0.jar
+azure-resourcemanager-appservice-2.21.0.jar
+azure-resourcemanager-authorization-2.21.0.jar
+azure-resourcemanager-cdn-2.21.0.jar
+azure-resourcemanager-compute-2.21.0.jar
+azure-resourcemanager-containerinstance-2.21.0.jar
+azure-resourcemanager-containerregistry-2.21.0.jar
+azure-resourcemanager-containerservice-2.21.0.jar
+azure-resourcemanager-cosmos-2.21.0.jar
+azure-resourcemanager-datafactory-1.0.0-beta.19.jar
+azure-resourcemanager-dns-2.21.0.jar
+azure-resourcemanager-eventhubs-2.21.0.jar
+azure-resourcemanager-keyvault-2.21.0.jar
+azure-resourcemanager-monitor-2.21.0.jar
+azure-resourcemanager-msi-2.21.0.jar
+azure-resourcemanager-network-2.21.0.jar
+azure-resourcemanager-privatedns-2.21.0.jar
+azure-resourcemanager-redis-2.21.0.jar
+azure-resourcemanager-resources-2.21.0.jar
+azure-resourcemanager-search-2.21.0.jar
+azure-resourcemanager-servicebus-2.21.0.jar
+azure-resourcemanager-sql-2.21.0.jar
+azure-resourcemanager-storage-2.21.0.jar
+azure-resourcemanager-trafficmanager-2.21.0.jar
+azure-security-keyvault-keys-4.5.2.jar
+azure-security-keyvault-secrets-4.5.2.jar
+azure-storage-common-12.19.1.jar
+azure-storage-file-share-12.16.1.jar
+nimbus-jose-jwt-9.22.jar
+woodstox-core-6.4.0.jar
+azure-core-management-1.9.0.jar