Browse Source

[Feature-14339][Parameter] Add parameter combining of project-level parameters (#14452)

3.2.1-prepare
Rick Cheng 1 year ago committed by GitHub
parent
commit
3de0b5a38c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
  1. 8
      docs/configs/docsdev.js
  2. 3
      docs/docs/en/guide/parameter/priority.md
  3. 23
      docs/docs/en/guide/parameter/project-parameter.md
  4. 3
      docs/docs/zh/guide/parameter/priority.md
  5. 23
      docs/docs/zh/guide/parameter/project-parameter.md
  6. BIN
      docs/img/new_ui/dev/parameter/project_parameter01.png
  7. BIN
      docs/img/new_ui/dev/parameter/project_parameter02.png
  8. BIN
      docs/img/new_ui/dev/parameter/project_parameter03.png
  9. 9
      dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ProjectParameterServiceImpl.java
  10. 121
      dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/controller/ProjectParameterControllerTest.java
  11. 166
      dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/service/ProjectParameterServiceTest.java
  12. 3
      dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/ProjectParameterMapper.java
  13. 9
      dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ProjectParameterMapper.xml
  14. 95
      dolphinscheduler-dao/src/test/java/org/apache/dolphinscheduler/dao/mapper/ProjectParameterMapperTest.java
  15. 2
      dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/expand/CuringParamsService.java
  16. 63
      dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/expand/CuringParamsServiceImpl.java
  17. 12
      dolphinscheduler-service/src/test/java/org/apache/dolphinscheduler/service/expand/CuringParamsServiceTest.java

8
docs/configs/docsdev.js

@ -250,6 +250,10 @@ export default {
title: 'Local Parameter', title: 'Local Parameter',
link: '/en-us/docs/dev/user_doc/guide/parameter/local.html', link: '/en-us/docs/dev/user_doc/guide/parameter/local.html',
}, },
{
title: 'Project-level Parameter',
link: '/en-us/docs/dev/user_doc/guide/parameter/project-parameter.html',
},
{ {
title: 'Parameter Context', title: 'Parameter Context',
link: '/en-us/docs/dev/user_doc/guide/parameter/context.html', link: '/en-us/docs/dev/user_doc/guide/parameter/context.html',
@ -975,6 +979,10 @@ export default {
title: '本地参数', title: '本地参数',
link: '/zh-cn/docs/dev/user_doc/guide/parameter/local.html', link: '/zh-cn/docs/dev/user_doc/guide/parameter/local.html',
}, },
{
title: '项目级别参数',
link: '/zh-cn/docs/dev/user_doc/guide/parameter/project-parameter.html',
},
{ {
title: '参数传递', title: '参数传递',
link: '/zh-cn/docs/dev/user_doc/guide/parameter/context.html', link: '/zh-cn/docs/dev/user_doc/guide/parameter/context.html',

3
docs/docs/en/guide/parameter/priority.md

@ -2,6 +2,7 @@
DolphinScheduler has three parameter types: DolphinScheduler has three parameter types:
* [Project-level Parameter](project-parameter.md): parameters defined at the project management page.
* [Global Parameter](global.md): parameters defined at the workflow define page. * [Global Parameter](global.md): parameters defined at the workflow define page.
* [Startup Parameter](startup-parameter.md): parameters defined at the workflow launch page. * [Startup Parameter](startup-parameter.md): parameters defined at the workflow launch page.
* [Parameter Context](context.md): parameters passed by upstream task nodes. * [Parameter Context](context.md): parameters passed by upstream task nodes.
@ -9,7 +10,7 @@ DolphinScheduler has three parameter types:
The user can define part of the parameters when creating workflow definitions. The user can define part of the parameters when creating workflow definitions.
As there are multiple sources of the parameter value, it will raise parameter priority issues when the parameter name is the same. The priority of DolphinScheduler parameters from high to low is: `Startup Parameter > Local Parameter > Parameter Context > Global Parameter`. As there are multiple sources of the parameter value, it will raise parameter priority issues when the parameter name is the same. The priority of DolphinScheduler parameters from high to low is: `Startup Parameter > Local Parameter > Parameter Context > Global Parameter > Project-level Parameter`.
In the case of upstream tasks can pass parameters to the downstream, there may be multiple tasks upstream that pass the same parameter name: In the case of upstream tasks can pass parameters to the downstream, there may be multiple tasks upstream that pass the same parameter name:

23
docs/docs/en/guide/parameter/project-parameter.md

@ -0,0 +1,23 @@
# Project-level Parameter
## Scope
Project-level parameters are valid for all task nodes under the entire project.
## Usage
### Define project-level parameters
On the project page, click Project Parameters and Create Parameters, and fill in the parameter name and parameter value. As shown below:
![project-parameter01](../../../../img/new_ui/dev/parameter/project_parameter01.png)
### Use project-level parameters
Take the shell task as an example, enter `echo ${param}` in the script content, where `param` is the project-level parameter created in the previous step.
![project-parameter02](../../../../img/new_ui/dev/parameter/project_parameter02.png)
Run the shell task. On the task instance page, you can view the task log to verify whether the parameters are valid.
![project-parameter03](../../../../img/new_ui/dev/parameter/project_parameter03.png)

3
docs/docs/zh/guide/parameter/priority.md

@ -2,12 +2,13 @@
DolphinScheduler 中所涉及的参数值的定义可能来自三种类型: DolphinScheduler 中所涉及的参数值的定义可能来自三种类型:
* [项目级别参数](project-parameter.md):在项目管理中定义的项目级别参数
* [全局参数](global.md):在工作流保存页面定义时定义的变量 * [全局参数](global.md):在工作流保存页面定义时定义的变量
* [启动参数](startup-parameter.md):在工作流启动页面定义的变量 * [启动参数](startup-parameter.md):在工作流启动页面定义的变量
* [上游任务传递的参数](context.md):上游任务传递过来的参数 * [上游任务传递的参数](context.md):上游任务传递过来的参数
* [本地参数](local.md):节点的自有变量,用户在“自定义参数”定义的变量,并且用户可以在工作流定义时定义该部分变量的值 * [本地参数](local.md):节点的自有变量,用户在“自定义参数”定义的变量,并且用户可以在工作流定义时定义该部分变量的值
因为参数的值存在多个来源,当参数名相同时,就需要会存在参数优先级的问题。DolphinScheduler 参数的优先级从高到低为:`启动参数 > 本地参数 > 上游任务传递的参数 > 全局参数` 因为参数的值存在多个来源,当参数名相同时,就需要会存在参数优先级的问题。DolphinScheduler 参数的优先级从高到低为:`启动参数 > 本地参数 > 上游任务传递的参数 > 全局参数 > 项目级别参数`
在上游任务传递的参数中,由于上游可能存在多个任务向下游传递参数,当上游传递的参数名称相同时: 在上游任务传递的参数中,由于上游可能存在多个任务向下游传递参数,当上游传递的参数名称相同时:

23
docs/docs/zh/guide/parameter/project-parameter.md

@ -0,0 +1,23 @@
# 项目级别参数
## 作用域
项目级别参数是针对整个项目下的所有任务节点都有效的参数。
## 使用方式
### 定义项目级别参数
在项目管理页面,点击项目级别参数,点击创建项目级别参数,填写参数名称和参数值。如下图所示:
![project-parameter01](../../../../img/new_ui/dev/parameter/project_parameter01.png)
### 使用项目级别参数
以shell任务为例,在脚本内容中输入`echo ${param}`,其中`param`为上一步创建的项目级别参数。
![project-parameter02](../../../../img/new_ui/dev/parameter/project_parameter02.png)
运行该shell任务,在任务实例页面,可以查看任务日志,验证参数是否有效。
![project-parameter03](../../../../img/new_ui/dev/parameter/project_parameter03.png)

BIN
docs/img/new_ui/dev/parameter/project_parameter01.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.1 MiB

BIN
docs/img/new_ui/dev/parameter/project_parameter02.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.3 MiB

BIN
docs/img/new_ui/dev/parameter/project_parameter03.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.7 MiB

9
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ProjectParameterServiceImpl.java

@ -78,7 +78,12 @@ public class ProjectParameterServiceImpl extends BaseServiceImpl implements Proj
return result; return result;
} }
ProjectParameter projectParameter = projectParameterMapper.queryByName(projectParameterName); // check if project parameter name exists
ProjectParameter projectParameter = projectParameterMapper.selectOne(new QueryWrapper<ProjectParameter>()
.lambda()
.eq(ProjectParameter::getProjectCode, projectCode)
.eq(ProjectParameter::getParamName, projectParameterName));
if (projectParameter != null) { if (projectParameter != null) {
log.warn("ProjectParameter {} already exists.", projectParameter.getParamName()); log.warn("ProjectParameter {} already exists.", projectParameter.getParamName());
putMsg(result, Status.PROJECT_PARAMETER_ALREADY_EXISTS, projectParameter.getParamName()); putMsg(result, Status.PROJECT_PARAMETER_ALREADY_EXISTS, projectParameter.getParamName());
@ -246,7 +251,7 @@ public class ProjectParameterServiceImpl extends BaseServiceImpl implements Proj
Page<ProjectParameter> page = new Page<>(pageNo, pageSize); Page<ProjectParameter> page = new Page<>(pageNo, pageSize);
IPage<ProjectParameter> iPage = IPage<ProjectParameter> iPage =
projectParameterMapper.queryProjectParameterListPaging(page, null, searchVal); projectParameterMapper.queryProjectParameterListPaging(page, projectCode, null, searchVal);
List<ProjectParameter> projectParameterList = iPage.getRecords(); List<ProjectParameter> projectParameterList = iPage.getRecords();

121
dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/controller/ProjectParameterControllerTest.java

@ -0,0 +1,121 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.api.controller;
import org.apache.dolphinscheduler.api.enums.Status;
import org.apache.dolphinscheduler.api.service.impl.ProjectParameterServiceImpl;
import org.apache.dolphinscheduler.api.utils.Result;
import org.apache.dolphinscheduler.common.enums.UserType;
import org.apache.dolphinscheduler.dao.entity.User;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.Mockito;
import org.mockito.junit.jupiter.MockitoExtension;
import org.mockito.junit.jupiter.MockitoSettings;
import org.mockito.quality.Strictness;
@ExtendWith(MockitoExtension.class)
@MockitoSettings(strictness = Strictness.LENIENT)
public class ProjectParameterControllerTest {
@InjectMocks
private ProjectParameterController projectParameterController;
@Mock
private ProjectParameterServiceImpl projectParameterService;
@Test
public void testCreateProjectParameter() {
User loginUser = getGeneralUser();
Mockito.when(projectParameterService.createProjectParameter(Mockito.any(), Mockito.anyLong(), Mockito.any(),
Mockito.any())).thenReturn(getSuccessResult());
Result result = projectParameterController.createProjectParameter(loginUser, 1, "key", "value");
Assertions.assertEquals(Status.SUCCESS.getCode(), result.getCode());
}
@Test
public void testUpdateProjectParameter() {
User loginUser = getGeneralUser();
Mockito.when(projectParameterService.updateProjectParameter(Mockito.any(), Mockito.anyLong(), Mockito.anyLong(),
Mockito.any(), Mockito.any())).thenReturn(getSuccessResult());
Result result = projectParameterController.updateProjectParameter(loginUser, 1, 1L, "key", "value");
Assertions.assertEquals(Status.SUCCESS.getCode(), result.getCode());
}
@Test
public void testDeleteProjectParametersByCode() {
User loginUser = getGeneralUser();
Mockito.when(projectParameterService.deleteProjectParametersByCode(Mockito.any(), Mockito.anyLong(),
Mockito.anyLong())).thenReturn(getSuccessResult());
Result result = projectParameterController.deleteProjectParametersByCode(loginUser, 1, 1);
Assertions.assertEquals(Status.SUCCESS.getCode(), result.getCode());
}
@Test
public void testBatchDeleteProjectParametersByCodes() {
User loginUser = getGeneralUser();
Mockito.when(projectParameterService.batchDeleteProjectParametersByCodes(Mockito.any(), Mockito.anyLong(),
Mockito.any())).thenReturn(getSuccessResult());
Result result = projectParameterController.batchDeleteProjectParametersByCodes(loginUser, 1, "1");
Assertions.assertEquals(Status.SUCCESS.getCode(), result.getCode());
}
@Test
public void testQueryProjectParameterListPaging() {
User loginUser = getGeneralUser();
Mockito.when(projectParameterService.queryProjectParameterListPaging(Mockito.any(), Mockito.anyLong(),
Mockito.anyInt(), Mockito.anyInt(), Mockito.any())).thenReturn(getSuccessResult());
Result result = projectParameterController.queryProjectParameterListPaging(loginUser, 1, "1", 1, 10);
Assertions.assertEquals(Status.SUCCESS.getCode(), result.getCode());
}
@Test
public void testQueryProjectParameterByCode() {
User loginUser = getGeneralUser();
Mockito.when(projectParameterService.queryProjectParameterByCode(Mockito.any(), Mockito.anyLong(),
Mockito.anyLong())).thenReturn(getSuccessResult());
Result result = projectParameterController.queryProjectParameterByCode(loginUser, 1, 1);
Assertions.assertEquals(Status.SUCCESS.getCode(), result.getCode());
}
private User getGeneralUser() {
User loginUser = new User();
loginUser.setUserType(UserType.GENERAL_USER);
loginUser.setUserName("userName");
loginUser.setId(1);
return loginUser;
}
private Result getSuccessResult() {
Result result = new Result();
result.setCode(Status.SUCCESS.getCode());
result.setMsg(Status.SUCCESS.getMsg());
return result;
}
}

166
dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/service/ProjectParameterServiceTest.java

@ -0,0 +1,166 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.api.service;
import org.apache.dolphinscheduler.api.enums.Status;
import org.apache.dolphinscheduler.api.service.impl.ProjectParameterServiceImpl;
import org.apache.dolphinscheduler.api.service.impl.ProjectServiceImpl;
import org.apache.dolphinscheduler.api.utils.Result;
import org.apache.dolphinscheduler.common.enums.UserType;
import org.apache.dolphinscheduler.dao.entity.Project;
import org.apache.dolphinscheduler.dao.entity.ProjectParameter;
import org.apache.dolphinscheduler.dao.entity.User;
import org.apache.dolphinscheduler.dao.mapper.ProjectMapper;
import org.apache.dolphinscheduler.dao.mapper.ProjectParameterMapper;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.Mockito;
import org.mockito.junit.jupiter.MockitoExtension;
import org.mockito.junit.jupiter.MockitoSettings;
import org.mockito.quality.Strictness;
@ExtendWith(MockitoExtension.class)
@MockitoSettings(strictness = Strictness.LENIENT)
public class ProjectParameterServiceTest {
@InjectMocks
private ProjectParameterServiceImpl projectParameterService;
@Mock
private ProjectMapper projectMapper;
@Mock
private ProjectParameterMapper projectParameterMapper;
@Mock
private ProjectServiceImpl projectService;
protected final static long projectCode = 1L;
@Test
public void testCreateProjectParameter() {
User loginUser = getGeneralUser();
// PROJECT_PARAMETER_ALREADY_EXISTS
Mockito.when(projectMapper.queryByCode(projectCode)).thenReturn(getProject(projectCode));
Mockito.when(projectParameterMapper.selectOne(Mockito.any())).thenReturn(getProjectParameter());
Mockito.when(projectService.hasProjectAndWritePerm(Mockito.any(), Mockito.any(), Mockito.any(Result.class)))
.thenReturn(true);
Result result = projectParameterService.createProjectParameter(loginUser, projectCode, "key", "value");
Assertions.assertEquals(Status.PROJECT_PARAMETER_ALREADY_EXISTS.getCode(), result.getCode());
// SUCCESS
Mockito.when(projectParameterMapper.selectOne(Mockito.any())).thenReturn(null);
Mockito.when(projectParameterMapper.insert(Mockito.any())).thenReturn(1);
result = projectParameterService.createProjectParameter(loginUser, projectCode, "key1", "value");
Assertions.assertEquals(Status.SUCCESS.getCode(), result.getCode());
}
@Test
public void testUpdateProjectParameter() {
User loginUser = getGeneralUser();
// PROJECT_PARAMETER_NOT_EXISTS
Mockito.when(projectMapper.queryByCode(projectCode)).thenReturn(getProject(projectCode));
Mockito.when(projectService.hasProjectAndWritePerm(Mockito.any(), Mockito.any(), Mockito.any(Result.class)))
.thenReturn(true);
Mockito.when(projectParameterMapper.queryByCode(Mockito.anyLong())).thenReturn(null);
Result result = projectParameterService.updateProjectParameter(loginUser, projectCode, 1, "key", "value");
Assertions.assertEquals(Status.PROJECT_PARAMETER_NOT_EXISTS.getCode(), result.getCode());
// PROJECT_PARAMETER_ALREADY_EXISTS
Mockito.when(projectParameterMapper.queryByCode(Mockito.anyLong())).thenReturn(getProjectParameter());
Mockito.when(projectParameterMapper.selectOne(Mockito.any())).thenReturn(getProjectParameter());
result = projectParameterService.updateProjectParameter(loginUser, projectCode, 1, "key", "value");
Assertions.assertEquals(Status.PROJECT_PARAMETER_ALREADY_EXISTS.getCode(), result.getCode());
// SUCCESS
Mockito.when(projectParameterMapper.selectOne(Mockito.any())).thenReturn(null);
Mockito.when(projectParameterMapper.updateById(Mockito.any())).thenReturn(1);
result = projectParameterService.updateProjectParameter(loginUser, projectCode, 1, "key1", "value");
Assertions.assertEquals(Status.SUCCESS.getCode(), result.getCode());
}
@Test
public void testDeleteProjectParametersByCode() {
User loginUser = getGeneralUser();
// PROJECT_PARAMETER_NOT_EXISTS
Mockito.when(projectMapper.queryByCode(projectCode)).thenReturn(getProject(projectCode));
Mockito.when(projectService.hasProjectAndWritePerm(Mockito.any(), Mockito.any(), Mockito.any(Result.class)))
.thenReturn(true);
Mockito.when(projectParameterMapper.queryByCode(Mockito.anyLong())).thenReturn(null);
Result result = projectParameterService.deleteProjectParametersByCode(loginUser, projectCode, 1);
Assertions.assertEquals(Status.PROJECT_PARAMETER_NOT_EXISTS.getCode(), result.getCode());
// SUCCESS
Mockito.when(projectParameterMapper.queryByCode(Mockito.anyLong())).thenReturn(getProjectParameter());
Mockito.when(projectParameterMapper.deleteById(Mockito.anyInt())).thenReturn(1);
result = projectParameterService.deleteProjectParametersByCode(loginUser, projectCode, 1);
Assertions.assertEquals(Status.SUCCESS.getCode(), result.getCode());
}
@Test
public void testQueryProjectParameterByCode() {
User loginUser = getGeneralUser();
// PROJECT_PARAMETER_NOT_EXISTS
Mockito.when(projectMapper.queryByCode(projectCode)).thenReturn(getProject(projectCode));
Mockito.when(projectService.hasProjectAndPerm(Mockito.any(), Mockito.any(), Mockito.any(Result.class),
Mockito.any())).thenReturn(true);
Mockito.when(projectParameterMapper.queryByCode(Mockito.anyLong())).thenReturn(null);
Result result = projectParameterService.queryProjectParameterByCode(loginUser, projectCode, 1);
Assertions.assertEquals(Status.PROJECT_PARAMETER_NOT_EXISTS.getCode(), result.getCode());
// SUCCESS
Mockito.when(projectParameterMapper.queryByCode(Mockito.anyLong())).thenReturn(getProjectParameter());
result = projectParameterService.queryProjectParameterByCode(loginUser, projectCode, 1);
Assertions.assertEquals(Status.SUCCESS.getCode(), result.getCode());
}
private User getGeneralUser() {
User loginUser = new User();
loginUser.setUserType(UserType.GENERAL_USER);
loginUser.setUserName("userName");
loginUser.setId(1);
return loginUser;
}
private Project getProject(long projectCode) {
Project project = new Project();
project.setCode(projectCode);
project.setId(1);
project.setName("test");
project.setUserId(1);
return project;
}
private ProjectParameter getProjectParameter() {
ProjectParameter projectParameter = new ProjectParameter();
projectParameter.setId(1);
projectParameter.setCode(1);
projectParameter.setProjectCode(1);
projectParameter.setParamName("key");
projectParameter.setParamValue("value");
return projectParameter;
}
}

3
dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/ProjectParameterMapper.java

@ -36,6 +36,9 @@ public interface ProjectParameterMapper extends BaseMapper<ProjectParameter> {
ProjectParameter queryByName(@Param("paramName") String paramName); ProjectParameter queryByName(@Param("paramName") String paramName);
IPage<ProjectParameter> queryProjectParameterListPaging(IPage<ProjectParameter> page, IPage<ProjectParameter> queryProjectParameterListPaging(IPage<ProjectParameter> page,
@Param("projectCode") long projectCode,
@Param("projectParameterIds") List<Integer> projectParameterIds, @Param("projectParameterIds") List<Integer> projectParameterIds,
@Param("searchName") String searchName); @Param("searchName") String searchName);
List<ProjectParameter> queryByProjectCode(@Param("projectCode") long projectCode);
} }

9
dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ProjectParameterMapper.xml

@ -53,7 +53,7 @@
select select
<include refid="baseSql"/> <include refid="baseSql"/>
from t_ds_project_parameter from t_ds_project_parameter
where 1=1 where project_code = #{projectCode}
<if test="projectParameterIds != null and projectParameterIds.size() > 0"> <if test="projectParameterIds != null and projectParameterIds.size() > 0">
and id in and id in
<foreach item="id" index="index" collection="projectParameterIds" open="(" separator="," close=")"> <foreach item="id" index="index" collection="projectParameterIds" open="(" separator="," close=")">
@ -68,4 +68,11 @@
order by id desc order by id desc
</select> </select>
<select id="queryByProjectCode" resultType="org.apache.dolphinscheduler.dao.entity.ProjectParameter">
select
<include refid="baseSql"/>
from t_ds_project_parameter
where project_code = #{projectCode}
</select>
</mapper> </mapper>

95
dolphinscheduler-dao/src/test/java/org/apache/dolphinscheduler/dao/mapper/ProjectParameterMapperTest.java

@ -0,0 +1,95 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.dao.mapper;
import org.apache.dolphinscheduler.dao.BaseDaoTest;
import org.apache.dolphinscheduler.dao.entity.ProjectParameter;
import java.util.Arrays;
import java.util.Date;
import java.util.List;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import com.baomidou.mybatisplus.core.metadata.IPage;
import com.baomidou.mybatisplus.extension.plugins.pagination.Page;
public class ProjectParameterMapperTest extends BaseDaoTest {
@Autowired
private ProjectParameterMapper projectParameterMapper;
private ProjectParameter insertOne(long code, String name, long projectCode) {
ProjectParameter projectParameter = new ProjectParameter();
projectParameter.setCode(code);
projectParameter.setParamName(name);
projectParameter.setProjectCode(projectCode);
projectParameter.setParamValue("value");
projectParameter.setCreateTime(new Date());
projectParameter.setUpdateTime(new Date());
projectParameter.setUserId(1);
projectParameterMapper.insert(projectParameter);
return projectParameter;
}
@Test
public void testUpdate() {
ProjectParameter projectParameter = insertOne(1, "name", 1);
projectParameter.setUpdateTime(new Date());
int update = projectParameterMapper.updateById(projectParameter);
Assertions.assertEquals(1, update);
}
@Test
public void testQueryByCode() {
ProjectParameter projectParameter = insertOne(1, "name", 1);
ProjectParameter res = projectParameterMapper.queryByCode(projectParameter.getCode());
Assertions.assertEquals(projectParameter.getCode(), res.getCode());
}
@Test
public void testQueryByCodes() {
insertOne(1, "name1", 1);
insertOne(2, "name2", 1);
List<ProjectParameter> res = projectParameterMapper.queryByCodes(Arrays.asList(1L, 2L));
Assertions.assertEquals(2, res.size());
}
@Test
public void testQueryByProjectCode() {
insertOne(1, "name1", 1);
insertOne(2, "name2", 2);
List<ProjectParameter> res = projectParameterMapper.queryByProjectCode(1);
Assertions.assertEquals(1, res.size());
}
@Test
public void testQueryProjectParameterListPaging() {
insertOne(1, "name1", 1);
insertOne(2, "name2", 2);
Page<ProjectParameter> page = new Page(1, 3);
IPage<ProjectParameter> res = projectParameterMapper.queryProjectParameterListPaging(page, 1, null, null);
Assertions.assertEquals(1, res.getRecords().size());
}
}

2
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/expand/CuringParamsService.java

@ -86,4 +86,6 @@ public interface CuringParamsService {
* @return * @return
*/ */
Map<String, Property> preBuildBusinessParams(ProcessInstance processInstance); Map<String, Property> preBuildBusinessParams(ProcessInstance processInstance);
Map<String, Property> getProjectParameterMap(long projectCode);
} }

63
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/expand/CuringGlobalParams.java → dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/expand/CuringParamsServiceImpl.java

@ -34,7 +34,11 @@ import org.apache.dolphinscheduler.common.utils.DateUtils;
import org.apache.dolphinscheduler.common.utils.JSONUtils; import org.apache.dolphinscheduler.common.utils.JSONUtils;
import org.apache.dolphinscheduler.common.utils.placeholder.BusinessTimeUtils; import org.apache.dolphinscheduler.common.utils.placeholder.BusinessTimeUtils;
import org.apache.dolphinscheduler.dao.entity.ProcessInstance; import org.apache.dolphinscheduler.dao.entity.ProcessInstance;
import org.apache.dolphinscheduler.dao.entity.ProjectParameter;
import org.apache.dolphinscheduler.dao.entity.TaskInstance; import org.apache.dolphinscheduler.dao.entity.TaskInstance;
import org.apache.dolphinscheduler.dao.mapper.ProjectParameterMapper;
import org.apache.dolphinscheduler.plugin.task.api.enums.DataType;
import org.apache.dolphinscheduler.plugin.task.api.enums.Direct;
import org.apache.dolphinscheduler.plugin.task.api.model.Property; import org.apache.dolphinscheduler.plugin.task.api.model.Property;
import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters; import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
import org.apache.dolphinscheduler.plugin.task.api.utils.MapUtils; import org.apache.dolphinscheduler.plugin.task.api.utils.MapUtils;
@ -56,11 +60,14 @@ import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component; import org.springframework.stereotype.Component;
@Component @Component
public class CuringGlobalParams implements CuringParamsService { public class CuringParamsServiceImpl implements CuringParamsService {
@Autowired @Autowired
private TimePlaceholderResolverExpandService timePlaceholderResolverExpandService; private TimePlaceholderResolverExpandService timePlaceholderResolverExpandService;
@Autowired
private ProjectParameterMapper projectParameterMapper;
@Override @Override
public String convertParameterPlaceholders(String val, Map<String, String> allParamMap) { public String convertParameterPlaceholders(String val, Map<String, String> allParamMap) {
return ParameterUtils.convertParameterPlaceholders(val, allParamMap); return ParameterUtils.convertParameterPlaceholders(val, allParamMap);
@ -140,6 +147,8 @@ public class CuringGlobalParams implements CuringParamsService {
public Map<String, Property> paramParsingPreparation(@NonNull TaskInstance taskInstance, public Map<String, Property> paramParsingPreparation(@NonNull TaskInstance taskInstance,
@NonNull AbstractParameters parameters, @NonNull AbstractParameters parameters,
@NonNull ProcessInstance processInstance) { @NonNull ProcessInstance processInstance) {
Map<String, Property> prepareParamsMap = new HashMap<>();
// assign value to definedParams here // assign value to definedParams here
Map<String, String> globalParamsMap = setGlobalParamsMap(processInstance); Map<String, String> globalParamsMap = setGlobalParamsMap(processInstance);
Map<String, Property> globalParams = ParameterUtils.getUserDefParamsMap(globalParamsMap); Map<String, Property> globalParams = ParameterUtils.getUserDefParamsMap(globalParamsMap);
@ -158,22 +167,36 @@ public class CuringGlobalParams implements CuringParamsService {
String timeZone = cmdParam.get(Constants.SCHEDULE_TIMEZONE); String timeZone = cmdParam.get(Constants.SCHEDULE_TIMEZONE);
// built-in params // built-in params
Map<String, String> params = setBuiltInParamsMap(taskInstance, timeZone); Map<String, String> builtInParams = setBuiltInParamsMap(taskInstance, timeZone);
if (MapUtils.isNotEmpty(params)) { // project-level params
globalParams.putAll(ParameterUtils.getUserDefParamsMap(params)); Map<String, Property> projectParams = getProjectParameterMap(taskInstance.getProjectCode());
if (MapUtils.isNotEmpty(builtInParams)) {
prepareParamsMap.putAll(ParameterUtils.getUserDefParamsMap(builtInParams));
}
if (MapUtils.isNotEmpty(projectParams)) {
prepareParamsMap.putAll(projectParams);
}
if (MapUtils.isNotEmpty(globalParams)) {
prepareParamsMap.putAll(globalParams);
} }
if (MapUtils.isNotEmpty(varParams)) { if (MapUtils.isNotEmpty(varParams)) {
globalParams.putAll(varParams); prepareParamsMap.putAll(varParams);
} }
if (MapUtils.isNotEmpty(localParams)) { if (MapUtils.isNotEmpty(localParams)) {
globalParams.putAll(localParams); prepareParamsMap.putAll(localParams);
} }
if (MapUtils.isNotEmpty(cmdParam)) { if (MapUtils.isNotEmpty(cmdParam)) {
globalParams.putAll(ParameterUtils.getUserDefParamsMap(cmdParam)); prepareParamsMap.putAll(ParameterUtils.getUserDefParamsMap(cmdParam));
} }
Iterator<Map.Entry<String, Property>> iter = globalParams.entrySet().iterator();
Iterator<Map.Entry<String, Property>> iter = prepareParamsMap.entrySet().iterator();
while (iter.hasNext()) { while (iter.hasNext()) {
Map.Entry<String, Property> en = iter.next(); Map.Entry<String, Property> en = iter.next();
Property property = en.getValue(); Property property = en.getValue();
@ -193,15 +216,13 @@ public class CuringGlobalParams implements CuringParamsService {
property.setValue(val); property.setValue(val);
} }
} }
if (MapUtils.isEmpty(globalParams)) {
globalParams = new HashMap<>();
}
// put schedule time param to params map // put schedule time param to params map
Map<String, Property> paramsMap = preBuildBusinessParams(processInstance); Map<String, Property> paramsMap = preBuildBusinessParams(processInstance);
if (MapUtils.isNotEmpty(paramsMap)) { if (MapUtils.isNotEmpty(paramsMap)) {
globalParams.putAll(paramsMap); prepareParamsMap.putAll(paramsMap);
} }
return globalParams; return prepareParamsMap;
} }
/** /**
@ -258,4 +279,20 @@ public class CuringGlobalParams implements CuringParamsService {
} }
return paramsMap; return paramsMap;
} }
@Override
public Map<String, Property> getProjectParameterMap(long projectCode) {
Map<String, Property> result = new HashMap<>(16);
List<ProjectParameter> projectParameterList = projectParameterMapper.queryByProjectCode(projectCode);
projectParameterList.forEach(projectParameter -> {
Property property = new Property(projectParameter.getParamName(),
Direct.IN,
DataType.VARCHAR,
projectParameter.getParamValue());
result.put(projectParameter.getParamName(), property);
});
return result;
}
} }

12
dolphinscheduler-service/src/test/java/org/apache/dolphinscheduler/service/expand/CuringGlobalParamsServiceTest.java → dolphinscheduler-service/src/test/java/org/apache/dolphinscheduler/service/expand/CuringParamsServiceTest.java

@ -25,6 +25,7 @@ import org.apache.dolphinscheduler.dao.entity.ProcessDefinition;
import org.apache.dolphinscheduler.dao.entity.ProcessInstance; import org.apache.dolphinscheduler.dao.entity.ProcessInstance;
import org.apache.dolphinscheduler.dao.entity.TaskDefinition; import org.apache.dolphinscheduler.dao.entity.TaskDefinition;
import org.apache.dolphinscheduler.dao.entity.TaskInstance; import org.apache.dolphinscheduler.dao.entity.TaskInstance;
import org.apache.dolphinscheduler.dao.mapper.ProjectParameterMapper;
import org.apache.dolphinscheduler.plugin.task.api.TaskConstants; import org.apache.dolphinscheduler.plugin.task.api.TaskConstants;
import org.apache.dolphinscheduler.plugin.task.api.enums.DataType; import org.apache.dolphinscheduler.plugin.task.api.enums.DataType;
import org.apache.dolphinscheduler.plugin.task.api.enums.Direct; import org.apache.dolphinscheduler.plugin.task.api.enums.Direct;
@ -33,6 +34,7 @@ import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters
import org.apache.dolphinscheduler.plugin.task.api.parameters.SubProcessParameters; import org.apache.dolphinscheduler.plugin.task.api.parameters.SubProcessParameters;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.Collections;
import java.util.Date; import java.util.Date;
import java.util.HashMap; import java.util.HashMap;
import java.util.List; import java.util.List;
@ -50,7 +52,7 @@ import org.mockito.junit.jupiter.MockitoExtension;
import com.google.common.collect.Lists; import com.google.common.collect.Lists;
@ExtendWith(MockitoExtension.class) @ExtendWith(MockitoExtension.class)
public class CuringGlobalParamsServiceTest { public class CuringParamsServiceTest {
private static final String placeHolderName = "$[yyyy-MM-dd-1]"; private static final String placeHolderName = "$[yyyy-MM-dd-1]";
@ -58,11 +60,14 @@ public class CuringGlobalParamsServiceTest {
private CuringParamsService curingGlobalParamsService; private CuringParamsService curingGlobalParamsService;
@InjectMocks @InjectMocks
private CuringGlobalParams dolphinSchedulerCuringGlobalParams; private CuringParamsServiceImpl dolphinSchedulerCuringGlobalParams;
@Mock @Mock
private TimePlaceholderResolverExpandService timePlaceholderResolverExpandService; private TimePlaceholderResolverExpandService timePlaceholderResolverExpandService;
@Mock
private ProjectParameterMapper projectParameterMapper;
@InjectMocks @InjectMocks
private TimePlaceholderResolverExpandServiceImpl timePlaceholderResolverExpandServiceImpl; private TimePlaceholderResolverExpandServiceImpl timePlaceholderResolverExpandServiceImpl;
@ -201,9 +206,12 @@ public class CuringGlobalParamsServiceTest {
taskInstance.setProcessDefine(processDefinition); taskInstance.setProcessDefine(processDefinition);
taskInstance.setProcessInstance(processInstance); taskInstance.setProcessInstance(processInstance);
taskInstance.setTaskDefine(taskDefinition); taskInstance.setTaskDefine(taskDefinition);
taskInstance.setProjectCode(3000001l);
AbstractParameters parameters = new SubProcessParameters(); AbstractParameters parameters = new SubProcessParameters();
Mockito.when(projectParameterMapper.queryByProjectCode(Mockito.anyLong())).thenReturn(Collections.emptyList());
Map<String, Property> propertyMap = Map<String, Property> propertyMap =
dolphinSchedulerCuringGlobalParams.paramParsingPreparation(taskInstance, parameters, processInstance); dolphinSchedulerCuringGlobalParams.paramParsingPreparation(taskInstance, parameters, processInstance);
Assertions.assertNotNull(propertyMap); Assertions.assertNotNull(propertyMap);
Loading…
Cancel
Save