diff --git a/docs/docs/en/guide/parameter/global.md b/docs/docs/en/guide/parameter/global.md index 86cfd69929..46e9fb151d 100644 --- a/docs/docs/en/guide/parameter/global.md +++ b/docs/docs/en/guide/parameter/global.md @@ -22,7 +22,7 @@ Create a shell task and enter `echo ${dt}` in the script content. In this case, ### Save the workflow and set global parameters -Set global parameter: On the workflow definition page, click the plus sign to the right of "Set Global", after filling in the variable name and value, save it. +Set global parameter: On the workflow definition page, click the plus sign to the right of "Set Global", after filling in the variable name and value, select the appropriate parameter value type, save it. ![global-parameter02](../../../../img/new_ui/dev/parameter/global_parameter02.png) diff --git a/docs/docs/en/guide/parameter/project-parameter.md b/docs/docs/en/guide/parameter/project-parameter.md index 235cd23512..f9509114d7 100644 --- a/docs/docs/en/guide/parameter/project-parameter.md +++ b/docs/docs/en/guide/parameter/project-parameter.md @@ -8,7 +8,7 @@ Project-level parameters are valid for all task nodes under the entire project. ### Define project-level parameters -On the project page, click Project Parameters and Create Parameters, and fill in the parameter name and parameter value. As shown below: +On the project page, click Project Parameters and Create Parameters, and fill in the parameter name and parameter value, select the appropriate parameter value type. As shown below: ![project-parameter01](../../../../img/new_ui/dev/parameter/project_parameter01.png) diff --git a/docs/docs/en/guide/parameter/startup-parameter.md b/docs/docs/en/guide/parameter/startup-parameter.md index bf3acc2034..11cf5a9f63 100644 --- a/docs/docs/en/guide/parameter/startup-parameter.md +++ b/docs/docs/en/guide/parameter/startup-parameter.md @@ -6,7 +6,7 @@ Parameters are valid for all task nodes of the entire workflow. It can be config ## Usage -Usage of startup parameters is: at the task launch page, click the '+' below the 'Startup Parameter' and fill in the key and value to save. The workflow will add them into global parameters. +Usage of startup parameters is: at the task launch page, click the '+' below the 'Startup Parameter' and fill in the key and value, select the appropriate parameter value type then to save. The workflow will add them into global parameters. ## Example diff --git a/docs/docs/zh/guide/parameter/global.md b/docs/docs/zh/guide/parameter/global.md index cd26a291b0..3b8b307f91 100644 --- a/docs/docs/zh/guide/parameter/global.md +++ b/docs/docs/zh/guide/parameter/global.md @@ -20,7 +20,7 @@ ### 保存工作流，并设置全局参数 -全局参数配置方式如下：在工作流定义页面，点击“设置全局”右边的加号，填写对应的变量名称和对应的值，保存即可。如下图所示： +全局参数配置方式如下：在工作流定义页面，点击“设置全局”右边的加号，填写对应的变量名称和对应的值，选择相应的参数值类型，保存即可。如下图所示： ![global-parameter02](../../../../img/new_ui/dev/parameter/global_parameter02.png) diff --git a/docs/docs/zh/guide/parameter/project-parameter.md b/docs/docs/zh/guide/parameter/project-parameter.md index df26e7a7df..8692b198a8 100644 --- a/docs/docs/zh/guide/parameter/project-parameter.md +++ b/docs/docs/zh/guide/parameter/project-parameter.md @@ -8,7 +8,7 @@ ### 定义项目级别参数 -在项目管理页面，点击项目级别参数，点击创建项目级别参数，填写参数名称和参数值。如下图所示： +在项目管理页面，点击项目级别参数，点击创建项目级别参数，填写参数名称和参数值，选择相应的参数值类型。如下图所示： ![project-parameter01](../../../../img/new_ui/dev/parameter/project_parameter01.png) diff --git a/docs/docs/zh/guide/parameter/startup-parameter.md b/docs/docs/zh/guide/parameter/startup-parameter.md index 7da65657a7..38e15787d0 100644 --- a/docs/docs/zh/guide/parameter/startup-parameter.md +++ b/docs/docs/zh/guide/parameter/startup-parameter.md @@ -6,7 +6,7 @@ ## 使用方式 -启动参数配置方式如下：在启动前参数设置界面，点击“启动参数“下面的加号，填写对应的参数名称和对应的值，点击确定，工作流会将启动参数加入全局参数中。 +启动参数配置方式如下：在启动前参数设置界面，点击“启动参数“下面的加号，填写对应的参数名称和对应的值，选择相应的参数值类型，点击确定，工作流会将启动参数加入全局参数中。 ## 任务样例 diff --git a/docs/img/new_ui/dev/parameter/context-subprocess02.png b/docs/img/new_ui/dev/parameter/context-subprocess02.png index 8c1acf5f11..7d7cd850ea 100644 Binary files a/docs/img/new_ui/dev/parameter/context-subprocess02.png and b/docs/img/new_ui/dev/parameter/context-subprocess02.png differ diff --git a/docs/img/new_ui/dev/parameter/context_parameter04.png b/docs/img/new_ui/dev/parameter/context_parameter04.png index b1dfd8b3cd..e307e607e7 100644 Binary files a/docs/img/new_ui/dev/parameter/context_parameter04.png and b/docs/img/new_ui/dev/parameter/context_parameter04.png differ diff --git a/docs/img/new_ui/dev/parameter/global_parameter02.png b/docs/img/new_ui/dev/parameter/global_parameter02.png index 1eeaf4b43a..3d6d535311 100644 Binary files a/docs/img/new_ui/dev/parameter/global_parameter02.png and b/docs/img/new_ui/dev/parameter/global_parameter02.png differ diff --git a/docs/img/new_ui/dev/parameter/project_parameter01.png b/docs/img/new_ui/dev/parameter/project_parameter01.png index 48ea149b40..5f6a483d4c 100644 Binary files a/docs/img/new_ui/dev/parameter/project_parameter01.png and b/docs/img/new_ui/dev/parameter/project_parameter01.png differ diff --git a/docs/img/new_ui/dev/parameter/startup_parameter02.png b/docs/img/new_ui/dev/parameter/startup_parameter02.png index cbeb96b658..d1819ad78e 100644 Binary files a/docs/img/new_ui/dev/parameter/startup_parameter02.png and b/docs/img/new_ui/dev/parameter/startup_parameter02.png differ diff --git a/docs/img/new_ui/dev/parameter/startup_parameter04.png b/docs/img/new_ui/dev/parameter/startup_parameter04.png index e32afb4b31..b5799b24a5 100644 Binary files a/docs/img/new_ui/dev/parameter/startup_parameter04.png and b/docs/img/new_ui/dev/parameter/startup_parameter04.png differ diff --git a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ExecutorController.java b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ExecutorController.java index a2a300b946..8c86de7fe6 100644 --- a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ExecutorController.java +++ b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ExecutorController.java @@ -44,6 +44,8 @@ import org.apache.dolphinscheduler.common.enums.WarningType; import org.apache.dolphinscheduler.common.utils.JSONUtils; import org.apache.dolphinscheduler.dao.entity.User; import org.apache.dolphinscheduler.extract.master.dto.WorkflowExecuteDto; +import org.apache.dolphinscheduler.plugin.task.api.model.Property; +import org.apache.dolphinscheduler.plugin.task.api.utils.PropertyUtils; import org.apache.commons.lang3.StringUtils; @@ -163,10 +165,8 @@ public class ExecutorController extends BaseController { if (timeout == null) { timeout = Constants.MAX_TASK_TIMEOUT; } - Map startParamMap = null; - if (startParams != null) { - startParamMap = JSONUtils.toMap(startParams); - } + + List startParamList = PropertyUtils.startParamsTransformPropertyList(startParams); if (complementDependentMode == null) { complementDependentMode = ComplementDependentMode.OFF_MODE; @@ -175,7 +175,7 @@ public class ExecutorController extends BaseController { Map result = execService.execProcessInstance(loginUser, projectCode, processDefinitionCode, scheduleTime, execType, failureStrategy, startNodeList, taskDependType, warningType, warningGroupId, runMode, processInstancePriority, - workerGroup, tenantCode, environmentCode, timeout, startParamMap, expectedParallelismNumber, dryRun, + workerGroup, tenantCode, environmentCode, timeout, startParamList, expectedParallelismNumber, dryRun, testFlag, complementDependentMode, version, allLevelDependent, executionOrder); return returnDataList(result); @@ -262,10 +262,7 @@ public class ExecutorController extends BaseController { timeout = Constants.MAX_TASK_TIMEOUT; } - Map startParamMap = null; - if (startParams != null) { - startParamMap = JSONUtils.toMap(startParams); - } + List startParamList = PropertyUtils.startParamsTransformPropertyList(startParams); if (complementDependentMode == null) { log.debug("Parameter complementDependentMode set to {} due to null.", ComplementDependentMode.OFF_MODE); @@ -283,7 +280,8 @@ public class ExecutorController extends BaseController { result = execService.execProcessInstance(loginUser, projectCode, processDefinitionCode, scheduleTime, execType, failureStrategy, startNodeList, taskDependType, warningType, warningGroupId, runMode, processInstancePriority, - workerGroup, tenantCode, environmentCode, timeout, startParamMap, expectedParallelismNumber, dryRun, + workerGroup, tenantCode, environmentCode, timeout, startParamList, expectedParallelismNumber, + dryRun, testFlag, complementDependentMode, null, allLevelDependent, executionOrder); diff --git a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ProjectParameterController.java b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ProjectParameterController.java index 706c7e939a..efec970e6f 100644 --- a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ProjectParameterController.java +++ b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ProjectParameterController.java @@ -61,7 +61,8 @@ public class ProjectParameterController extends BaseController { @Operation(summary = "createProjectParameter", description = "CREATE_PROJECT_PARAMETER_NOTES") @Parameters({ @Parameter(name = "projectParameterName", description = "PROJECT_PARAMETER_NAME", schema = @Schema(implementation = String.class)), - @Parameter(name = "projectParameterValue", description = "PROJECT_PARAMETER_VALUE", schema = @Schema(implementation = String.class)) + @Parameter(name = "projectParameterValue", description = "PROJECT_PARAMETER_VALUE", schema = @Schema(implementation = String.class)), + @Parameter(name = "projectParameterDataType", description = "PROJECT_PARAMETER_DATA_TYPE", schema = @Schema(implementation = String.class)) }) @PostMapping() @ResponseStatus(HttpStatus.CREATED) @@ -69,9 +70,10 @@ public class ProjectParameterController extends BaseController { public Result createProjectParameter(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser, @Parameter(name = "projectCode", description = "PROJECT_CODE", required = true) @PathVariable long projectCode, @RequestParam("projectParameterName") String projectParameterName, - @RequestParam(value = "projectParameterValue") String projectParameterValue) { + @RequestParam(value = "projectParameterValue") String projectParameterValue, + @RequestParam(value = "projectParameterDataType", defaultValue = "VARCHAR") String projectParameterDataType) { return projectParameterService.createProjectParameter(loginUser, projectCode, projectParameterName, - projectParameterValue); + projectParameterValue, projectParameterDataType); } @Operation(summary = "updateProjectParameter", description = "UPDATE_PROJECT_PARAMETER_NOTES") @@ -79,6 +81,7 @@ public class ProjectParameterController extends BaseController { @Parameter(name = "code", description = "PROJECT_PARAMETER_CODE", schema = @Schema(implementation = long.class, example = "123456")), @Parameter(name = "projectParameterName", description = "PROJECT_PARAMETER_NAME", schema = @Schema(implementation = String.class)), @Parameter(name = "projectParameterValue", description = "PROJECT_PARAMETER_VALUE", schema = @Schema(implementation = String.class)), + @Parameter(name = "projectParameterDataType", description = "PROJECT_PARAMETER_DATA_TYPE", schema = @Schema(implementation = String.class)) }) @PutMapping(value = "/{code}") @ResponseStatus(HttpStatus.OK) @@ -87,9 +90,10 @@ public class ProjectParameterController extends BaseController { @Parameter(name = "projectCode", description = "PROJECT_CODE", required = true) @PathVariable long projectCode, @PathVariable("code") Long code, @RequestParam("projectParameterName") String projectParameterName, - @RequestParam(value = "projectParameterValue") String projectParameterValue) { + @RequestParam(value = "projectParameterValue") String projectParameterValue, + @RequestParam(value = "projectParameterDataType") String projectParameterDataType) { return projectParameterService.updateProjectParameter(loginUser, projectCode, code, projectParameterName, - projectParameterValue); + projectParameterValue, projectParameterDataType); } @Operation(summary = "deleteProjectParametersByCode", description = "DELETE_PROJECT_PARAMETER_NOTES") @@ -133,13 +137,14 @@ public class ProjectParameterController extends BaseController { @Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser, @Parameter(name = "projectCode", description = "PROJECT_CODE", required = true) @PathVariable long projectCode, @RequestParam(value = "searchVal", required = false) String searchVal, + @RequestParam(value = "projectParameterDataType", required = false) String projectParameterDataType, @RequestParam("pageNo") Integer pageNo, @RequestParam("pageSize") Integer pageSize) { checkPageParams(pageNo, pageSize); searchVal = ParameterUtils.handleEscapes(searchVal); return projectParameterService.queryProjectParameterListPaging(loginUser, projectCode, pageSize, pageNo, - searchVal); + searchVal, projectParameterDataType); } @Operation(summary = "queryProjectParameterByCode", description = "QUERY_PROJECT_PARAMETER_NOTES") diff --git a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/ExecutorService.java b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/ExecutorService.java index 3166d3e718..55ca697412 100644 --- a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/ExecutorService.java +++ b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/ExecutorService.java @@ -30,7 +30,9 @@ import org.apache.dolphinscheduler.common.enums.WarningType; import org.apache.dolphinscheduler.dao.entity.ProcessDefinition; import org.apache.dolphinscheduler.dao.entity.User; import org.apache.dolphinscheduler.extract.master.dto.WorkflowExecuteDto; +import org.apache.dolphinscheduler.plugin.task.api.model.Property; +import java.util.List; import java.util.Map; /** @@ -57,7 +59,7 @@ public interface ExecutorService { * @param environmentCode environment code * @param runMode run mode * @param timeout timeout - * @param startParams the global param values which pass to new process instance + * @param startParamList the global param values which pass to new process instance * @param expectedParallelismNumber the expected parallelism number when execute complement in parallel mode * @param executionOrder the execution order when complementing data * @return execute process instance code @@ -71,7 +73,7 @@ public interface ExecutorService { Priority processInstancePriority, String workerGroup, String tenantCode, Long environmentCode, Integer timeout, - Map startParams, Integer expectedParallelismNumber, + List startParamList, Integer expectedParallelismNumber, int dryRun, int testFlag, ComplementDependentMode complementDependentMode, Integer version, boolean allLevelDependent, ExecutionOrder executionOrder); diff --git a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/ProjectParameterService.java b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/ProjectParameterService.java index ef984dfac4..af2b047488 100644 --- a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/ProjectParameterService.java +++ b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/ProjectParameterService.java @@ -23,17 +23,17 @@ import org.apache.dolphinscheduler.dao.entity.User; public interface ProjectParameterService { Result createProjectParameter(User loginUser, long projectCode, String projectParameterName, - String projectParameterValue); + String projectParameterValue, String projectParameterDataType); Result updateProjectParameter(User loginUser, long projectCode, long code, String projectParameterName, - String projectParameterValue); + String projectParameterValue, String projectParameterDataType); Result deleteProjectParametersByCode(User loginUser, long projectCode, long code); Result batchDeleteProjectParametersByCodes(User loginUser, long projectCode, String codes); Result queryProjectParameterListPaging(User loginUser, long projectCode, Integer pageSize, Integer pageNo, - String searchVal); + String searchVal, String projectParameterDataType); Result queryProjectParameterByCode(User loginUser, long projectCode, long code); } diff --git a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java index dafef79615..3e613b11fa 100644 --- a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java +++ b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java @@ -90,6 +90,7 @@ import org.apache.dolphinscheduler.extract.master.transportor.StreamingTaskTrigg import org.apache.dolphinscheduler.extract.master.transportor.StreamingTaskTriggerResponse; import org.apache.dolphinscheduler.extract.master.transportor.WorkflowInstanceStateChangeEvent; import org.apache.dolphinscheduler.plugin.task.api.TaskConstants; +import org.apache.dolphinscheduler.plugin.task.api.model.Property; import org.apache.dolphinscheduler.registry.api.enums.RegistryNodeType; import org.apache.dolphinscheduler.service.command.CommandService; import org.apache.dolphinscheduler.service.cron.CronUtils; @@ -204,7 +205,7 @@ public class ExecutorServiceImpl extends BaseServiceImpl implements ExecutorServ * @param environmentCode environment code * @param runMode run mode * @param timeout timeout - * @param startParams the global param values which pass to new process instance + * @param startParamList the global param values which pass to new process instance * @param expectedParallelismNumber the expected parallelism number when execute complement in parallel mode * @param testFlag testFlag * @param executionOrder the execution order when complementing data @@ -220,7 +221,7 @@ public class ExecutorServiceImpl extends BaseServiceImpl implements ExecutorServ Priority processInstancePriority, String workerGroup, String tenantCode, Long environmentCode, Integer timeout, - Map startParams, Integer expectedParallelismNumber, + List startParamList, Integer expectedParallelismNumber, int dryRun, int testFlag, ComplementDependentMode complementDependentMode, Integer version, boolean allLevelDependent, ExecutionOrder executionOrder) { @@ -270,7 +271,7 @@ public class ExecutorServiceImpl extends BaseServiceImpl implements ExecutorServ startNodeList, cronTime, warningType, loginUser.getId(), warningGroupId, runMode, processInstancePriority, workerGroup, tenantCode, - environmentCode, startParams, expectedParallelismNumber, dryRun, testFlag, + environmentCode, startParamList, expectedParallelismNumber, dryRun, testFlag, complementDependentMode, allLevelDependent, executionOrder); if (create > 0) { @@ -732,7 +733,7 @@ public class ExecutorServiceImpl extends BaseServiceImpl implements ExecutorServ WarningType warningType, int executorId, Integer warningGroupId, RunMode runMode, Priority processInstancePriority, String workerGroup, String tenantCode, Long environmentCode, - Map startParams, Integer expectedParallelismNumber, int dryRun, + List startParamList, Integer expectedParallelismNumber, int dryRun, int testFlag, ComplementDependentMode complementDependentMode, boolean allLevelDependent, ExecutionOrder executionOrder) { @@ -761,8 +762,8 @@ public class ExecutorServiceImpl extends BaseServiceImpl implements ExecutorServ if (warningType != null) { command.setWarningType(warningType); } - if (startParams != null && startParams.size() > 0) { - cmdParam.put(CMD_PARAM_START_PARAMS, JSONUtils.toJsonString(startParams)); + if (CollectionUtils.isNotEmpty(startParamList)) { + cmdParam.put(CMD_PARAM_START_PARAMS, JSONUtils.toJsonString(startParamList)); } command.setCommandParam(JSONUtils.toJsonString(cmdParam)); command.setExecutorId(executorId); diff --git a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ProjectParameterServiceImpl.java b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ProjectParameterServiceImpl.java index 17d1d04712..e23101da36 100644 --- a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ProjectParameterServiceImpl.java +++ b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ProjectParameterServiceImpl.java @@ -68,7 +68,7 @@ public class ProjectParameterServiceImpl extends BaseServiceImpl implements Proj @Override @Transactional public Result createProjectParameter(User loginUser, long projectCode, String projectParameterName, - String projectParameterValue) { + String projectParameterValue, String projectParameterDataType) { Result result = new Result(); // check if user have write perm for project @@ -97,6 +97,7 @@ public class ProjectParameterServiceImpl extends BaseServiceImpl implements Proj .builder() .paramName(projectParameterName) .paramValue(projectParameterValue) + .paramDataType(projectParameterDataType) .code(CodeGenerateUtils.genCode()) .projectCode(projectCode) .userId(loginUser.getId()) @@ -122,7 +123,7 @@ public class ProjectParameterServiceImpl extends BaseServiceImpl implements Proj @Override public Result updateProjectParameter(User loginUser, long projectCode, long code, String projectParameterName, - String projectParameterValue) { + String projectParameterValue, String projectParameterDataType) { Result result = new Result(); // check if user have write perm for project @@ -155,6 +156,7 @@ public class ProjectParameterServiceImpl extends BaseServiceImpl implements Proj projectParameter.setParamName(projectParameterName); projectParameter.setParamValue(projectParameterValue); + projectParameter.setParamDataType(projectParameterDataType); projectParameter.setUpdateTime(new Date()); projectParameter.setOperator(loginUser.getId()); @@ -236,7 +238,7 @@ public class ProjectParameterServiceImpl extends BaseServiceImpl implements Proj @Override public Result queryProjectParameterListPaging(User loginUser, long projectCode, Integer pageSize, Integer pageNo, - String searchVal) { + String searchVal, String projectParameterDataType) { Result result = new Result(); Project project = projectMapper.queryByCode(projectCode); @@ -249,7 +251,8 @@ public class ProjectParameterServiceImpl extends BaseServiceImpl implements Proj Page page = new Page<>(pageNo, pageSize); IPage iPage = - projectParameterMapper.queryProjectParameterListPaging(page, projectCode, null, searchVal); + projectParameterMapper.queryProjectParameterListPaging(page, projectCode, null, searchVal, + projectParameterDataType); List projectParameterList = iPage.getRecords(); diff --git a/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/controller/ExecuteFunctionControllerTest.java b/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/controller/ExecuteFunctionControllerTest.java index 2c1a62ccc6..5b77acc099 100644 --- a/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/controller/ExecuteFunctionControllerTest.java +++ b/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/controller/ExecuteFunctionControllerTest.java @@ -38,8 +38,13 @@ import org.apache.dolphinscheduler.common.enums.RunMode; import org.apache.dolphinscheduler.common.enums.TaskDependType; import org.apache.dolphinscheduler.common.enums.WarningType; import org.apache.dolphinscheduler.dao.entity.User; +import org.apache.dolphinscheduler.plugin.task.api.enums.DataType; +import org.apache.dolphinscheduler.plugin.task.api.enums.Direct; +import org.apache.dolphinscheduler.plugin.task.api.model.Property; +import java.util.Collections; import java.util.HashMap; +import java.util.List; import java.util.Map; import org.junit.jupiter.api.Test; @@ -75,7 +80,8 @@ public class ExecuteFunctionControllerTest extends AbstractControllerTest { final String tenantCode = "root"; final Long environmentCode = 4L; final Integer timeout = 5; - final ImmutableMap startParams = ImmutableMap.of("start", "params"); + final List startParams = + Collections.singletonList(new Property("start", Direct.IN, DataType.VARCHAR, "params")); final Integer expectedParallelismNumber = 6; final int dryRun = 7; final int testFlag = 0; diff --git a/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/controller/ProjectParameterControllerTest.java b/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/controller/ProjectParameterControllerTest.java index f1bd20bb23..ea8bbf67fd 100644 --- a/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/controller/ProjectParameterControllerTest.java +++ b/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/controller/ProjectParameterControllerTest.java @@ -22,6 +22,7 @@ import org.apache.dolphinscheduler.api.service.impl.ProjectParameterServiceImpl; import org.apache.dolphinscheduler.api.utils.Result; import org.apache.dolphinscheduler.common.enums.UserType; import org.apache.dolphinscheduler.dao.entity.User; +import org.apache.dolphinscheduler.plugin.task.api.enums.DataType; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; @@ -48,8 +49,9 @@ public class ProjectParameterControllerTest { User loginUser = getGeneralUser(); Mockito.when(projectParameterService.createProjectParameter(Mockito.any(), Mockito.anyLong(), Mockito.any(), - Mockito.any())).thenReturn(getSuccessResult()); - Result result = projectParameterController.createProjectParameter(loginUser, 1, "key", "value"); + Mockito.any(), Mockito.any())).thenReturn(getSuccessResult()); + Result result = projectParameterController.createProjectParameter(loginUser, 1, "key", "value", + DataType.VARCHAR.name()); Assertions.assertEquals(Status.SUCCESS.getCode(), result.getCode()); } @@ -58,8 +60,9 @@ public class ProjectParameterControllerTest { User loginUser = getGeneralUser(); Mockito.when(projectParameterService.updateProjectParameter(Mockito.any(), Mockito.anyLong(), Mockito.anyLong(), - Mockito.any(), Mockito.any())).thenReturn(getSuccessResult()); - Result result = projectParameterController.updateProjectParameter(loginUser, 1, 1L, "key", "value"); + Mockito.any(), Mockito.any(), Mockito.any())).thenReturn(getSuccessResult()); + Result result = projectParameterController.updateProjectParameter(loginUser, 1, 1L, "key", "value", + DataType.LONG.name()); Assertions.assertEquals(Status.SUCCESS.getCode(), result.getCode()); } @@ -88,8 +91,9 @@ public class ProjectParameterControllerTest { User loginUser = getGeneralUser(); Mockito.when(projectParameterService.queryProjectParameterListPaging(Mockito.any(), Mockito.anyLong(), - Mockito.anyInt(), Mockito.anyInt(), Mockito.any())).thenReturn(getSuccessResult()); - Result result = projectParameterController.queryProjectParameterListPaging(loginUser, 1, "1", 1, 10); + Mockito.anyInt(), Mockito.anyInt(), Mockito.any(), Mockito.any())).thenReturn(getSuccessResult()); + Result result = projectParameterController.queryProjectParameterListPaging(loginUser, 1, "1", + DataType.VARCHAR.name(), 1, 10); Assertions.assertEquals(Status.SUCCESS.getCode(), result.getCode()); } diff --git a/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/service/ProjectParameterServiceTest.java b/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/service/ProjectParameterServiceTest.java index ca51690ce6..ef8ffd2768 100644 --- a/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/service/ProjectParameterServiceTest.java +++ b/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/service/ProjectParameterServiceTest.java @@ -36,6 +36,7 @@ import org.apache.dolphinscheduler.dao.entity.ProjectParameter; import org.apache.dolphinscheduler.dao.entity.User; import org.apache.dolphinscheduler.dao.mapper.ProjectMapper; import org.apache.dolphinscheduler.dao.mapper.ProjectParameterMapper; +import org.apache.dolphinscheduler.plugin.task.api.enums.DataType; import java.util.Collections; @@ -76,7 +77,8 @@ public class ProjectParameterServiceTest { // PERMISSION DENIED when(projectService.hasProjectAndWritePerm(Mockito.any(), Mockito.any(), Mockito.any(Result.class))) .thenReturn(false); - Result result = projectParameterService.createProjectParameter(loginUser, projectCode, "key", "value"); + Result result = projectParameterService.createProjectParameter(loginUser, projectCode, "key", "value", + DataType.VARCHAR.name()); assertNull(result.getData()); assertNull(result.getCode()); assertNull(result.getMsg()); @@ -88,25 +90,29 @@ public class ProjectParameterServiceTest { try (MockedStatic ignored = Mockito.mockStatic(CodeGenerateUtils.class)) { when(CodeGenerateUtils.genCode()).thenThrow(CodeGenerateUtils.CodeGenerateException.class); - result = projectParameterService.createProjectParameter(loginUser, projectCode, "key", "value"); + result = projectParameterService.createProjectParameter(loginUser, projectCode, "key", "value", + DataType.VARCHAR.name()); assertEquals(Status.CREATE_PROJECT_PARAMETER_ERROR.getCode(), result.getCode()); } // PROJECT_PARAMETER_ALREADY_EXISTS when(projectMapper.queryByCode(projectCode)).thenReturn(getProject(projectCode)); when(projectParameterMapper.selectOne(Mockito.any())).thenReturn(getProjectParameter()); - result = projectParameterService.createProjectParameter(loginUser, projectCode, "key", "value"); + result = projectParameterService.createProjectParameter(loginUser, projectCode, "key", "value", + DataType.VARCHAR.name()); assertEquals(Status.PROJECT_PARAMETER_ALREADY_EXISTS.getCode(), result.getCode()); // INSERT DATA ERROR when(projectParameterMapper.selectOne(Mockito.any())).thenReturn(null); when(projectParameterMapper.insert(Mockito.any())).thenReturn(-1); - result = projectParameterService.createProjectParameter(loginUser, projectCode, "key1", "value"); + result = projectParameterService.createProjectParameter(loginUser, projectCode, "key1", "value", + DataType.VARCHAR.name()); assertEquals(Status.CREATE_PROJECT_PARAMETER_ERROR.getCode(), result.getCode()); // SUCCESS when(projectParameterMapper.insert(Mockito.any())).thenReturn(1); - result = projectParameterService.createProjectParameter(loginUser, projectCode, "key1", "value"); + result = projectParameterService.createProjectParameter(loginUser, projectCode, "key1", "value", + DataType.VARCHAR.name()); assertEquals(Status.SUCCESS.getCode(), result.getCode()); } @@ -117,7 +123,8 @@ public class ProjectParameterServiceTest { // NO PERMISSION when(projectService.hasProjectAndWritePerm(Mockito.any(), Mockito.any(), Mockito.any(Result.class))) .thenReturn(false); - Result result = projectParameterService.updateProjectParameter(loginUser, projectCode, 1, "key", "value"); + Result result = projectParameterService.updateProjectParameter(loginUser, projectCode, 1, "key", "value", + DataType.VARCHAR.name()); assertNull(result.getData()); assertNull(result.getCode()); assertNull(result.getMsg()); @@ -127,28 +134,33 @@ public class ProjectParameterServiceTest { when(projectService.hasProjectAndWritePerm(Mockito.any(), Mockito.any(), Mockito.any(Result.class))) .thenReturn(true); when(projectParameterMapper.queryByCode(Mockito.anyLong())).thenReturn(null); - result = projectParameterService.updateProjectParameter(loginUser, projectCode, 1, "key", "value"); + result = projectParameterService.updateProjectParameter(loginUser, projectCode, 1, "key", "value", + DataType.VARCHAR.name()); assertEquals(Status.PROJECT_PARAMETER_NOT_EXISTS.getCode(), result.getCode()); // PROJECT_PARAMETER_ALREADY_EXISTS when(projectParameterMapper.queryByCode(Mockito.anyLong())).thenReturn(getProjectParameter()); when(projectParameterMapper.selectOne(Mockito.any())).thenReturn(getProjectParameter()); - result = projectParameterService.updateProjectParameter(loginUser, projectCode, 1, "key", "value"); + result = projectParameterService.updateProjectParameter(loginUser, projectCode, 1, "key", "value", + DataType.VARCHAR.name()); assertEquals(Status.PROJECT_PARAMETER_ALREADY_EXISTS.getCode(), result.getCode()); // PROJECT_UPDATE_ERROR when(projectParameterMapper.selectOne(Mockito.any())).thenReturn(null); when(projectParameterMapper.updateById(Mockito.any())).thenReturn(-1); - result = projectParameterService.updateProjectParameter(loginUser, projectCode, 1, "key1", "value"); + result = projectParameterService.updateProjectParameter(loginUser, projectCode, 1, "key1", "value", + DataType.VARCHAR.name()); assertEquals(Status.UPDATE_PROJECT_PARAMETER_ERROR.getCode(), result.getCode()); // SUCCESS when(projectParameterMapper.updateById(Mockito.any())).thenReturn(1); - result = projectParameterService.updateProjectParameter(loginUser, projectCode, 1, "key1", "value"); + result = projectParameterService.updateProjectParameter(loginUser, projectCode, 1, "key1", "value", + DataType.LONG.name()); assertEquals(Status.SUCCESS.getCode(), result.getCode()); ProjectParameter projectParameter = (ProjectParameter) result.getData(); assertNotNull(projectParameter.getOperator()); assertNotNull(projectParameter.getUpdateTime()); + assertEquals(DataType.LONG.name(), projectParameter.getParamDataType()); } @Test @@ -223,7 +235,8 @@ public class ProjectParameterServiceTest { .thenReturn(false); Result result = - projectParameterService.queryProjectParameterListPaging(loginUser, projectCode, pageSize, pageNo, null); + projectParameterService.queryProjectParameterListPaging(loginUser, projectCode, pageSize, pageNo, null, + DataType.VARCHAR.name()); assertNull(result.getData()); assertNull(result.getCode()); assertNull(result.getMsg()); @@ -235,9 +248,10 @@ public class ProjectParameterServiceTest { Page page = new Page<>(pageNo, pageSize); page.setRecords(Collections.singletonList(getProjectParameter())); - when(projectParameterMapper.queryProjectParameterListPaging(any(), anyLong(), any(), any())).thenReturn(page); + when(projectParameterMapper.queryProjectParameterListPaging(any(), anyLong(), any(), any(), any())) + .thenReturn(page); result = projectParameterService.queryProjectParameterListPaging(loginUser, projectCode, pageSize, pageNo, - null); + null, null); assertEquals(Status.SUCCESS.getCode(), result.getCode()); } diff --git a/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/ProjectParameter.java b/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/ProjectParameter.java index 03e9140145..4343d07c29 100644 --- a/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/ProjectParameter.java +++ b/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/ProjectParameter.java @@ -55,6 +55,9 @@ public class ProjectParameter { @TableField("param_value") private String paramValue; + @TableField("param_data_type") + private String paramDataType; + private Date createTime; private Date updateTime; diff --git a/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/ProjectParameterMapper.java b/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/ProjectParameterMapper.java index 0e7810158c..3f700a9a84 100644 --- a/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/ProjectParameterMapper.java +++ b/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/ProjectParameterMapper.java @@ -38,7 +38,8 @@ public interface ProjectParameterMapper extends BaseMapper { IPage queryProjectParameterListPaging(IPage page, @Param("projectCode") long projectCode, @Param("projectParameterIds") List projectParameterIds, - @Param("searchName") String searchName); + @Param("searchName") String searchName, + @Param("projectParameterDataType") String projectParameterDataType); List queryByProjectCode(@Param("projectCode") long projectCode); } diff --git a/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ProjectParameterMapper.xml b/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ProjectParameterMapper.xml index 5b22d40a81..4e1e972a04 100644 --- a/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ProjectParameterMapper.xml +++ b/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ProjectParameterMapper.xml @@ -19,7 +19,7 @@ - id, param_name, param_value, code, project_code, user_id, operator, create_time, update_time + id, param_name, param_value, param_data_type, code, project_code, user_id, operator, create_time, update_time select - pp.id, param_name, param_value, code, project_code, user_id, operator, pp.create_time, pp.update_time, + pp.id, param_name, param_value, param_data_type, code, project_code, user_id, operator, pp.create_time, pp.update_time, u.user_name as create_user, u2.user_name as modify_user from t_ds_project_parameter pp @@ -69,6 +69,9 @@ OR param_value LIKE concat('%', #{searchName}, '%') ) + + AND param_data_type = #{projectParameterDataType} + order by pp.update_time desc diff --git a/dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_h2.sql b/dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_h2.sql index 42c893bb05..a907ec69a1 100644 --- a/dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_h2.sql +++ b/dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_h2.sql @@ -667,6 +667,7 @@ CREATE TABLE t_ds_project_parameter id int(11) NOT NULL AUTO_INCREMENT, param_name varchar(255) NOT NULL, param_value text NOT NULL, + param_data_type varchar(50) DEFAULT 'VARCHAR', code bigint(20) NOT NULL, project_code bigint(20) NOT NULL, user_id int(11) DEFAULT NULL, diff --git a/dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql b/dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql index b30bc13887..3a4f17a357 100644 --- a/dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql +++ b/dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql @@ -668,6 +668,7 @@ CREATE TABLE `t_ds_project_parameter` ( `id` int(11) NOT NULL AUTO_INCREMENT COMMENT 'key', `param_name` varchar(255) NOT NULL COMMENT 'project parameter name', `param_value` text NOT NULL COMMENT 'project parameter value', + `param_data_type` varchar(50) DEFAULT 'VARCHAR' COMMENT 'project parameter data type', `code` bigint(20) NOT NULL COMMENT 'encoding', `project_code` bigint(20) NOT NULL COMMENT 'project code', `user_id` int(11) DEFAULT NULL COMMENT 'creator id', diff --git a/dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_postgresql.sql b/dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_postgresql.sql index 2d224092c4..d314c6ef09 100644 --- a/dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_postgresql.sql +++ b/dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_postgresql.sql @@ -595,6 +595,7 @@ CREATE TABLE t_ds_project_parameter ( id int NOT NULL , param_name varchar(255) NOT NULL , param_value text NOT NULL , + param_data_type varchar(50) DEFAULT 'VARCHAR', code bigint NOT NULL, project_code bigint NOT NULL, user_id int DEFAULT NULL , diff --git a/dolphinscheduler-dao/src/main/resources/sql/upgrade/3.2.2_schema/mysql/dolphinscheduler_ddl.sql b/dolphinscheduler-dao/src/main/resources/sql/upgrade/3.2.2_schema/mysql/dolphinscheduler_ddl.sql index d10ac6b710..000173b397 100644 --- a/dolphinscheduler-dao/src/main/resources/sql/upgrade/3.2.2_schema/mysql/dolphinscheduler_ddl.sql +++ b/dolphinscheduler-dao/src/main/resources/sql/upgrade/3.2.2_schema/mysql/dolphinscheduler_ddl.sql @@ -53,4 +53,6 @@ END; d// delimiter ; CALL modify_data_t_ds_audit_log_input_entry; -DROP PROCEDURE modify_data_t_ds_audit_log_input_entry; \ No newline at end of file +DROP PROCEDURE modify_data_t_ds_audit_log_input_entry; + +ALTER TABLE t_ds_project_parameter ADD `param_data_type` varchar (50) DEFAULT 'VARCHAR' COMMENT 'project parameter data type'; \ No newline at end of file diff --git a/dolphinscheduler-dao/src/main/resources/sql/upgrade/3.2.2_schema/postgresql/dolphinscheduler_ddl.sql b/dolphinscheduler-dao/src/main/resources/sql/upgrade/3.2.2_schema/postgresql/dolphinscheduler_ddl.sql index ee06588e58..4b4cb66651 100644 --- a/dolphinscheduler-dao/src/main/resources/sql/upgrade/3.2.2_schema/postgresql/dolphinscheduler_ddl.sql +++ b/dolphinscheduler-dao/src/main/resources/sql/upgrade/3.2.2_schema/postgresql/dolphinscheduler_ddl.sql @@ -56,4 +56,6 @@ $$ LANGUAGE plpgsql; d// select modify_data_t_ds_audit_log_input_entry(); -DROP FUNCTION IF EXISTS modify_data_t_ds_audit_log_input_entry(); \ No newline at end of file +DROP FUNCTION IF EXISTS modify_data_t_ds_audit_log_input_entry(); + +ALTER TABLE t_ds_project_parameter ADD COLUMN IF NOT EXISTS param_data_type varchar(50) DEFAULT 'VARCHAR'; \ No newline at end of file diff --git a/dolphinscheduler-dao/src/test/java/org/apache/dolphinscheduler/dao/mapper/ProjectParameterMapperTest.java b/dolphinscheduler-dao/src/test/java/org/apache/dolphinscheduler/dao/mapper/ProjectParameterMapperTest.java index 48e51d9ca3..1c13d95a0e 100644 --- a/dolphinscheduler-dao/src/test/java/org/apache/dolphinscheduler/dao/mapper/ProjectParameterMapperTest.java +++ b/dolphinscheduler-dao/src/test/java/org/apache/dolphinscheduler/dao/mapper/ProjectParameterMapperTest.java @@ -89,7 +89,7 @@ public class ProjectParameterMapperTest extends BaseDaoTest { insertOne(2, "name2", 2); Page page = new Page(1, 3); - IPage res = projectParameterMapper.queryProjectParameterListPaging(page, 1, null, null); + IPage res = projectParameterMapper.queryProjectParameterListPaging(page, 1, null, null, null); Assertions.assertEquals(1, res.getRecords().size()); } } diff --git a/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/expand/CuringParamsServiceImpl.java b/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/expand/CuringParamsServiceImpl.java index afcfae3fd6..5f042b7593 100644 --- a/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/expand/CuringParamsServiceImpl.java +++ b/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/expand/CuringParamsServiceImpl.java @@ -45,7 +45,9 @@ import org.apache.dolphinscheduler.plugin.task.api.model.Property; import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters; import org.apache.dolphinscheduler.plugin.task.api.utils.MapUtils; import org.apache.dolphinscheduler.plugin.task.api.utils.ParameterUtils; +import org.apache.dolphinscheduler.plugin.task.api.utils.PropertyUtils; +import org.apache.commons.collections4.CollectionUtils; import org.apache.commons.lang3.StringUtils; import java.util.Date; @@ -54,6 +56,7 @@ import java.util.Iterator; import java.util.List; import java.util.Map; import java.util.Set; +import java.util.function.Function; import java.util.stream.Collectors; import javax.annotation.Nullable; @@ -150,9 +153,11 @@ public class CuringParamsServiceImpl implements CuringParamsService { return new HashMap<>(); } String startParamJson = cmdParam.get(CommandKeyConstants.CMD_PARAM_START_PARAMS); - Map startParamMap = JSONUtils.toMap(startParamJson); - return startParamMap.entrySet().stream().collect(Collectors.toMap(Map.Entry::getKey, - entry -> new Property(entry.getKey(), Direct.IN, DataType.VARCHAR, entry.getValue()))); + List propertyList = PropertyUtils.startParamsTransformPropertyList(startParamJson); + if (CollectionUtils.isEmpty(propertyList)) { + return new HashMap<>(); + } + return propertyList.stream().collect(Collectors.toMap(Property::getProp, Function.identity())); } @Override @@ -181,8 +186,7 @@ public class CuringParamsServiceImpl implements CuringParamsService { Map prepareParamsMap = new HashMap<>(); // assign value to definedParams here - Map globalParamsMap = setGlobalParamsMap(processInstance); - Map globalParams = ParameterUtils.getUserDefParamsMap(globalParamsMap); + Map globalParams = setGlobalParamsMap(processInstance); // combining local and global parameters Map localParams = parameters.getInputLocalParametersMap(); @@ -287,15 +291,16 @@ public class CuringParamsServiceImpl implements CuringParamsService { Long.toString(taskInstance.getProcessInstance().getProcessDefinition().getProjectCode())); return params; } - private Map setGlobalParamsMap(ProcessInstance processInstance) { - Map globalParamsMap = new HashMap<>(16); + private Map setGlobalParamsMap(ProcessInstance processInstance) { + Map globalParamsMap = new HashMap<>(16); // global params string String globalParamsStr = processInstance.getGlobalParams(); if (globalParamsStr != null) { List globalParamsList = JSONUtils.toList(globalParamsStr, Property.class); globalParamsMap - .putAll(globalParamsList.stream().collect(Collectors.toMap(Property::getProp, Property::getValue))); + .putAll(globalParamsList.stream() + .collect(Collectors.toMap(Property::getProp, Function.identity()))); } return globalParamsMap; } @@ -323,7 +328,7 @@ public class CuringParamsServiceImpl implements CuringParamsService { projectParameterList.forEach(projectParameter -> { Property property = new Property(projectParameter.getParamName(), Direct.IN, - DataType.VARCHAR, + Enum.valueOf(DataType.class, projectParameter.getParamDataType()), projectParameter.getParamValue()); result.put(projectParameter.getParamName(), property); }); diff --git a/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/utils/PropertyUtils.java b/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/utils/PropertyUtils.java new file mode 100644 index 0000000000..41c115ab0f --- /dev/null +++ b/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/utils/PropertyUtils.java @@ -0,0 +1,64 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.dolphinscheduler.plugin.task.api.utils; + +import org.apache.dolphinscheduler.common.utils.JSONUtils; +import org.apache.dolphinscheduler.plugin.task.api.enums.DataType; +import org.apache.dolphinscheduler.plugin.task.api.enums.Direct; +import org.apache.dolphinscheduler.plugin.task.api.model.Property; + +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; + +import com.google.gson.JsonElement; +import com.google.gson.JsonParser; + +/** + * property utils + */ +public class PropertyUtils { + + private PropertyUtils() { + throw new IllegalStateException("PropertyUtils class"); + } + + /** + * startParams transform propertyList + * + * @param startParams startParams + * @return startParamList + */ + public static List startParamsTransformPropertyList(String startParams) { + List startParamList = null; + if (startParams != null) { + JsonElement jsonElement = JsonParser.parseString(startParams); + boolean isJson = jsonElement.isJsonObject(); + if (isJson) { + Map startParamMap = JSONUtils.toMap(startParams); + startParamList = startParamMap.entrySet().stream() + .map(entry -> new Property(entry.getKey(), Direct.IN, DataType.VARCHAR, entry.getValue())) + .collect(Collectors.toList()); + } else { + startParamList = JSONUtils.toList(startParams, Property.class); + } + } + return startParamList; + } + +} diff --git a/dolphinscheduler-ui/src/locales/en_US/project.ts b/dolphinscheduler-ui/src/locales/en_US/project.ts index 5e5ee4b2a9..68eb7b3102 100644 --- a/dolphinscheduler-ui/src/locales/en_US/project.ts +++ b/dolphinscheduler-ui/src/locales/en_US/project.ts @@ -930,12 +930,14 @@ export default { code: 'Parameter Code', name: 'Parameter Name', value: 'Parameter Value', + data_type: 'Parameter Data Type', create_user: 'Create User', modify_user: 'Modify User', create_time: 'Create Time', update_time: 'Update Time', name_tips: 'Please enter your parameter name', value_tips: 'Please enter your parameter value', + data_type_tips: 'Please choose your parameter data type', operation: 'Operation', edit: 'Edit', delete: 'Delete', diff --git a/dolphinscheduler-ui/src/locales/zh_CN/project.ts b/dolphinscheduler-ui/src/locales/zh_CN/project.ts index 85de0122fb..e1bfd6f672 100644 --- a/dolphinscheduler-ui/src/locales/zh_CN/project.ts +++ b/dolphinscheduler-ui/src/locales/zh_CN/project.ts @@ -900,12 +900,14 @@ export default { code: '参数编码', name: '参数名称', value: '参数值', + data_type: '参数类型', create_user: '创建用户', modify_user: '修改用户', create_time: '创建时间', update_time: '更新时间', name_tips: '请输入参数名称', value_tips: '请输入参数值', + data_type_tips: '请选择参数类型', operation: '操作', edit: '编辑', delete: '删除', diff --git a/dolphinscheduler-ui/src/service/modules/projects-parameter/types.ts b/dolphinscheduler-ui/src/service/modules/projects-parameter/types.ts index 9e0f747136..0e2a9c76bb 100644 --- a/dolphinscheduler-ui/src/service/modules/projects-parameter/types.ts +++ b/dolphinscheduler-ui/src/service/modules/projects-parameter/types.ts @@ -19,6 +19,7 @@ interface ListReq { pageNo: number pageSize: number searchVal?: string + projectParameterDataType?: string } interface ProjectParameterCodeReq { @@ -28,6 +29,7 @@ interface ProjectParameterCodeReq { interface ProjectParameterReq { projectParameterName: string projectParameterValue: string + projectParameterDataType: string } interface UpdateProjectParameterReq extends ProjectParameterReq { @@ -39,6 +41,7 @@ interface ProjectParameterList { code: number name: string value: string + data_type: string createTime: string updateTime: string } diff --git a/dolphinscheduler-ui/src/views/projects/parameter/components/parameter-modal.tsx b/dolphinscheduler-ui/src/views/projects/parameter/components/parameter-modal.tsx index c69ae864b2..d0e20569d1 100644 --- a/dolphinscheduler-ui/src/views/projects/parameter/components/parameter-modal.tsx +++ b/dolphinscheduler-ui/src/views/projects/parameter/components/parameter-modal.tsx @@ -23,9 +23,10 @@ import { watch } from 'vue' import Modal from '@/components/modal' -import { NForm, NFormItem, NInput } from 'naive-ui' +import { NForm, NFormItem, NInput, NSelect } from 'naive-ui' import { useModal } from './use-modal' import { useI18n } from 'vue-i18n' +import { DATA_TYPES_MAP, DEFAULT_DATA_TYPE } from "@/views/projects/parameter/data_type" const ParameterModal = defineComponent({ name: 'ParameterModal', @@ -52,9 +53,11 @@ const ParameterModal = defineComponent({ if (props.statusRef === 0) { variables.model.projectParameterName = '' variables.model.projectParameterValue = '' + variables.model.projectParameterDataType = DEFAULT_DATA_TYPE } else { variables.model.projectParameterName = props.row.paramName variables.model.projectParameterValue = props.row.paramValue + variables.model.projectParameterDataType = props.row.paramDataType } ctx.emit('cancelModal', props.showModalRef) } @@ -78,10 +81,12 @@ const ParameterModal = defineComponent({ if (props.statusRef === 0) { variables.model.projectParameterName = '' variables.model.projectParameterValue = '' + variables.model.projectParameterDataType = DEFAULT_DATA_TYPE } else { variables.model.code = props.row.code variables.model.projectParameterName = props.row.paramName variables.model.projectParameterValue = props.row.paramValue + variables.model.projectParameterDataType = props.row.paramDataType } } ) @@ -92,6 +97,7 @@ const ParameterModal = defineComponent({ variables.model.code = props.row.code variables.model.projectParameterName = props.row.paramName variables.model.projectParameterValue = props.row.paramValue + variables.model.projectParameterDataType = props.row.paramDataType } ) @@ -112,7 +118,8 @@ const ParameterModal = defineComponent({ onConfirm={this.confirmModal} confirmDisabled={ !this.model.projectParameterName || - !this.model.projectParameterValue + !this.model.projectParameterValue || + !this.model.projectParameterDataType } confirmClassName='btn-submit' cancelClassName='btn-cancel' @@ -135,6 +142,15 @@ const ParameterModal = defineComponent({ v-model={[this.model.projectParameterValue, 'value']} /> + + { + return { value: item, label: item } + })} + v-model={[this.model.projectParameterDataType, 'value']} + /> + ) }} diff --git a/dolphinscheduler-ui/src/views/projects/parameter/components/use-modal.ts b/dolphinscheduler-ui/src/views/projects/parameter/components/use-modal.ts index 56b5186846..50dae38a49 100644 --- a/dolphinscheduler-ui/src/views/projects/parameter/components/use-modal.ts +++ b/dolphinscheduler-ui/src/views/projects/parameter/components/use-modal.ts @@ -27,6 +27,7 @@ import { UpdateProjectParameterReq } from '@/service/modules/projects-parameter/types' import { useRouter } from 'vue-router' +import { DEFAULT_DATA_TYPE } from "@/views/projects/parameter/data_type"; export function useModal( props: any, @@ -41,7 +42,8 @@ export function useModal( model: { code: ref(-1), projectParameterName: ref(''), - projectParameterValue: ref('') + projectParameterValue: ref(''), + projectParameterDataType: ref(DEFAULT_DATA_TYPE) }, saving: false, rules: { @@ -62,6 +64,15 @@ export function useModal( return new Error(t('project.parameter.value_tips')) } } + }, + data_type: { + required: true, + trigger: ['input', 'blur'], + validator() { + if (variables.model.projectParameterDataType === '') { + return new Error(t('project.parameter.data_type_tips')) + } + } } } }) @@ -83,12 +94,14 @@ export function useModal( const submitModal = () => { const data: ProjectParameterReq = { projectParameterName: variables.model.projectParameterName, - projectParameterValue: variables.model.projectParameterValue + projectParameterValue: variables.model.projectParameterValue, + projectParameterDataType: variables.model.projectParameterDataType } createProjectParameter(data, variables.projectCode).then(() => { variables.model.projectParameterName = '' variables.model.projectParameterValue = '' + variables.model.projectParameterDataType = DEFAULT_DATA_TYPE ctx.emit('confirmModal', props.showModalRef) }) } @@ -97,7 +110,8 @@ export function useModal( const data: UpdateProjectParameterReq = { code: variables.model.code, projectParameterName: variables.model.projectParameterName, - projectParameterValue: variables.model.projectParameterValue + projectParameterValue: variables.model.projectParameterValue, + projectParameterDataType: variables.model.projectParameterDataType } updateProjectParameter(data, variables.projectCode).then(() => { diff --git a/dolphinscheduler-ui/src/views/projects/parameter/data_type.ts b/dolphinscheduler-ui/src/views/projects/parameter/data_type.ts new file mode 100644 index 0000000000..1d90384436 --- /dev/null +++ b/dolphinscheduler-ui/src/views/projects/parameter/data_type.ts @@ -0,0 +1,54 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +export const DATA_TYPES_MAP = { + VARCHAR: { + alias: 'VARCHAR' + }, + INTEGER: { + alias: 'INTEGER' + }, + LONG: { + alias: 'LONG' + }, + FLOAT: { + alias: 'FLOAT' + }, + DOUBLE: { + alias: 'DOUBLE' + }, + DATE: { + alias: 'DATE' + }, + TIME: { + alias: 'TIME' + }, + TIMESTAMP: { + alias: 'TIMESTAMP' + }, + BOOLEAN: { + alias: 'BOOLEAN' + }, + LIST: { + alias: 'LIST' + }, + FILE: { + alias: 'FILE' + } +} + +export const DEFAULT_DATA_TYPE = 'VARCHAR' diff --git a/dolphinscheduler-ui/src/views/projects/parameter/index.tsx b/dolphinscheduler-ui/src/views/projects/parameter/index.tsx index f8d0719f18..6b9519830b 100644 --- a/dolphinscheduler-ui/src/views/projects/parameter/index.tsx +++ b/dolphinscheduler-ui/src/views/projects/parameter/index.tsx @@ -21,7 +21,8 @@ import { NInput, NPagination, NSpace, - NButton + NButton, + NSelect } from 'naive-ui' import { defineComponent, onMounted, toRefs, watch } from 'vue' import { useI18n } from 'vue-i18n' @@ -29,6 +30,7 @@ import { useTable } from '@/views/projects/parameter/use-table' import Card from '@/components/card' import ParameterModal from '@/views/projects/parameter/components/parameter-modal' import { SearchOutlined } from '@vicons/antd' +import { DATA_TYPES_MAP } from "@/views/projects/parameter/data_type" export default defineComponent({ name: 'ProjectParameterList', @@ -40,6 +42,7 @@ export default defineComponent({ pageSize: variables.pageSize, pageNo: variables.page, searchVal: variables.searchVal, + projectParameterDataType: variables.projectParameterDataType, projectCode: variables.projectCode }) } @@ -116,6 +119,16 @@ export default defineComponent({ v-model={[this.searchVal, 'value']} placeholder={t('project.parameter.name')} /> + { + return { value: item, label: item } + })} + placeholder={t('project.parameter.data_type_tips')} + style={{ width: '180px' }} + clearable + /> diff --git a/dolphinscheduler-ui/src/views/projects/parameter/use-table.ts b/dolphinscheduler-ui/src/views/projects/parameter/use-table.ts index 9d0bf35dcd..6d98a7e163 100644 --- a/dolphinscheduler-ui/src/views/projects/parameter/use-table.ts +++ b/dolphinscheduler-ui/src/views/projects/parameter/use-table.ts @@ -45,6 +45,7 @@ export function useTable() { page: ref(1), pageSize: ref(10), searchVal: ref(), + projectParameterDataType: ref(), totalPage: ref(1), showRef: ref(false), statusRef: ref(0), @@ -69,6 +70,11 @@ export function useTable() { key: 'paramValue', ...COLUMN_WIDTH_CONFIG['name'] }, + { + title: t('project.parameter.data_type'), + key: 'paramDataType', + ...COLUMN_WIDTH_CONFIG['name'] + }, { title: t('project.parameter.create_user'), key: 'createUser', @@ -192,7 +198,8 @@ export function useTable() { getTableData({ pageSize: variables.pageSize, pageNo: variables.page, - searchVal: variables.searchVal + searchVal: variables.searchVal, + projectParameterDataType: variables.projectParameterDataType }) }) } diff --git a/dolphinscheduler-ui/src/views/projects/workflow/components/dag/dag-save-modal.tsx b/dolphinscheduler-ui/src/views/projects/workflow/components/dag/dag-save-modal.tsx index 9a55f8d3b2..bfd76e79b1 100644 --- a/dolphinscheduler-ui/src/views/projects/workflow/components/dag/dag-save-modal.tsx +++ b/dolphinscheduler-ui/src/views/projects/workflow/components/dag/dag-save-modal.tsx @@ -154,7 +154,8 @@ export default defineComponent({ formValue.value.globalParams = process.globalParamList.map((param) => ({ key: param.prop, value: param.value, - direct: param.direct + direct: param.direct, + type: param.type })) } } @@ -239,6 +240,7 @@ export default defineComponent({ return { key: '', direct: 'IN', + type: 'VARCHAR', value: '' } }} @@ -246,16 +248,16 @@ export default defineComponent({ > {{ default: (param: { - value: { key: string; direct: string; value: string } + value: { key: string; direct: string; type: string; value: string } }) => ( - + - + - + + + + - {this.startParamsList.length === 0 ? ( - - - - - - ) : ( - - {this.startParamsList.map((item, index) => ( - - - this.updateParamsList(index, param) - } - /> - this.removeStartParams(index)} - class='btn-delete-custom-parameter' - > - - - - - - - - - - - ))} - - )} + { + return { + key: '', + direct: 'IN', + type: 'VARCHAR', + value: '' + } + }} + class='input-startup-params' + > + {{ + default: (param: { + value: { prop: string; direct: string; type: string; value: string } + }) => ( + + + + + + + + + + + + + + + ) + }} +