Browse Source
* [BUG-#5678][Registry]fix registry init node miss (#5686) * [Improvement][UI] Update the update time after the user information is successfully modified (#5684) * improve edit the userinfo success, but the updatetime is not the latest. * Improved shell task execution result log information, adding process.waitFor() and process.exitValue() information to the original log (#5691) Co-authored-by: shenglm <shenglm840722@126.com> * [Feature-#5565][Master Worker-Server] Global Param passed by sense dependencies (#5603) * add globalParams new plan with varPool * add unit test * add python task varPoolParams Co-authored-by: wangxj <wangxj31> * Issue robot translation judgment changed to Chinese (#5694) Co-authored-by: chenxingchun <438044805@qq.com> * the update function should use post instead of get (#5703) * enhance form verify (#5696) * checkState only supports %s not {} (#5711) * [Fix-5701]When deleting a user, the accessToken associated with the user should also be deleted (#5697) * update * fix the codestyle error * fix the compile error * support rollback * [Fix-5699][UI] Fix update user error in user information (#5700) * [Improvement] the automatically generated spi service name in alert-plugin is wrong (#5676) * bug fix the auto generated spi service can't be recongized * include a new method * [Improvement-5622][project management] Modify the title (#5723) * [Fix-5714] When updating the existing alarm instance, the creation time should't be updated (#5715) * add a new init method. * [Fix#5758] There are some problems in the api documentation that need to be improved (#5759) * add the necessary parameters * openapi improve * fix code style error * [FIX-#5721][master-server] Global params parameter missing (#5757) Co-authored-by: wangxj <wangxj31> * [Fix-5738][UI] The cancel button in the pop-up dialog of `batch copy` and `batch move` doesn't work. (#5739) * Update relatedItems.vue * Update relatedItems.vue * [Improvement#5741][Worker] Improve task process status log (#5776) * [Improvement-5773][server] need to support two parameters related to task (#5774) * add some new parameter for task * restore official properties * improve imports * modify a variable's name Co-authored-by: jiang hua <jiang.hua@zhaopin.com.cn> * [FIX-5786][Improvement][Server] When the Worker turns down, the MasterServer cannot handle the Remove event correctly and throws NPE * [Improvement][Worker] Task log may be lost #5775 (#5783) * [Imporvement #5725][CheckStyle] upgrade checkstyle file (#5789) * [Imporvement #5725][CheckStyle] upgrade checkstyle file Upgrade checkstyle.xml to support checkstyle version 8.24+ * change ci checkstyle version * [Fix-5795][Improvement][Server] The starttime field in the HttpTask log is not displayed as expected. (#5796) * improve timestamp format make the startime in the log of httptask to be easier to read. * fix bad code smell and update the note. * [Imporvement #5621][job instance] start-time and end-time (#5621) (#5797) ·the list of workflow instances is sorted by start time and end time ·This closes #5621 * fix (#5803) Co-authored-by: shuangbofu <fusb@tuya.com> * fix: Remove duplicate "registryClient.close" method calls (#5805) Co-authored-by: wen-hemin <wenhemin@apache.com> * [Improvement][SPI] support load single plugin (#5794) change load operation of 'registry.plugin.dir' * [Improvement][Api Module] refactor registry client, remove spring annotation (#5814) * fix: refactor registry client, remove spring annotation * fix UT * fix UT * fix checkstyle * fix UT * fix UT * fix UT * fix: Rename RegistryCenterUtils method name Co-authored-by: wen-hemin <wenhemin@apache.com> * [Fix-5699][UI] Fix update user error in user information introduced by #5700 (#5735) * [Fix-5726] When we used the UI page, we found some problems such as parameter validation, parameter update shows success but actually work (#5727) * enhance the validation in UI * enchance form verifaction * simplify disable condition * fix: Remove unused class (#5833) Co-authored-by: wen-hemin <wenhemin@apache.com> * [fix-5737] [Bug][Datasource] datsource other param check error (#5835) Co-authored-by: wanggang <wanggy01@servyou.com.cn> * [Fix-5719][K8s] Fix Ingress tls: got map expected array On TLS enabled On Kubernetes [Fix-5719][K8s] Fix Ingress tls: got map expected array On TLS enabled On Kubernetes * [Fix-5825][BUG][WEB] the resource tree in the process definition of latest dev branch can't display correctly (#5826) * resoures-shows-error * fix codestyle error * add license header for new js * fix codesmell * [Improvement-5852][server] Support two parameters related to task for the rest of type of tasks. (#5867) * provide two system parameters to support the rest of type of tasks * provide two system parameters to support the rest of type of tasks * improve test conversion * [Improvement][Fix-5769][UI]When we try to delete the existing dag, the console in web browser would shows exception (#5770) * fix bug * cache the this variable * Avoid self name * fix code style compile error * [Fix-5781][UT] Fix test coverage in sonar (#5817) * build(UT): make jacoco running in offline-instrumentation issue: #5781 * build(UT): remove the jacoco agent dependency in microbench issue: #5781 * [Fix-5808][Server] When we try to transfer data using datax between different types of data sources, the worker will exit with ClassCastException (#5809) * bug fix * fix bug * simplify the code format * add a new parameter to make it easier to understand. * [Fix-5830][Improvement][UI] Improve the selection style in dag edit dialog (#5829) * improve the selection style * update another file * remove unnecessary css part. * [Fix-5904][upgrade]fix dev branch upgrade mysql sql script error (#5821) * fix dev branch upgrade mysql sql script error. * Update naming convention. * [Improvement][Api Module] refactor DataSourceParam and DependentParam, remove spring annotation (#5832) * fix: refactor api utils class, remove spring annotation. * fix: Optimization comments Co-authored-by: wen-hemin <wenhemin@apache.com> * correct the wrong annotion from zk queue implemented to java priority blocking queue (#5906) Co-authored-by: ywang46 <ywang46@paypal.com> Co-authored-by: Kirs <acm_master@163.com> Co-authored-by: kyoty <echohlne@gmail.com> Co-authored-by: ji04xiaogang <ji04xiaogang@163.com> Co-authored-by: shenglm <shenglm840722@126.com> Co-authored-by: wangxj3 <857234426@qq.com> Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com> Co-authored-by: chenxingchun <438044805@qq.com> Co-authored-by: Shiwen Cheng <chengshiwen0103@gmail.com> Co-authored-by: Jianchao Wang <akingchao@qq.com> Co-authored-by: Tanvi Moharir <74228962+tanvimoharir@users.noreply.github.com> Co-authored-by: Hua Jiang <jianghuachinacom@163.com> Co-authored-by: jiang hua <jiang.hua@zhaopin.com.cn> Co-authored-by: Wenjun Ruan <861923274@qq.com> Co-authored-by: Tandoy <56899730+Tandoy@users.noreply.github.com> Co-authored-by: 傅双波 <786183073@qq.com> Co-authored-by: shuangbofu <fusb@tuya.com> Co-authored-by: wen-hemin <39549317+wen-hemin@users.noreply.github.com> Co-authored-by: wen-hemin <wenhemin@apache.com> Co-authored-by: geosmart <geosmart@hotmail.com> Co-authored-by: wanggang <wanggy01@servyou.com.cn> Co-authored-by: AzureCN <colorazure@163.com> Co-authored-by: 深刻 <tsund@qq.com> Co-authored-by: zhuangchong <37063904+zhuangchong@users.noreply.github.com> Co-authored-by: Yao WANG <Yao.MR.CN@gmail.com> Co-authored-by: ywang46 <ywang46@paypal.com> Co-authored-by: JinyLeeChina <297062848@qq.com>2.0.7-release
JinyLeeChina
3 years ago
committed by
GitHub
57 changed files with 367 additions and 956 deletions
@ -1,84 +0,0 @@ |
|||||||
/* |
|
||||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
|
||||||
* contributor license agreements. See the NOTICE file distributed with |
|
||||||
* this work for additional information regarding copyright ownership. |
|
||||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
|
||||||
* (the "License"); you may not use this file except in compliance with |
|
||||||
* the License. You may obtain a copy of the License at |
|
||||||
* |
|
||||||
* http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
* |
|
||||||
* Unless required by applicable law or agreed to in writing, software |
|
||||||
* distributed under the License is distributed on an "AS IS" BASIS, |
|
||||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
|
||||||
* See the License for the specific language governing permissions and |
|
||||||
* limitations under the License. |
|
||||||
*/ |
|
||||||
package org.apache.dolphinscheduler.api.utils.exportprocess; |
|
||||||
|
|
||||||
import com.fasterxml.jackson.databind.JsonNode; |
|
||||||
import com.fasterxml.jackson.databind.node.ObjectNode; |
|
||||||
import org.apache.dolphinscheduler.common.enums.TaskType; |
|
||||||
import org.apache.dolphinscheduler.dao.entity.DataSource; |
|
||||||
import org.apache.dolphinscheduler.dao.mapper.DataSourceMapper; |
|
||||||
import org.springframework.beans.factory.InitializingBean; |
|
||||||
import org.springframework.beans.factory.annotation.Autowired; |
|
||||||
import org.springframework.stereotype.Service; |
|
||||||
|
|
||||||
import java.util.List; |
|
||||||
|
|
||||||
/** |
|
||||||
* task node add datasource param strategy |
|
||||||
*/ |
|
||||||
@Service |
|
||||||
public class DataSourceParam implements ProcessAddTaskParam, InitializingBean { |
|
||||||
|
|
||||||
private static final String PARAMS = "params"; |
|
||||||
@Autowired |
|
||||||
private DataSourceMapper dataSourceMapper; |
|
||||||
|
|
||||||
/** |
|
||||||
* add datasource params |
|
||||||
* @param taskNode task node json object |
|
||||||
* @return task node json object |
|
||||||
*/ |
|
||||||
@Override |
|
||||||
public JsonNode addExportSpecialParam(JsonNode taskNode) { |
|
||||||
// add sqlParameters
|
|
||||||
ObjectNode sqlParameters = (ObjectNode) taskNode.path(PARAMS); |
|
||||||
DataSource dataSource = dataSourceMapper.selectById(sqlParameters.get("datasource").asInt()); |
|
||||||
if (null != dataSource) { |
|
||||||
sqlParameters.put("datasourceName", dataSource.getName()); |
|
||||||
} |
|
||||||
((ObjectNode)taskNode).set(PARAMS, sqlParameters); |
|
||||||
|
|
||||||
return taskNode; |
|
||||||
} |
|
||||||
|
|
||||||
/** |
|
||||||
* import process add datasource params |
|
||||||
* @param taskNode task node json object |
|
||||||
* @return task node json object |
|
||||||
*/ |
|
||||||
@Override |
|
||||||
public JsonNode addImportSpecialParam(JsonNode taskNode) { |
|
||||||
ObjectNode sqlParameters = (ObjectNode) taskNode.path(PARAMS); |
|
||||||
List<DataSource> dataSources = dataSourceMapper.queryDataSourceByName(sqlParameters.path("datasourceName").asText()); |
|
||||||
if (!dataSources.isEmpty()) { |
|
||||||
DataSource dataSource = dataSources.get(0); |
|
||||||
sqlParameters.put("datasource", dataSource.getId()); |
|
||||||
} |
|
||||||
((ObjectNode)taskNode).set(PARAMS, sqlParameters); |
|
||||||
return taskNode; |
|
||||||
} |
|
||||||
|
|
||||||
|
|
||||||
/** |
|
||||||
* put datasource strategy |
|
||||||
*/ |
|
||||||
@Override |
|
||||||
public void afterPropertiesSet() { |
|
||||||
TaskNodeParamFactory.register(TaskType.SQL.getDesc(), this); |
|
||||||
TaskNodeParamFactory.register(TaskType.PROCEDURE.getDesc(), this); |
|
||||||
} |
|
||||||
} |
|
||||||
@ -1,114 +0,0 @@ |
|||||||
/* |
|
||||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
|
||||||
* contributor license agreements. See the NOTICE file distributed with |
|
||||||
* this work for additional information regarding copyright ownership. |
|
||||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
|
||||||
* (the "License"); you may not use this file except in compliance with |
|
||||||
* the License. You may obtain a copy of the License at |
|
||||||
* |
|
||||||
* http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
* |
|
||||||
* Unless required by applicable law or agreed to in writing, software |
|
||||||
* distributed under the License is distributed on an "AS IS" BASIS, |
|
||||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
|
||||||
* See the License for the specific language governing permissions and |
|
||||||
* limitations under the License. |
|
||||||
*/ |
|
||||||
package org.apache.dolphinscheduler.api.utils.exportprocess; |
|
||||||
|
|
||||||
import com.fasterxml.jackson.databind.JsonNode; |
|
||||||
import com.fasterxml.jackson.databind.node.ArrayNode; |
|
||||||
import com.fasterxml.jackson.databind.node.ObjectNode; |
|
||||||
import org.apache.dolphinscheduler.common.enums.TaskType; |
|
||||||
import org.apache.dolphinscheduler.common.utils.*; |
|
||||||
import org.apache.dolphinscheduler.dao.entity.ProcessDefinition; |
|
||||||
import org.apache.dolphinscheduler.dao.entity.Project; |
|
||||||
import org.apache.dolphinscheduler.dao.mapper.ProcessDefinitionMapper; |
|
||||||
import org.apache.dolphinscheduler.dao.mapper.ProjectMapper; |
|
||||||
import org.springframework.beans.factory.InitializingBean; |
|
||||||
import org.springframework.beans.factory.annotation.Autowired; |
|
||||||
import org.springframework.stereotype.Service; |
|
||||||
|
|
||||||
/** |
|
||||||
* task node add dependent param strategy |
|
||||||
*/ |
|
||||||
@Service |
|
||||||
public class DependentParam implements ProcessAddTaskParam, InitializingBean { |
|
||||||
|
|
||||||
private static final String DEPENDENCE = "dependence"; |
|
||||||
|
|
||||||
@Autowired |
|
||||||
ProcessDefinitionMapper processDefineMapper; |
|
||||||
|
|
||||||
@Autowired |
|
||||||
ProjectMapper projectMapper; |
|
||||||
|
|
||||||
/** |
|
||||||
* add dependent param |
|
||||||
* @param taskNode task node json object |
|
||||||
* @return task node json object |
|
||||||
*/ |
|
||||||
@Override |
|
||||||
public JsonNode addExportSpecialParam(JsonNode taskNode) { |
|
||||||
// add dependent param
|
|
||||||
ObjectNode dependentParameters = JSONUtils.parseObject(taskNode.path(DEPENDENCE).asText()); |
|
||||||
|
|
||||||
if (null != dependentParameters) { |
|
||||||
ArrayNode dependTaskList = (ArrayNode) dependentParameters.get("dependTaskList"); |
|
||||||
for (int j = 0; j < dependTaskList.size(); j++) { |
|
||||||
JsonNode dependentTaskModel = dependTaskList.path(j); |
|
||||||
ArrayNode dependItemList = (ArrayNode) dependentTaskModel.get("dependItemList"); |
|
||||||
for (int k = 0; k < dependItemList.size(); k++) { |
|
||||||
ObjectNode dependentItem = (ObjectNode) dependItemList.path(k); |
|
||||||
int definitionId = dependentItem.path("definitionId").asInt(); |
|
||||||
ProcessDefinition definition = processDefineMapper.queryByDefineId(definitionId); |
|
||||||
if (null != definition) { |
|
||||||
dependentItem.put("projectName", definition.getProjectName()); |
|
||||||
dependentItem.put("definitionName", definition.getName()); |
|
||||||
} |
|
||||||
} |
|
||||||
} |
|
||||||
((ObjectNode)taskNode).set(DEPENDENCE, dependentParameters); |
|
||||||
} |
|
||||||
|
|
||||||
return taskNode; |
|
||||||
} |
|
||||||
|
|
||||||
/** |
|
||||||
* import process add dependent param |
|
||||||
* @param taskNode task node json object |
|
||||||
* @return |
|
||||||
*/ |
|
||||||
@Override |
|
||||||
public JsonNode addImportSpecialParam(JsonNode taskNode) { |
|
||||||
ObjectNode dependentParameters = JSONUtils.parseObject(taskNode.path(DEPENDENCE).asText()); |
|
||||||
if(dependentParameters != null){ |
|
||||||
ArrayNode dependTaskList = (ArrayNode) dependentParameters.path("dependTaskList"); |
|
||||||
for (int h = 0; h < dependTaskList.size(); h++) { |
|
||||||
ObjectNode dependentTaskModel = (ObjectNode) dependTaskList.path(h); |
|
||||||
ArrayNode dependItemList = (ArrayNode) dependentTaskModel.get("dependItemList"); |
|
||||||
for (int k = 0; k < dependItemList.size(); k++) { |
|
||||||
ObjectNode dependentItem = (ObjectNode) dependItemList.path(k); |
|
||||||
Project dependentItemProject = projectMapper.queryByName(dependentItem.path("projectName").asText()); |
|
||||||
if(dependentItemProject != null){ |
|
||||||
ProcessDefinition definition = processDefineMapper.queryByDefineName(dependentItemProject.getCode(),dependentItem.path("definitionName").asText()); |
|
||||||
if(definition != null){ |
|
||||||
dependentItem.put("projectId",dependentItemProject.getId()); |
|
||||||
dependentItem.put("definitionId",definition.getId()); |
|
||||||
} |
|
||||||
} |
|
||||||
} |
|
||||||
} |
|
||||||
((ObjectNode)taskNode).set(DEPENDENCE, dependentParameters); |
|
||||||
} |
|
||||||
return taskNode; |
|
||||||
} |
|
||||||
|
|
||||||
/** |
|
||||||
* put dependent strategy |
|
||||||
*/ |
|
||||||
@Override |
|
||||||
public void afterPropertiesSet() { |
|
||||||
TaskNodeParamFactory.register(TaskType.DEPENDENT.getDesc(), this); |
|
||||||
} |
|
||||||
} |
|
||||||
@ -1,39 +0,0 @@ |
|||||||
/* |
|
||||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
|
||||||
* contributor license agreements. See the NOTICE file distributed with |
|
||||||
* this work for additional information regarding copyright ownership. |
|
||||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
|
||||||
* (the "License"); you may not use this file except in compliance with |
|
||||||
* the License. You may obtain a copy of the License at |
|
||||||
* |
|
||||||
* http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
* |
|
||||||
* Unless required by applicable law or agreed to in writing, software |
|
||||||
* distributed under the License is distributed on an "AS IS" BASIS, |
|
||||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
|
||||||
* See the License for the specific language governing permissions and |
|
||||||
* limitations under the License. |
|
||||||
*/ |
|
||||||
package org.apache.dolphinscheduler.api.utils.exportprocess; |
|
||||||
|
|
||||||
import com.fasterxml.jackson.databind.JsonNode; |
|
||||||
|
|
||||||
/** |
|
||||||
* ProcessAddTaskParam |
|
||||||
*/ |
|
||||||
public interface ProcessAddTaskParam { |
|
||||||
|
|
||||||
/** |
|
||||||
* add export task special param: sql task dependent task |
|
||||||
* @param taskNode task node json object |
|
||||||
* @return task node json object |
|
||||||
*/ |
|
||||||
JsonNode addExportSpecialParam(JsonNode taskNode); |
|
||||||
|
|
||||||
/** |
|
||||||
* add task special param: sql task dependent task |
|
||||||
* @param taskNode task node json object |
|
||||||
* @return task node json object |
|
||||||
*/ |
|
||||||
JsonNode addImportSpecialParam(JsonNode taskNode); |
|
||||||
} |
|
||||||
@ -1,38 +0,0 @@ |
|||||||
/* |
|
||||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
|
||||||
* contributor license agreements. See the NOTICE file distributed with |
|
||||||
* this work for additional information regarding copyright ownership. |
|
||||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
|
||||||
* (the "License"); you may not use this file except in compliance with |
|
||||||
* the License. You may obtain a copy of the License at |
|
||||||
* |
|
||||||
* http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
* |
|
||||||
* Unless required by applicable law or agreed to in writing, software |
|
||||||
* distributed under the License is distributed on an "AS IS" BASIS, |
|
||||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
|
||||||
* See the License for the specific language governing permissions and |
|
||||||
* limitations under the License. |
|
||||||
*/ |
|
||||||
package org.apache.dolphinscheduler.api.utils.exportprocess; |
|
||||||
|
|
||||||
import java.util.Map; |
|
||||||
import java.util.concurrent.ConcurrentHashMap; |
|
||||||
|
|
||||||
/** |
|
||||||
* task node param factory |
|
||||||
*/ |
|
||||||
public class TaskNodeParamFactory { |
|
||||||
|
|
||||||
private static Map<String, ProcessAddTaskParam> taskServices = new ConcurrentHashMap<>(); |
|
||||||
|
|
||||||
public static ProcessAddTaskParam getByTaskType(String taskType){ |
|
||||||
return taskServices.get(taskType); |
|
||||||
} |
|
||||||
|
|
||||||
static void register(String taskType, ProcessAddTaskParam addSpecialTaskParam){ |
|
||||||
if (null != taskType) { |
|
||||||
taskServices.put(taskType, addSpecialTaskParam); |
|
||||||
} |
|
||||||
} |
|
||||||
} |
|
||||||
@ -1,84 +0,0 @@ |
|||||||
/* |
|
||||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
|
||||||
* contributor license agreements. See the NOTICE file distributed with |
|
||||||
* this work for additional information regarding copyright ownership. |
|
||||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
|
||||||
* (the "License"); you may not use this file except in compliance with |
|
||||||
* the License. You may obtain a copy of the License at |
|
||||||
* |
|
||||||
* http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
* |
|
||||||
* Unless required by applicable law or agreed to in writing, software |
|
||||||
* distributed under the License is distributed on an "AS IS" BASIS, |
|
||||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
|
||||||
* See the License for the specific language governing permissions and |
|
||||||
* limitations under the License. |
|
||||||
*/ |
|
||||||
|
|
||||||
package org.apache.dolphinscheduler.api.utils.exportprocess; |
|
||||||
|
|
||||||
import org.apache.dolphinscheduler.api.controller.AbstractControllerTest; |
|
||||||
import org.apache.dolphinscheduler.common.utils.JSONUtils; |
|
||||||
import org.apache.dolphinscheduler.common.utils.StringUtils; |
|
||||||
|
|
||||||
import org.json.JSONException; |
|
||||||
import org.junit.Test; |
|
||||||
import org.skyscreamer.jsonassert.JSONAssert; |
|
||||||
|
|
||||||
import com.fasterxml.jackson.databind.JsonNode; |
|
||||||
import com.fasterxml.jackson.databind.node.ObjectNode; |
|
||||||
|
|
||||||
/** |
|
||||||
* DataSourceParamTest |
|
||||||
*/ |
|
||||||
public class DataSourceParamTest extends AbstractControllerTest { |
|
||||||
|
|
||||||
@Test |
|
||||||
public void testAddExportDependentSpecialParam() throws JSONException { |
|
||||||
|
|
||||||
String sqlJson = "{\"type\":\"SQL\",\"id\":\"tasks-27297\",\"name\":\"sql\"," |
|
||||||
+ "\"params\":{\"type\":\"MYSQL\",\"datasource\":1,\"sql\":\"select * from test\"," |
|
||||||
+ "\"udfs\":\"\",\"sqlType\":\"1\",\"title\":\"\",\"receivers\":\"\",\"receiversCc\":\"\",\"showType\":\"TABLE\"" |
|
||||||
+ ",\"localParams\":[],\"connParams\":\"\"," |
|
||||||
+ "\"preStatements\":[],\"postStatements\":[]}," |
|
||||||
+ "\"description\":\"\",\"runFlag\":\"NORMAL\",\"dependence\":{},\"maxRetryTimes\":\"0\"," |
|
||||||
+ "\"retryInterval\":\"1\",\"timeout\":{\"strategy\":\"\"," |
|
||||||
+ "\"enable\":false},\"taskInstancePriority\":\"MEDIUM\",\"workerGroupId\":-1," |
|
||||||
+ "\"preTasks\":[\"dependent\"]}"; |
|
||||||
|
|
||||||
ObjectNode taskNode = JSONUtils.parseObject(sqlJson); |
|
||||||
if (StringUtils.isNotEmpty(taskNode.path("type").asText())) { |
|
||||||
String taskType = taskNode.path("type").asText(); |
|
||||||
|
|
||||||
ProcessAddTaskParam addTaskParam = TaskNodeParamFactory.getByTaskType(taskType); |
|
||||||
|
|
||||||
JsonNode sql = addTaskParam.addExportSpecialParam(taskNode); |
|
||||||
|
|
||||||
JSONAssert.assertEquals(taskNode.toString(), sql.toString(), false); |
|
||||||
} |
|
||||||
} |
|
||||||
|
|
||||||
@Test |
|
||||||
public void testAddImportDependentSpecialParam() throws JSONException { |
|
||||||
String sqlJson = "{\"workerGroupId\":-1,\"description\":\"\",\"runFlag\":\"NORMAL\"," |
|
||||||
+ "\"type\":\"SQL\",\"params\":{\"postStatements\":[]," |
|
||||||
+ "\"connParams\":\"\",\"receiversCc\":\"\",\"udfs\":\"\"," |
|
||||||
+ "\"type\":\"MYSQL\",\"title\":\"\",\"sql\":\"show tables\",\"" |
|
||||||
+ "preStatements\":[],\"sqlType\":\"1\",\"receivers\":\"\",\"datasource\":1," |
|
||||||
+ "\"showType\":\"TABLE\",\"localParams\":[],\"datasourceName\":\"dsmetadata\"},\"timeout\"" |
|
||||||
+ ":{\"enable\":false,\"strategy\":\"\"},\"maxRetryTimes\":\"0\"," |
|
||||||
+ "\"taskInstancePriority\":\"MEDIUM\",\"name\":\"mysql\",\"dependence\":{}," |
|
||||||
+ "\"retryInterval\":\"1\",\"preTasks\":[\"dependent\"],\"id\":\"tasks-8745\"}"; |
|
||||||
|
|
||||||
ObjectNode taskNode = JSONUtils.parseObject(sqlJson); |
|
||||||
if (StringUtils.isNotEmpty(taskNode.path("type").asText())) { |
|
||||||
String taskType = taskNode.path("type").asText(); |
|
||||||
|
|
||||||
ProcessAddTaskParam addTaskParam = TaskNodeParamFactory.getByTaskType(taskType); |
|
||||||
|
|
||||||
JsonNode sql = addTaskParam.addImportSpecialParam(taskNode); |
|
||||||
|
|
||||||
JSONAssert.assertEquals(taskNode.toString(), sql.toString(), false); |
|
||||||
} |
|
||||||
} |
|
||||||
} |
|
||||||
@ -1,110 +0,0 @@ |
|||||||
/* |
|
||||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
|
||||||
* contributor license agreements. See the NOTICE file distributed with |
|
||||||
* this work for additional information regarding copyright ownership. |
|
||||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
|
||||||
* (the "License"); you may not use this file except in compliance with |
|
||||||
* the License. You may obtain a copy of the License at |
|
||||||
* |
|
||||||
* http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
* |
|
||||||
* Unless required by applicable law or agreed to in writing, software |
|
||||||
* distributed under the License is distributed on an "AS IS" BASIS, |
|
||||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
|
||||||
* See the License for the specific language governing permissions and |
|
||||||
* limitations under the License. |
|
||||||
*/ |
|
||||||
|
|
||||||
package org.apache.dolphinscheduler.api.utils.exportprocess; |
|
||||||
|
|
||||||
import org.apache.dolphinscheduler.api.controller.AbstractControllerTest; |
|
||||||
import org.apache.dolphinscheduler.common.utils.JSONUtils; |
|
||||||
import org.apache.dolphinscheduler.common.utils.StringUtils; |
|
||||||
|
|
||||||
import org.json.JSONException; |
|
||||||
import org.junit.Test; |
|
||||||
import org.skyscreamer.jsonassert.JSONAssert; |
|
||||||
|
|
||||||
import com.fasterxml.jackson.databind.JsonNode; |
|
||||||
import com.fasterxml.jackson.databind.node.ObjectNode; |
|
||||||
|
|
||||||
/** |
|
||||||
* DependentParamTest |
|
||||||
*/ |
|
||||||
public class DependentParamTest extends AbstractControllerTest { |
|
||||||
|
|
||||||
@Test |
|
||||||
public void testAddExportDependentSpecialParam() throws JSONException { |
|
||||||
String dependentJson = "{\"type\":\"DEPENDENT\",\"id\":\"tasks-33787\"," |
|
||||||
+ "\"name\":\"dependent\",\"params\":{},\"description\":\"\",\"runFlag\":\"NORMAL\"," |
|
||||||
+ "\"dependence\":{\"relation\":\"AND\",\"dependTaskList\":[{\"relation\":\"AND\"," |
|
||||||
+ "\"dependItemList\":[{\"projectId\":2,\"definitionId\":46,\"depTasks\":\"ALL\"," |
|
||||||
+ "\"cycle\":\"day\",\"dateValue\":\"today\"}]}]}}"; |
|
||||||
|
|
||||||
ObjectNode taskNode = JSONUtils.parseObject(dependentJson); |
|
||||||
if (StringUtils.isNotEmpty(taskNode.path("type").asText())) { |
|
||||||
String taskType = taskNode.path("type").asText(); |
|
||||||
|
|
||||||
ProcessAddTaskParam addTaskParam = TaskNodeParamFactory.getByTaskType(taskType); |
|
||||||
|
|
||||||
JsonNode dependent = addTaskParam.addExportSpecialParam(taskNode); |
|
||||||
|
|
||||||
JSONAssert.assertEquals(taskNode.toString(), dependent.toString(), false); |
|
||||||
} |
|
||||||
|
|
||||||
String dependentEmpty = "{\"type\":\"DEPENDENT\",\"id\":\"tasks-33787\"," |
|
||||||
+ "\"name\":\"dependent\",\"params\":{},\"description\":\"\",\"runFlag\":\"NORMAL\"}"; |
|
||||||
|
|
||||||
ObjectNode taskEmpty = JSONUtils.parseObject(dependentEmpty); |
|
||||||
if (StringUtils.isNotEmpty(taskEmpty.path("type").asText())) { |
|
||||||
String taskType = taskEmpty.path("type").asText(); |
|
||||||
|
|
||||||
ProcessAddTaskParam addTaskParam = TaskNodeParamFactory.getByTaskType(taskType); |
|
||||||
|
|
||||||
JsonNode dependent = addTaskParam.addImportSpecialParam(taskEmpty); |
|
||||||
|
|
||||||
JSONAssert.assertEquals(taskEmpty.toString(), dependent.toString(), false); |
|
||||||
} |
|
||||||
|
|
||||||
} |
|
||||||
|
|
||||||
@Test |
|
||||||
public void testAddImportDependentSpecialParam() throws JSONException { |
|
||||||
String dependentJson = "{\"workerGroupId\":-1,\"description\":\"\",\"runFlag\":\"NORMAL\"" |
|
||||||
+ ",\"type\":\"DEPENDENT\",\"params\":{},\"timeout\":{\"enable\":false," |
|
||||||
+ "\"strategy\":\"\"},\"maxRetryTimes\":\"0\",\"taskInstancePriority\":\"MEDIUM\"" |
|
||||||
+ ",\"name\":\"dependent\"," |
|
||||||
+ "\"dependence\":{\"dependTaskList\":[{\"dependItemList\":[{\"dateValue\":\"today\"," |
|
||||||
+ "\"definitionName\":\"shell-1\",\"depTasks\":\"shell-1\",\"projectName\":\"test\"," |
|
||||||
+ "\"projectId\":1,\"cycle\":\"day\",\"definitionId\":7}],\"relation\":\"AND\"}]," |
|
||||||
+ "\"relation\":\"AND\"},\"retryInterval\":\"1\",\"preTasks\":[],\"id\":\"tasks-55485\"}"; |
|
||||||
|
|
||||||
ObjectNode taskNode = JSONUtils.parseObject(dependentJson); |
|
||||||
if (StringUtils.isNotEmpty(taskNode.path("type").asText())) { |
|
||||||
String taskType = taskNode.path("type").asText(); |
|
||||||
|
|
||||||
ProcessAddTaskParam addTaskParam = TaskNodeParamFactory.getByTaskType(taskType); |
|
||||||
|
|
||||||
JsonNode dependent = addTaskParam.addImportSpecialParam(taskNode); |
|
||||||
|
|
||||||
JSONAssert.assertEquals(taskNode.toString(), dependent.toString(), false); |
|
||||||
} |
|
||||||
|
|
||||||
String dependentEmpty = "{\"workerGroupId\":-1,\"description\":\"\",\"runFlag\":\"NORMAL\"" |
|
||||||
+ ",\"type\":\"DEPENDENT\",\"params\":{},\"timeout\":{\"enable\":false," |
|
||||||
+ "\"strategy\":\"\"},\"maxRetryTimes\":\"0\",\"taskInstancePriority\":\"MEDIUM\"" |
|
||||||
+ ",\"name\":\"dependent\",\"retryInterval\":\"1\",\"preTasks\":[],\"id\":\"tasks-55485\"}"; |
|
||||||
|
|
||||||
JsonNode taskNodeEmpty = JSONUtils.parseObject(dependentEmpty); |
|
||||||
if (StringUtils.isNotEmpty(taskNodeEmpty.path("type").asText())) { |
|
||||||
String taskType = taskNodeEmpty.path("type").asText(); |
|
||||||
|
|
||||||
ProcessAddTaskParam addTaskParam = TaskNodeParamFactory.getByTaskType(taskType); |
|
||||||
|
|
||||||
JsonNode dependent = addTaskParam.addImportSpecialParam(taskNode); |
|
||||||
|
|
||||||
JSONAssert.assertEquals(taskNodeEmpty.toString(), dependent.toString(), false); |
|
||||||
} |
|
||||||
|
|
||||||
} |
|
||||||
} |
|
||||||
@ -1,54 +0,0 @@ |
|||||||
/* |
|
||||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
|
||||||
* contributor license agreements. See the NOTICE file distributed with |
|
||||||
* this work for additional information regarding copyright ownership. |
|
||||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
|
||||||
* (the "License"); you may not use this file except in compliance with |
|
||||||
* the License. You may obtain a copy of the License at |
|
||||||
* |
|
||||||
* http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
* |
|
||||||
* Unless required by applicable law or agreed to in writing, software |
|
||||||
* distributed under the License is distributed on an "AS IS" BASIS, |
|
||||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
|
||||||
* See the License for the specific language governing permissions and |
|
||||||
* limitations under the License. |
|
||||||
*/ |
|
||||||
|
|
||||||
package org.apache.dolphinscheduler.server.utils; |
|
||||||
|
|
||||||
import org.apache.dolphinscheduler.service.registry.RegistryClient; |
|
||||||
import org.slf4j.Logger; |
|
||||||
import org.slf4j.LoggerFactory; |
|
||||||
import org.springframework.boot.CommandLineRunner; |
|
||||||
import org.springframework.boot.WebApplicationType; |
|
||||||
import org.springframework.boot.builder.SpringApplicationBuilder; |
|
||||||
import org.springframework.context.annotation.ComponentScan; |
|
||||||
|
|
||||||
@ComponentScan("org.apache.dolphinscheduler") |
|
||||||
public class RemoveZKNode implements CommandLineRunner { |
|
||||||
|
|
||||||
private static final Integer ARGS_LENGTH = 1; |
|
||||||
|
|
||||||
private static final Logger logger = LoggerFactory.getLogger(RemoveZKNode.class); |
|
||||||
|
|
||||||
/** |
|
||||||
* zookeeper operator |
|
||||||
*/ |
|
||||||
private RegistryClient registryClient = RegistryClient.getInstance(); |
|
||||||
|
|
||||||
public static void main(String[] args) { |
|
||||||
new SpringApplicationBuilder(RemoveZKNode.class).web(WebApplicationType.NONE).run(args); |
|
||||||
} |
|
||||||
|
|
||||||
@Override |
|
||||||
public void run(String... args) throws Exception { |
|
||||||
if (args.length != ARGS_LENGTH) { |
|
||||||
logger.error("Usage: <node>"); |
|
||||||
return; |
|
||||||
} |
|
||||||
|
|
||||||
registryClient.remove(args[0]); |
|
||||||
registryClient.close(); |
|
||||||
} |
|
||||||
} |
|
||||||
@ -0,0 +1,44 @@ |
|||||||
|
/* |
||||||
|
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||||
|
* contributor license agreements. See the NOTICE file distributed with |
||||||
|
* this work for additional information regarding copyright ownership. |
||||||
|
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||||
|
* (the "License"); you may not use this file except in compliance with |
||||||
|
* the License. You may obtain a copy of the License at |
||||||
|
* |
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
* |
||||||
|
* Unless required by applicable law or agreed to in writing, software |
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS, |
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||||
|
* See the License for the specific language governing permissions and |
||||||
|
* limitations under the License. |
||||||
|
*/ |
||||||
|
export function diGuiTree (items) { // Recursive convenience tree structure
|
||||||
|
items.forEach(item => { |
||||||
|
item.children === '' || item.children === undefined || item.children === null || item.children.length === 0 |
||||||
|
? operationTree(item) : diGuiTree(item.children) |
||||||
|
}) |
||||||
|
} |
||||||
|
|
||||||
|
export function operationTree (item) { |
||||||
|
if (item.dirctory) { |
||||||
|
item.isDisabled = true |
||||||
|
} |
||||||
|
delete item.children |
||||||
|
} |
||||||
|
|
||||||
|
export function searchTree (element, id) { |
||||||
|
// 根据id查找节点
|
||||||
|
if (element.id === id) { |
||||||
|
return element |
||||||
|
} else if (element.children) { |
||||||
|
let i |
||||||
|
let result = null |
||||||
|
for (i = 0; result === null && i < element.children.length; i++) { |
||||||
|
result = searchTree(element.children[i], id) |
||||||
|
} |
||||||
|
return result |
||||||
|
} |
||||||
|
return null |
||||||
|
} |
||||||
Loading…
Reference in new issue