Browse Source
* [Improvement][dao]When I search for the keyword description, the web UI shows empty (#5952) * [Bug][WorkerServer] SqlTask NullPointerException #5549 * [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428 * [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428 * [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428 * [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428 * [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428 * [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428 * [Improvement][dao]When I search for the keyword description, the web UI shows empty #5428 * fix the readme typing issue (#5998) * Fix unchecked type conversions * Use indentation level reported by checkstyle * Reorganize CI workflows to fasten the wasted time and resources (#6011) * Add standalone server module to make it easier to develop (#6022) * Task node of SWITCH (#5939) * [Feature-#5273][server-master] Task node of SWITCH (#5922) Co-authored-by: wangxj <wangxj31> * remove description of bonecp (#6030) Co-authored-by: shaojwu <shaojwu@ebay.com> * [Improvement][Api Module]split alert group list-paging interface (#5941) * [Improvement][Api Module]split alert group list-paging interface * [FIX-#6007]Wrong complement date (#6026) * [FIX-#6007]Wrong complement date * [style]Wrong complement date * [Improvement-6024][dist] Remove useless packaging commands (#6029) ·Remove useless packaging commands in dolphinscheduler-bin.xml This closes #6024 Co-authored-by: mask <liuhu@zhiyoutec.com> * [FIX-5908][MasterServer] When executing an compensation task, the execution thread would have a NPE (#5909) * fix the npe in MasterExec * fix the compile error * Add `.asf.yaml` to easily set the GitHub metadata (#6035) * fix dead server cannot stop (#6046) * Enhancement Translation (#6042) * replaced Loading... with i18n * modified Edit zh_CN translation * Delete zh_CN.js Co-authored-by: David <dailidong66@gmail.com> * fix bug #6053 zh_CN.js is lost * [Fix-6038][ui] width of "SQL Statement" in Dag FormLineModal will be shrunk if sql line is too long (#6040) This closes #6038 * [Improvement] Fix inefficient map iterator (#6004) * Fix inefficient map iterator * Use forEach and remove call to valueOf * Modify AbstractParameters * Enhance `StandaloneServer` so that we don't need to update the version number manually (#6074) * Remove invalid character in `.asf.yaml` (#6075) * Remove invalid character `\n` in `.asf.yaml` (#6077) It turns out that the invalid character is `\n` * Add alert server into standalone-server as well and some minor polish (#6087) * Support starting standalone server in Docker image (#6102) Also remove unused class * [Feature-4355][Master-Worker-API] improvements of master and scheduler module (#6095) * [Feature-4355][Master-Worker-API] improvements of master and scheduler module (#6085) * master refactor: 1. spi for task submit and other actions(pause, kill) 2. remove threads for process instance and task instance. 3. add events for process instance and task instance * ut npe * add try catch * code style * fix critical bugs * fix critical bugs * fix critical bugs * fix critical bugs * Remove unused params in SwitchTaskTest (#6109) * [Feature-5987][Server] Support to set multiple environment configs for a certain worker. (#6082) * support multi environments * add some test cases * add an environment vue component * improve environment form * improve environment form * add environment worker group relation * add environment worker group relation * add the environment choice for formModel * set an environment for the task * modify the modal form of starting process * add the environment config to TaskExecutionContext * add the environment config to the timing form * fix conflicts * fix issues of the code style * fix some issues of the code style * fix some issues of the code style * fix some issues of the code style * fix some issues of the code style * fix some issues of the code style * fix some bugs in the code review * add the same table and columns to support H2 * fix some bugs * [Plugin][Task]Task Spi * [Plugin][Task]Task Spi * [Plugin][Task]Task Spi * [Plugin][Task]Task Spi * [Plugin][Task]Task Spi * [Plugin][Task]Task Spi Co-authored-by: didiaode18 <563646039@qq.com> Co-authored-by: Roy <yongjuncao1213@gmail.com> Co-authored-by: lyxell <alyxell@kth.se> Co-authored-by: Wenjun Ruan <wenjun@apache.org> Co-authored-by: kezhenxu94 <kezhenxu94@apache.org> Co-authored-by: myangle1120 <942542838@qq.com> Co-authored-by: wangxj3 <857234426@qq.com> Co-authored-by: gabry.wu <gabrywu@apache.org> Co-authored-by: shaojwu <shaojwu@ebay.com> Co-authored-by: Shukun Zhang <60541766+andream7@users.noreply.github.com> Co-authored-by: linquan <1175687813@qq.com> Co-authored-by: mask <39329477+Narcasserun@users.noreply.github.com> Co-authored-by: mask <liuhu@zhiyoutec.com> Co-authored-by: kyoty <echohlne@gmail.com> Co-authored-by: RichardStark <49977764+RichardStark@users.noreply.github.com> Co-authored-by: David <dailidong66@gmail.com> Co-authored-by: lenboo <baoliang.leon@gmail.com> Co-authored-by: lilyzhou <lj_zhou@outlook.com> Co-authored-by: OS <29528966+lenboo@users.noreply.github.com> Co-authored-by: junfan.zhang <zuston.shacha@gmail.com> Co-authored-by: Hua Jiang <jianghuachinacom@163.com>2.0.7-release
Kirs
3 years ago
committed by
GitHub
169 changed files with 8494 additions and 2194 deletions
@ -1,60 +0,0 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.alert.utils; |
||||
|
||||
import static org.junit.Assert.assertEquals; |
||||
import static org.junit.Assert.assertNull; |
||||
|
||||
import java.util.Arrays; |
||||
|
||||
import org.junit.Test; |
||||
import org.slf4j.Logger; |
||||
import org.slf4j.LoggerFactory; |
||||
|
||||
public class FuncUtilsTest { |
||||
|
||||
private static final Logger logger = LoggerFactory.getLogger(FuncUtilsTest.class); |
||||
|
||||
/** |
||||
* Test mkString |
||||
*/ |
||||
@Test |
||||
public void testMKString() { |
||||
|
||||
//Define users list
|
||||
Iterable<String> users = Arrays.asList("user1", "user2", "user3"); |
||||
//Define split
|
||||
String split = "|"; |
||||
|
||||
//Invoke mkString with correctParams
|
||||
String result = FuncUtils.mkString(users, split); |
||||
logger.info(result); |
||||
|
||||
//Expected result string
|
||||
assertEquals("user1|user2|user3", result); |
||||
|
||||
//Null list expected return null
|
||||
result = FuncUtils.mkString(null, split); |
||||
assertNull(result); |
||||
|
||||
//Null split expected return null
|
||||
result = FuncUtils.mkString(users, null); |
||||
assertNull(result); |
||||
|
||||
} |
||||
} |
||||
@ -0,0 +1,240 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.api.controller; |
||||
|
||||
import static org.apache.dolphinscheduler.api.enums.Status.CREATE_ENVIRONMENT_ERROR; |
||||
import static org.apache.dolphinscheduler.api.enums.Status.DELETE_ENVIRONMENT_ERROR; |
||||
import static org.apache.dolphinscheduler.api.enums.Status.QUERY_ENVIRONMENT_BY_CODE_ERROR; |
||||
import static org.apache.dolphinscheduler.api.enums.Status.QUERY_ENVIRONMENT_ERROR; |
||||
import static org.apache.dolphinscheduler.api.enums.Status.UPDATE_ENVIRONMENT_ERROR; |
||||
import static org.apache.dolphinscheduler.api.enums.Status.VERIFY_ENVIRONMENT_ERROR; |
||||
|
||||
import org.apache.dolphinscheduler.api.aspect.AccessLogAnnotation; |
||||
import org.apache.dolphinscheduler.api.exceptions.ApiException; |
||||
import org.apache.dolphinscheduler.api.service.EnvironmentService; |
||||
import org.apache.dolphinscheduler.api.utils.Result; |
||||
import org.apache.dolphinscheduler.common.Constants; |
||||
import org.apache.dolphinscheduler.common.utils.ParameterUtils; |
||||
import org.apache.dolphinscheduler.dao.entity.User; |
||||
|
||||
import java.util.Map; |
||||
|
||||
import org.springframework.beans.factory.annotation.Autowired; |
||||
import org.springframework.http.HttpStatus; |
||||
import org.springframework.web.bind.annotation.GetMapping; |
||||
import org.springframework.web.bind.annotation.PostMapping; |
||||
import org.springframework.web.bind.annotation.RequestAttribute; |
||||
import org.springframework.web.bind.annotation.RequestMapping; |
||||
import org.springframework.web.bind.annotation.RequestParam; |
||||
import org.springframework.web.bind.annotation.ResponseStatus; |
||||
import org.springframework.web.bind.annotation.RestController; |
||||
|
||||
import io.swagger.annotations.Api; |
||||
import io.swagger.annotations.ApiImplicitParam; |
||||
import io.swagger.annotations.ApiImplicitParams; |
||||
import io.swagger.annotations.ApiOperation; |
||||
import springfox.documentation.annotations.ApiIgnore; |
||||
|
||||
/** |
||||
* environment controller |
||||
*/ |
||||
@Api(tags = "ENVIRONMENT_TAG") |
||||
@RestController |
||||
@RequestMapping("environment") |
||||
public class EnvironmentController extends BaseController { |
||||
|
||||
@Autowired |
||||
private EnvironmentService environmentService; |
||||
|
||||
/** |
||||
* create environment |
||||
* |
||||
* @param loginUser login user |
||||
* @param name environment name |
||||
* @param config config |
||||
* @param description description |
||||
* @return returns an error if it exists |
||||
*/ |
||||
@ApiOperation(value = "createEnvironment", notes = "CREATE_ENVIRONMENT_NOTES") |
||||
@ApiImplicitParams({ |
||||
@ApiImplicitParam(name = "name", value = "ENVIRONMENT_NAME", required = true, dataType = "String"), |
||||
@ApiImplicitParam(name = "config", value = "CONFIG", required = true, dataType = "String"), |
||||
@ApiImplicitParam(name = "description", value = "ENVIRONMENT_DESC", dataType = "String"), |
||||
@ApiImplicitParam(name = "workerGroups", value = "WORKER_GROUP_LIST", dataType = "String") |
||||
}) |
||||
@PostMapping(value = "/create") |
||||
@ResponseStatus(HttpStatus.CREATED) |
||||
@ApiException(CREATE_ENVIRONMENT_ERROR) |
||||
@AccessLogAnnotation(ignoreRequestArgs = "loginUser") |
||||
public Result createProject(@ApiIgnore @RequestAttribute(value = Constants.SESSION_USER) User loginUser, |
||||
@RequestParam("name") String name, |
||||
@RequestParam("config") String config, |
||||
@RequestParam(value = "description", required = false) String description, |
||||
@RequestParam(value = "workerGroups", required = false) String workerGroups) { |
||||
|
||||
Map<String, Object> result = environmentService.createEnvironment(loginUser, name, config, description, workerGroups); |
||||
return returnDataList(result); |
||||
} |
||||
|
||||
/** |
||||
* update environment |
||||
* |
||||
* @param loginUser login user |
||||
* @param code environment code |
||||
* @param name environment name |
||||
* @param config environment config |
||||
* @param description description |
||||
* @return update result code |
||||
*/ |
||||
@ApiOperation(value = "updateEnvironment", notes = "UPDATE_ENVIRONMENT_NOTES") |
||||
@ApiImplicitParams({ |
||||
@ApiImplicitParam(name = "code", value = "ENVIRONMENT_CODE", required = true, dataType = "Long", example = "100"), |
||||
@ApiImplicitParam(name = "name", value = "ENVIRONMENT_NAME", required = true, dataType = "String"), |
||||
@ApiImplicitParam(name = "config", value = "ENVIRONMENT_CONFIG", required = true, dataType = "String"), |
||||
@ApiImplicitParam(name = "description", value = "ENVIRONMENT_DESC", dataType = "String"), |
||||
@ApiImplicitParam(name = "workerGroups", value = "WORKER_GROUP_LIST", dataType = "String") |
||||
}) |
||||
@PostMapping(value = "/update") |
||||
@ResponseStatus(HttpStatus.OK) |
||||
@ApiException(UPDATE_ENVIRONMENT_ERROR) |
||||
@AccessLogAnnotation(ignoreRequestArgs = "loginUser") |
||||
public Result updateEnvironment(@ApiIgnore @RequestAttribute(value = Constants.SESSION_USER) User loginUser, |
||||
@RequestParam("code") Long code, |
||||
@RequestParam("name") String name, |
||||
@RequestParam("config") String config, |
||||
@RequestParam(value = "description", required = false) String description, |
||||
@RequestParam(value = "workerGroups", required = false) String workerGroups) { |
||||
Map<String, Object> result = environmentService.updateEnvironmentByCode(loginUser, code, name, config, description, workerGroups); |
||||
return returnDataList(result); |
||||
} |
||||
|
||||
/** |
||||
* query environment details by code |
||||
* |
||||
* @param environmentCode environment code |
||||
* @return environment detail information |
||||
*/ |
||||
@ApiOperation(value = "queryEnvironmentByCode", notes = "QUERY_ENVIRONMENT_BY_CODE_NOTES") |
||||
@ApiImplicitParams({ |
||||
@ApiImplicitParam(name = "environmentCode", value = "ENVIRONMENT_CODE", required = true, dataType = "Long", example = "100") |
||||
}) |
||||
@GetMapping(value = "/query-by-code") |
||||
@ResponseStatus(HttpStatus.OK) |
||||
@ApiException(QUERY_ENVIRONMENT_BY_CODE_ERROR) |
||||
@AccessLogAnnotation(ignoreRequestArgs = "loginUser") |
||||
public Result queryEnvironmentByCode(@ApiIgnore @RequestAttribute(value = Constants.SESSION_USER) User loginUser, |
||||
@RequestParam("environmentCode") Long environmentCode) { |
||||
|
||||
Map<String, Object> result = environmentService.queryEnvironmentByCode(environmentCode); |
||||
return returnDataList(result); |
||||
} |
||||
|
||||
/** |
||||
* query environment list paging |
||||
* |
||||
* @param searchVal search value |
||||
* @param pageSize page size |
||||
* @param pageNo page number |
||||
* @return environment list which the login user have permission to see |
||||
*/ |
||||
@ApiOperation(value = "queryEnvironmentListPaging", notes = "QUERY_ENVIRONMENT_LIST_PAGING_NOTES") |
||||
@ApiImplicitParams({ |
||||
@ApiImplicitParam(name = "searchVal", value = "SEARCH_VAL", dataType = "String"), |
||||
@ApiImplicitParam(name = "pageSize", value = "PAGE_SIZE", required = true, dataType = "Int", example = "20"), |
||||
@ApiImplicitParam(name = "pageNo", value = "PAGE_NO", required = true, dataType = "Int", example = "1") |
||||
}) |
||||
@GetMapping(value = "/list-paging") |
||||
@ResponseStatus(HttpStatus.OK) |
||||
@ApiException(QUERY_ENVIRONMENT_ERROR) |
||||
@AccessLogAnnotation(ignoreRequestArgs = "loginUser") |
||||
public Result queryEnvironmentListPaging(@ApiIgnore @RequestAttribute(value = Constants.SESSION_USER) User loginUser, |
||||
@RequestParam(value = "searchVal", required = false) String searchVal, |
||||
@RequestParam("pageSize") Integer pageSize, |
||||
@RequestParam("pageNo") Integer pageNo |
||||
) { |
||||
|
||||
Result result = checkPageParams(pageNo, pageSize); |
||||
if (!result.checkResult()) { |
||||
return result; |
||||
} |
||||
searchVal = ParameterUtils.handleEscapes(searchVal); |
||||
result = environmentService.queryEnvironmentListPaging(pageNo, pageSize, searchVal); |
||||
return result; |
||||
} |
||||
|
||||
/** |
||||
* delete environment by code |
||||
* |
||||
* @param loginUser login user |
||||
* @param environmentCode environment code |
||||
* @return delete result code |
||||
*/ |
||||
@ApiOperation(value = "deleteEnvironmentByCode", notes = "DELETE_ENVIRONMENT_BY_CODE_NOTES") |
||||
@ApiImplicitParams({ |
||||
@ApiImplicitParam(name = "environmentCode", value = "ENVIRONMENT_CODE", required = true, dataType = "Long", example = "100") |
||||
}) |
||||
@PostMapping(value = "/delete") |
||||
@ResponseStatus(HttpStatus.OK) |
||||
@ApiException(DELETE_ENVIRONMENT_ERROR) |
||||
@AccessLogAnnotation(ignoreRequestArgs = "loginUser") |
||||
public Result deleteEnvironment(@ApiIgnore @RequestAttribute(value = Constants.SESSION_USER) User loginUser, |
||||
@RequestParam("environmentCode") Long environmentCode |
||||
) { |
||||
|
||||
Map<String, Object> result = environmentService.deleteEnvironmentByCode(loginUser, environmentCode); |
||||
return returnDataList(result); |
||||
} |
||||
|
||||
/** |
||||
* query all environment list |
||||
* |
||||
* @param loginUser login user |
||||
* @return all environment list |
||||
*/ |
||||
@ApiOperation(value = "queryAllEnvironmentList", notes = "QUERY_ALL_ENVIRONMENT_LIST_NOTES") |
||||
@GetMapping(value = "/query-environment-list") |
||||
@ResponseStatus(HttpStatus.OK) |
||||
@ApiException(QUERY_ENVIRONMENT_ERROR) |
||||
@AccessLogAnnotation(ignoreRequestArgs = "loginUser") |
||||
public Result queryAllEnvironmentList(@ApiIgnore @RequestAttribute(value = Constants.SESSION_USER) User loginUser) { |
||||
Map<String, Object> result = environmentService.queryAllEnvironmentList(); |
||||
return returnDataList(result); |
||||
} |
||||
|
||||
/** |
||||
* verify environment and environment name |
||||
* |
||||
* @param loginUser login user |
||||
* @param environmentName environment name |
||||
* @return true if the environment name not exists, otherwise return false |
||||
*/ |
||||
@ApiOperation(value = "verifyEnvironment", notes = "VERIFY_ENVIRONMENT_NOTES") |
||||
@ApiImplicitParams({ |
||||
@ApiImplicitParam(name = "environmentName", value = "ENVIRONMENT_NAME", required = true, dataType = "String") |
||||
}) |
||||
@PostMapping(value = "/verify-environment") |
||||
@ResponseStatus(HttpStatus.OK) |
||||
@ApiException(VERIFY_ENVIRONMENT_ERROR) |
||||
@AccessLogAnnotation(ignoreRequestArgs = "loginUser") |
||||
public Result verifyEnvironment(@ApiIgnore @RequestAttribute(value = Constants.SESSION_USER) User loginUser, |
||||
@RequestParam(value = "environmentName") String environmentName |
||||
) { |
||||
Map<String, Object> result = environmentService.verifyEnvironment(environmentName); |
||||
return returnDataList(result); |
||||
} |
||||
} |
||||
@ -0,0 +1,129 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.api.dto; |
||||
|
||||
import java.util.Date; |
||||
import java.util.List; |
||||
|
||||
/** |
||||
* EnvironmentDto |
||||
*/ |
||||
public class EnvironmentDto { |
||||
|
||||
private int id; |
||||
|
||||
/** |
||||
* environment code |
||||
*/ |
||||
private Long code; |
||||
|
||||
/** |
||||
* environment name |
||||
*/ |
||||
private String name; |
||||
|
||||
/** |
||||
* config content |
||||
*/ |
||||
private String config; |
||||
|
||||
private String description; |
||||
|
||||
private List<String> workerGroups; |
||||
|
||||
/** |
||||
* operator user id |
||||
*/ |
||||
private Integer operator; |
||||
|
||||
private Date createTime; |
||||
|
||||
private Date updateTime; |
||||
|
||||
public int getId() { |
||||
return id; |
||||
} |
||||
|
||||
public void setId(int id) { |
||||
this.id = id; |
||||
} |
||||
|
||||
public String getName() { |
||||
return name; |
||||
} |
||||
|
||||
public void setName(String name) { |
||||
this.name = name; |
||||
} |
||||
|
||||
public Long getCode() { |
||||
return this.code; |
||||
} |
||||
|
||||
public void setCode(Long code) { |
||||
this.code = code; |
||||
} |
||||
|
||||
public String getConfig() { |
||||
return this.config; |
||||
} |
||||
|
||||
public void setConfig(String config) { |
||||
this.config = config; |
||||
} |
||||
|
||||
public String getDescription() { |
||||
return this.description; |
||||
} |
||||
|
||||
public void setDescription(String description) { |
||||
this.description = description; |
||||
} |
||||
|
||||
public Integer getOperator() { |
||||
return this.operator; |
||||
} |
||||
|
||||
public void setOperator(Integer operator) { |
||||
this.operator = operator; |
||||
} |
||||
|
||||
public Date getCreateTime() { |
||||
return createTime; |
||||
} |
||||
|
||||
public void setCreateTime(Date createTime) { |
||||
this.createTime = createTime; |
||||
} |
||||
|
||||
public Date getUpdateTime() { |
||||
return updateTime; |
||||
} |
||||
|
||||
public void setUpdateTime(Date updateTime) { |
||||
this.updateTime = updateTime; |
||||
} |
||||
|
||||
public List<String> getWorkerGroups() { |
||||
return workerGroups; |
||||
} |
||||
|
||||
public void setWorkerGroups(List<String> workerGroups) { |
||||
this.workerGroups = workerGroups; |
||||
} |
||||
} |
||||
@ -0,0 +1,102 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.api.service; |
||||
|
||||
import org.apache.dolphinscheduler.api.utils.Result; |
||||
import org.apache.dolphinscheduler.dao.entity.User; |
||||
|
||||
import java.util.Map; |
||||
|
||||
/** |
||||
* environment service |
||||
*/ |
||||
public interface EnvironmentService { |
||||
|
||||
/** |
||||
* create environment |
||||
* |
||||
* @param loginUser login user |
||||
* @param name environment name |
||||
* @param config environment config |
||||
* @param desc environment desc |
||||
* @param workerGroups worker groups |
||||
*/ |
||||
Map<String, Object> createEnvironment(User loginUser, String name, String config, String desc, String workerGroups); |
||||
|
||||
/** |
||||
* query environment |
||||
* |
||||
* @param name environment name |
||||
*/ |
||||
Map<String, Object> queryEnvironmentByName(String name); |
||||
|
||||
/** |
||||
* query environment |
||||
* |
||||
* @param code environment code |
||||
*/ |
||||
Map<String, Object> queryEnvironmentByCode(Long code); |
||||
|
||||
|
||||
/** |
||||
* delete environment |
||||
* |
||||
* @param loginUser login user |
||||
* @param code environment code |
||||
*/ |
||||
Map<String, Object> deleteEnvironmentByCode(User loginUser, Long code); |
||||
|
||||
/** |
||||
* update environment |
||||
* |
||||
* @param loginUser login user |
||||
* @param code environment code |
||||
* @param name environment name |
||||
* @param config environment config |
||||
* @param desc environment desc |
||||
* @param workerGroups worker groups |
||||
*/ |
||||
Map<String, Object> updateEnvironmentByCode(User loginUser, Long code, String name, String config, String desc, String workerGroups); |
||||
|
||||
/** |
||||
* query environment paging |
||||
* |
||||
* @param pageNo page number |
||||
* @param searchVal search value |
||||
* @param pageSize page size |
||||
* @return environment list page |
||||
*/ |
||||
Result queryEnvironmentListPaging(Integer pageNo, Integer pageSize, String searchVal); |
||||
|
||||
/** |
||||
* query all environment |
||||
* |
||||
* @return all environment list |
||||
*/ |
||||
Map<String, Object> queryAllEnvironmentList(); |
||||
|
||||
/** |
||||
* verify environment name |
||||
* |
||||
* @param environmentName environment name |
||||
* @return true if the environment name not exists, otherwise return false |
||||
*/ |
||||
Map<String, Object> verifyEnvironment(String environmentName); |
||||
|
||||
} |
||||
|
||||
@ -0,0 +1,41 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.api.service; |
||||
|
||||
import java.util.Map; |
||||
|
||||
/** |
||||
* environment worker group relation service |
||||
*/ |
||||
public interface EnvironmentWorkerGroupRelationService { |
||||
|
||||
/** |
||||
* query environment worker group relation |
||||
* |
||||
* @param environmentCode environment code |
||||
*/ |
||||
Map<String, Object> queryEnvironmentWorkerGroupRelation(Long environmentCode); |
||||
|
||||
/** |
||||
* query all environment worker group relation |
||||
* |
||||
* @return all relation list |
||||
*/ |
||||
Map<String, Object> queryAllEnvironmentWorkerGroupRelationList(); |
||||
} |
||||
|
||||
@ -0,0 +1,463 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.api.service.impl; |
||||
|
||||
import org.apache.dolphinscheduler.api.dto.EnvironmentDto; |
||||
import org.apache.dolphinscheduler.api.enums.Status; |
||||
import org.apache.dolphinscheduler.api.service.EnvironmentService; |
||||
import org.apache.dolphinscheduler.api.utils.PageInfo; |
||||
import org.apache.dolphinscheduler.api.utils.Result; |
||||
import org.apache.dolphinscheduler.common.Constants; |
||||
import org.apache.dolphinscheduler.common.utils.CollectionUtils; |
||||
import org.apache.dolphinscheduler.common.utils.JSONUtils; |
||||
import org.apache.dolphinscheduler.common.utils.SnowFlakeUtils; |
||||
import org.apache.dolphinscheduler.common.utils.SnowFlakeUtils.SnowFlakeException; |
||||
import org.apache.dolphinscheduler.common.utils.StringUtils; |
||||
import org.apache.dolphinscheduler.dao.entity.Environment; |
||||
import org.apache.dolphinscheduler.dao.entity.EnvironmentWorkerGroupRelation; |
||||
import org.apache.dolphinscheduler.dao.entity.TaskDefinition; |
||||
import org.apache.dolphinscheduler.dao.entity.User; |
||||
import org.apache.dolphinscheduler.dao.mapper.EnvironmentMapper; |
||||
import org.apache.dolphinscheduler.dao.mapper.EnvironmentWorkerGroupRelationMapper; |
||||
import org.apache.dolphinscheduler.dao.mapper.TaskDefinitionMapper; |
||||
|
||||
import org.apache.commons.collections4.SetUtils; |
||||
|
||||
import java.util.ArrayList; |
||||
import java.util.Date; |
||||
import java.util.HashMap; |
||||
import java.util.List; |
||||
import java.util.Map; |
||||
import java.util.Objects; |
||||
import java.util.Set; |
||||
import java.util.TreeSet; |
||||
import java.util.stream.Collectors; |
||||
|
||||
import org.slf4j.Logger; |
||||
import org.slf4j.LoggerFactory; |
||||
import org.springframework.beans.BeanUtils; |
||||
import org.springframework.beans.factory.annotation.Autowired; |
||||
import org.springframework.stereotype.Service; |
||||
import org.springframework.transaction.annotation.Transactional; |
||||
|
||||
import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper; |
||||
import com.baomidou.mybatisplus.core.conditions.update.UpdateWrapper; |
||||
import com.baomidou.mybatisplus.core.metadata.IPage; |
||||
import com.baomidou.mybatisplus.extension.plugins.pagination.Page; |
||||
import com.fasterxml.jackson.core.type.TypeReference; |
||||
|
||||
/** |
||||
* task definition service impl |
||||
*/ |
||||
@Service |
||||
public class EnvironmentServiceImpl extends BaseServiceImpl implements EnvironmentService { |
||||
|
||||
private static final Logger logger = LoggerFactory.getLogger(EnvironmentServiceImpl.class); |
||||
|
||||
@Autowired |
||||
private EnvironmentMapper environmentMapper; |
||||
|
||||
@Autowired |
||||
private EnvironmentWorkerGroupRelationMapper relationMapper; |
||||
|
||||
@Autowired |
||||
private TaskDefinitionMapper taskDefinitionMapper; |
||||
|
||||
/** |
||||
* create environment |
||||
* |
||||
* @param loginUser login user |
||||
* @param name environment name |
||||
* @param config environment config |
||||
* @param desc environment desc |
||||
* @param workerGroups worker groups |
||||
*/ |
||||
@Transactional(rollbackFor = RuntimeException.class) |
||||
@Override |
||||
public Map<String, Object> createEnvironment(User loginUser, String name, String config, String desc, String workerGroups) { |
||||
Map<String, Object> result = new HashMap<>(); |
||||
if (isNotAdmin(loginUser, result)) { |
||||
return result; |
||||
} |
||||
|
||||
Map<String, Object> checkResult = checkParams(name,config,workerGroups); |
||||
if (checkResult.get(Constants.STATUS) != Status.SUCCESS) { |
||||
return checkResult; |
||||
} |
||||
|
||||
Environment environment = environmentMapper.queryByEnvironmentName(name); |
||||
if (environment != null) { |
||||
putMsg(result, Status.ENVIRONMENT_NAME_EXISTS, name); |
||||
return result; |
||||
} |
||||
|
||||
Environment env = new Environment(); |
||||
env.setName(name); |
||||
env.setConfig(config); |
||||
env.setDescription(desc); |
||||
env.setOperator(loginUser.getId()); |
||||
env.setCreateTime(new Date()); |
||||
env.setUpdateTime(new Date()); |
||||
long code = 0L; |
||||
try { |
||||
code = SnowFlakeUtils.getInstance().nextId(); |
||||
env.setCode(code); |
||||
} catch (SnowFlakeException e) { |
||||
logger.error("Environment code get error, ", e); |
||||
} |
||||
if (code == 0L) { |
||||
putMsg(result, Status.INTERNAL_SERVER_ERROR_ARGS, "Error generating environment code"); |
||||
return result; |
||||
} |
||||
|
||||
if (environmentMapper.insert(env) > 0) { |
||||
if (StringUtils.isNotEmpty(workerGroups)) { |
||||
List<String> workerGroupList = JSONUtils.parseObject(workerGroups, new TypeReference<List<String>>(){}); |
||||
if (CollectionUtils.isNotEmpty(workerGroupList)) { |
||||
workerGroupList.stream().forEach(workerGroup -> { |
||||
if (StringUtils.isNotEmpty(workerGroup)) { |
||||
EnvironmentWorkerGroupRelation relation = new EnvironmentWorkerGroupRelation(); |
||||
relation.setEnvironmentCode(env.getCode()); |
||||
relation.setWorkerGroup(workerGroup); |
||||
relation.setOperator(loginUser.getId()); |
||||
relation.setCreateTime(new Date()); |
||||
relation.setUpdateTime(new Date()); |
||||
relationMapper.insert(relation); |
||||
} |
||||
}); |
||||
} |
||||
} |
||||
result.put(Constants.DATA_LIST, env.getCode()); |
||||
putMsg(result, Status.SUCCESS); |
||||
} else { |
||||
putMsg(result, Status.CREATE_ENVIRONMENT_ERROR); |
||||
} |
||||
return result; |
||||
} |
||||
|
||||
/** |
||||
* query environment paging |
||||
* |
||||
* @param pageNo page number |
||||
* @param searchVal search value |
||||
* @param pageSize page size |
||||
* @return environment list page |
||||
*/ |
||||
@Override |
||||
public Result queryEnvironmentListPaging(Integer pageNo, Integer pageSize, String searchVal) { |
||||
Result result = new Result(); |
||||
|
||||
Page<Environment> page = new Page<>(pageNo, pageSize); |
||||
|
||||
IPage<Environment> environmentIPage = environmentMapper.queryEnvironmentListPaging(page, searchVal); |
||||
|
||||
PageInfo<EnvironmentDto> pageInfo = new PageInfo<>(pageNo, pageSize); |
||||
pageInfo.setTotal((int) environmentIPage.getTotal()); |
||||
|
||||
if (CollectionUtils.isNotEmpty(environmentIPage.getRecords())) { |
||||
Map<Long, List<String>> relationMap = relationMapper.selectList(null).stream() |
||||
.collect(Collectors.groupingBy(EnvironmentWorkerGroupRelation::getEnvironmentCode,Collectors.mapping(EnvironmentWorkerGroupRelation::getWorkerGroup,Collectors.toList()))); |
||||
|
||||
List<EnvironmentDto> dtoList = environmentIPage.getRecords().stream().map(environment -> { |
||||
EnvironmentDto dto = new EnvironmentDto(); |
||||
BeanUtils.copyProperties(environment,dto); |
||||
List<String> workerGroups = relationMap.getOrDefault(environment.getCode(),new ArrayList<String>()); |
||||
dto.setWorkerGroups(workerGroups); |
||||
return dto; |
||||
}).collect(Collectors.toList()); |
||||
|
||||
pageInfo.setTotalList(dtoList); |
||||
} else { |
||||
pageInfo.setTotalList(new ArrayList<>()); |
||||
} |
||||
|
||||
result.setData(pageInfo); |
||||
putMsg(result, Status.SUCCESS); |
||||
return result; |
||||
} |
||||
|
||||
/** |
||||
* query all environment |
||||
* |
||||
* @return all environment list |
||||
*/ |
||||
@Override |
||||
public Map<String, Object> queryAllEnvironmentList() { |
||||
Map<String,Object> result = new HashMap<>(); |
||||
List<Environment> environmentList = environmentMapper.queryAllEnvironmentList(); |
||||
|
||||
if (CollectionUtils.isNotEmpty(environmentList)) { |
||||
Map<Long, List<String>> relationMap = relationMapper.selectList(null).stream() |
||||
.collect(Collectors.groupingBy(EnvironmentWorkerGroupRelation::getEnvironmentCode,Collectors.mapping(EnvironmentWorkerGroupRelation::getWorkerGroup,Collectors.toList()))); |
||||
|
||||
List<EnvironmentDto> dtoList = environmentList.stream().map(environment -> { |
||||
EnvironmentDto dto = new EnvironmentDto(); |
||||
BeanUtils.copyProperties(environment,dto); |
||||
List<String> workerGroups = relationMap.getOrDefault(environment.getCode(),new ArrayList<String>()); |
||||
dto.setWorkerGroups(workerGroups); |
||||
return dto; |
||||
}).collect(Collectors.toList()); |
||||
result.put(Constants.DATA_LIST,dtoList); |
||||
} else { |
||||
result.put(Constants.DATA_LIST, new ArrayList<>()); |
||||
} |
||||
|
||||
putMsg(result,Status.SUCCESS); |
||||
return result; |
||||
} |
||||
|
||||
/** |
||||
* query environment |
||||
* |
||||
* @param code environment code |
||||
*/ |
||||
@Override |
||||
public Map<String, Object> queryEnvironmentByCode(Long code) { |
||||
Map<String, Object> result = new HashMap<>(); |
||||
|
||||
Environment env = environmentMapper.queryByEnvironmentCode(code); |
||||
|
||||
if (env == null) { |
||||
putMsg(result, Status.QUERY_ENVIRONMENT_BY_CODE_ERROR, code); |
||||
} else { |
||||
List<String> workerGroups = relationMapper.queryByEnvironmentCode(env.getCode()).stream() |
||||
.map(item -> item.getWorkerGroup()) |
||||
.collect(Collectors.toList()); |
||||
|
||||
EnvironmentDto dto = new EnvironmentDto(); |
||||
BeanUtils.copyProperties(env,dto); |
||||
dto.setWorkerGroups(workerGroups); |
||||
result.put(Constants.DATA_LIST, dto); |
||||
putMsg(result, Status.SUCCESS); |
||||
} |
||||
return result; |
||||
} |
||||
|
||||
/** |
||||
* query environment |
||||
* |
||||
* @param name environment name |
||||
*/ |
||||
@Override |
||||
public Map<String, Object> queryEnvironmentByName(String name) { |
||||
Map<String, Object> result = new HashMap<>(); |
||||
|
||||
Environment env = environmentMapper.queryByEnvironmentName(name); |
||||
if (env == null) { |
||||
putMsg(result, Status.QUERY_ENVIRONMENT_BY_NAME_ERROR, name); |
||||
} else { |
||||
List<String> workerGroups = relationMapper.queryByEnvironmentCode(env.getCode()).stream() |
||||
.map(item -> item.getWorkerGroup()) |
||||
.collect(Collectors.toList()); |
||||
|
||||
EnvironmentDto dto = new EnvironmentDto(); |
||||
BeanUtils.copyProperties(env,dto); |
||||
dto.setWorkerGroups(workerGroups); |
||||
result.put(Constants.DATA_LIST, dto); |
||||
putMsg(result, Status.SUCCESS); |
||||
} |
||||
return result; |
||||
} |
||||
|
||||
/** |
||||
* delete environment |
||||
* |
||||
* @param loginUser login user |
||||
* @param code environment code |
||||
*/ |
||||
@Transactional(rollbackFor = RuntimeException.class) |
||||
@Override |
||||
public Map<String, Object> deleteEnvironmentByCode(User loginUser, Long code) { |
||||
Map<String, Object> result = new HashMap<>(); |
||||
if (isNotAdmin(loginUser, result)) { |
||||
return result; |
||||
} |
||||
|
||||
Integer relatedTaskNumber = taskDefinitionMapper |
||||
.selectCount(new QueryWrapper<TaskDefinition>().lambda().eq(TaskDefinition::getEnvironmentCode,code)); |
||||
|
||||
if (relatedTaskNumber > 0) { |
||||
putMsg(result, Status.DELETE_ENVIRONMENT_RELATED_TASK_EXISTS); |
||||
return result; |
||||
} |
||||
|
||||
int delete = environmentMapper.deleteByCode(code); |
||||
if (delete > 0) { |
||||
relationMapper.delete(new QueryWrapper<EnvironmentWorkerGroupRelation>() |
||||
.lambda() |
||||
.eq(EnvironmentWorkerGroupRelation::getEnvironmentCode,code)); |
||||
putMsg(result, Status.SUCCESS); |
||||
} else { |
||||
putMsg(result, Status.DELETE_ENVIRONMENT_ERROR); |
||||
} |
||||
return result; |
||||
} |
||||
|
||||
/** |
||||
* update environment |
||||
* |
||||
* @param loginUser login user |
||||
* @param code environment code |
||||
* @param name environment name |
||||
* @param config environment config |
||||
* @param desc environment desc |
||||
* @param workerGroups worker groups |
||||
*/ |
||||
@Transactional(rollbackFor = RuntimeException.class) |
||||
@Override |
||||
public Map<String, Object> updateEnvironmentByCode(User loginUser, Long code, String name, String config, String desc, String workerGroups) { |
||||
Map<String, Object> result = new HashMap<>(); |
||||
if (isNotAdmin(loginUser, result)) { |
||||
return result; |
||||
} |
||||
|
||||
Map<String, Object> checkResult = checkParams(name,config,workerGroups); |
||||
if (checkResult.get(Constants.STATUS) != Status.SUCCESS) { |
||||
return checkResult; |
||||
} |
||||
|
||||
Environment environment = environmentMapper.queryByEnvironmentName(name); |
||||
if (environment != null && !environment.getCode().equals(code)) { |
||||
putMsg(result, Status.ENVIRONMENT_NAME_EXISTS, name); |
||||
return result; |
||||
} |
||||
|
||||
Set<String> workerGroupSet; |
||||
if (StringUtils.isNotEmpty(workerGroups)) { |
||||
workerGroupSet = JSONUtils.parseObject(workerGroups, new TypeReference<Set<String>>() {}); |
||||
} else { |
||||
workerGroupSet = new TreeSet<>(); |
||||
} |
||||
|
||||
Set<String> existWorkerGroupSet = relationMapper |
||||
.queryByEnvironmentCode(code) |
||||
.stream() |
||||
.map(item -> item.getWorkerGroup()) |
||||
.collect(Collectors.toSet()); |
||||
|
||||
Set<String> deleteWorkerGroupSet = SetUtils.difference(existWorkerGroupSet,workerGroupSet).toSet(); |
||||
Set<String> addWorkerGroupSet = SetUtils.difference(workerGroupSet,existWorkerGroupSet).toSet(); |
||||
|
||||
// verify whether the relation of this environment and worker groups can be adjusted
|
||||
checkResult = checkUsedEnvironmentWorkerGroupRelation(deleteWorkerGroupSet, name, code); |
||||
if (checkResult.get(Constants.STATUS) != Status.SUCCESS) { |
||||
return checkResult; |
||||
} |
||||
|
||||
Environment env = new Environment(); |
||||
env.setCode(code); |
||||
env.setName(name); |
||||
env.setConfig(config); |
||||
env.setDescription(desc); |
||||
env.setOperator(loginUser.getId()); |
||||
env.setUpdateTime(new Date()); |
||||
|
||||
int update = environmentMapper.update(env, new UpdateWrapper<Environment>().lambda().eq(Environment::getCode,code)); |
||||
if (update > 0) { |
||||
deleteWorkerGroupSet.stream().forEach(key -> { |
||||
if (StringUtils.isNotEmpty(key)) { |
||||
relationMapper.delete(new QueryWrapper<EnvironmentWorkerGroupRelation>() |
||||
.lambda() |
||||
.eq(EnvironmentWorkerGroupRelation::getEnvironmentCode,code)); |
||||
} |
||||
}); |
||||
addWorkerGroupSet.stream().forEach(key -> { |
||||
if (StringUtils.isNotEmpty(key)) { |
||||
EnvironmentWorkerGroupRelation relation = new EnvironmentWorkerGroupRelation(); |
||||
relation.setEnvironmentCode(code); |
||||
relation.setWorkerGroup(key); |
||||
relation.setUpdateTime(new Date()); |
||||
relation.setCreateTime(new Date()); |
||||
relation.setOperator(loginUser.getId()); |
||||
relationMapper.insert(relation); |
||||
} |
||||
}); |
||||
putMsg(result, Status.SUCCESS); |
||||
} else { |
||||
putMsg(result, Status.UPDATE_ENVIRONMENT_ERROR, name); |
||||
} |
||||
return result; |
||||
} |
||||
|
||||
|
||||
|
||||
/** |
||||
* verify environment name |
||||
* |
||||
* @param environmentName environment name |
||||
* @return true if the environment name not exists, otherwise return false |
||||
*/ |
||||
@Override |
||||
public Map<String, Object> verifyEnvironment(String environmentName) { |
||||
Map<String, Object> result = new HashMap<>(); |
||||
|
||||
if (StringUtils.isEmpty(environmentName)) { |
||||
putMsg(result, Status.ENVIRONMENT_NAME_IS_NULL); |
||||
return result; |
||||
} |
||||
|
||||
Environment environment = environmentMapper.queryByEnvironmentName(environmentName); |
||||
if (environment != null) { |
||||
putMsg(result, Status.ENVIRONMENT_NAME_EXISTS, environmentName); |
||||
return result; |
||||
} |
||||
|
||||
result.put(Constants.STATUS, Status.SUCCESS); |
||||
return result; |
||||
} |
||||
|
||||
private Map<String, Object> checkUsedEnvironmentWorkerGroupRelation(Set<String> deleteKeySet,String environmentName, Long environmentCode) { |
||||
Map<String, Object> result = new HashMap<>(); |
||||
for (String workerGroup : deleteKeySet) { |
||||
TaskDefinition taskDefinition = taskDefinitionMapper |
||||
.selectOne(new QueryWrapper<TaskDefinition>().lambda() |
||||
.eq(TaskDefinition::getEnvironmentCode,environmentCode) |
||||
.eq(TaskDefinition::getWorkerGroup,workerGroup)); |
||||
|
||||
if (Objects.nonNull(taskDefinition)) { |
||||
putMsg(result, Status.UPDATE_ENVIRONMENT_WORKER_GROUP_RELATION_ERROR,workerGroup,environmentName,taskDefinition.getName()); |
||||
return result; |
||||
} |
||||
} |
||||
result.put(Constants.STATUS, Status.SUCCESS); |
||||
return result; |
||||
} |
||||
|
||||
public Map<String, Object> checkParams(String name, String config, String workerGroups) { |
||||
Map<String, Object> result = new HashMap<>(); |
||||
if (StringUtils.isEmpty(name)) { |
||||
putMsg(result, Status.ENVIRONMENT_NAME_IS_NULL); |
||||
return result; |
||||
} |
||||
if (StringUtils.isEmpty(config)) { |
||||
putMsg(result, Status.ENVIRONMENT_CONFIG_IS_NULL); |
||||
return result; |
||||
} |
||||
if (StringUtils.isNotEmpty(workerGroups)) { |
||||
List<String> workerGroupList = JSONUtils.parseObject(workerGroups, new TypeReference<List<String>>(){}); |
||||
if (Objects.isNull(workerGroupList)) { |
||||
putMsg(result, Status.ENVIRONMENT_WORKER_GROUPS_IS_INVALID); |
||||
return result; |
||||
} |
||||
} |
||||
result.put(Constants.STATUS, Status.SUCCESS); |
||||
return result; |
||||
} |
||||
|
||||
} |
||||
|
||||
@ -0,0 +1,76 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.api.service.impl; |
||||
|
||||
import org.apache.dolphinscheduler.api.enums.Status; |
||||
import org.apache.dolphinscheduler.api.service.EnvironmentWorkerGroupRelationService; |
||||
import org.apache.dolphinscheduler.common.Constants; |
||||
import org.apache.dolphinscheduler.dao.entity.EnvironmentWorkerGroupRelation; |
||||
import org.apache.dolphinscheduler.dao.mapper.EnvironmentWorkerGroupRelationMapper; |
||||
|
||||
import java.util.HashMap; |
||||
import java.util.List; |
||||
import java.util.Map; |
||||
|
||||
import org.slf4j.Logger; |
||||
import org.slf4j.LoggerFactory; |
||||
import org.springframework.beans.factory.annotation.Autowired; |
||||
import org.springframework.stereotype.Service; |
||||
|
||||
/** |
||||
* task definition service impl |
||||
*/ |
||||
@Service |
||||
public class EnvironmentWorkerGroupRelationServiceImpl extends BaseServiceImpl implements |
||||
EnvironmentWorkerGroupRelationService { |
||||
|
||||
private static final Logger logger = LoggerFactory.getLogger(EnvironmentWorkerGroupRelationServiceImpl.class); |
||||
|
||||
@Autowired |
||||
private EnvironmentWorkerGroupRelationMapper environmentWorkerGroupRelationMapper; |
||||
|
||||
/** |
||||
* query environment worker group relation |
||||
* |
||||
* @param environmentCode environment code |
||||
*/ |
||||
@Override |
||||
public Map<String, Object> queryEnvironmentWorkerGroupRelation(Long environmentCode) { |
||||
Map<String, Object> result = new HashMap<>(); |
||||
List<EnvironmentWorkerGroupRelation> relations = environmentWorkerGroupRelationMapper.queryByEnvironmentCode(environmentCode); |
||||
result.put(Constants.DATA_LIST, relations); |
||||
putMsg(result, Status.SUCCESS); |
||||
return result; |
||||
} |
||||
|
||||
/** |
||||
* query all environment worker group relation |
||||
* |
||||
* @return all relation list |
||||
*/ |
||||
@Override |
||||
public Map<String, Object> queryAllEnvironmentWorkerGroupRelationList() { |
||||
Map<String, Object> result = new HashMap<>(); |
||||
|
||||
List<EnvironmentWorkerGroupRelation> relations = environmentWorkerGroupRelationMapper.selectList(null); |
||||
|
||||
result.put(Constants.DATA_LIST,relations); |
||||
putMsg(result,Status.SUCCESS); |
||||
return result; |
||||
} |
||||
} |
||||
@ -0,0 +1,208 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.api.controller; |
||||
|
||||
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get; |
||||
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.post; |
||||
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.content; |
||||
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status; |
||||
|
||||
import org.apache.dolphinscheduler.api.enums.Status; |
||||
import org.apache.dolphinscheduler.api.utils.Result; |
||||
import org.apache.dolphinscheduler.common.utils.JSONUtils; |
||||
import org.apache.dolphinscheduler.common.utils.Preconditions; |
||||
|
||||
import org.junit.After; |
||||
import org.junit.Assert; |
||||
import org.junit.Before; |
||||
import org.junit.Test; |
||||
import org.slf4j.Logger; |
||||
import org.slf4j.LoggerFactory; |
||||
import org.springframework.http.MediaType; |
||||
import org.springframework.test.web.servlet.MvcResult; |
||||
import org.springframework.util.LinkedMultiValueMap; |
||||
import org.springframework.util.MultiValueMap; |
||||
|
||||
import com.fasterxml.jackson.core.type.TypeReference; |
||||
|
||||
/** |
||||
* environment controller test |
||||
*/ |
||||
public class EnvironmentControllerTest extends AbstractControllerTest { |
||||
|
||||
private static Logger logger = LoggerFactory.getLogger(EnvironmentControllerTest.class); |
||||
|
||||
private String environmentCode; |
||||
|
||||
public static final String environmentName = "Env1"; |
||||
|
||||
public static final String config = "this is config content"; |
||||
|
||||
public static final String desc = "this is environment description"; |
||||
|
||||
@Before |
||||
public void before() throws Exception { |
||||
testCreateEnvironment(); |
||||
} |
||||
|
||||
@After |
||||
public void after() throws Exception { |
||||
testDeleteEnvironment(); |
||||
} |
||||
|
||||
public void testCreateEnvironment() throws Exception { |
||||
|
||||
MultiValueMap<String, String> paramsMap = new LinkedMultiValueMap<>(); |
||||
paramsMap.add("name",environmentName); |
||||
paramsMap.add("config",config); |
||||
paramsMap.add("description",desc); |
||||
|
||||
MvcResult mvcResult = mockMvc.perform(post("/environment/create") |
||||
.header(SESSION_ID, sessionId) |
||||
.params(paramsMap)) |
||||
.andExpect(status().isCreated()) |
||||
.andExpect(content().contentType(MediaType.APPLICATION_JSON_UTF8)) |
||||
.andReturn(); |
||||
|
||||
Result result = JSONUtils.parseObject(mvcResult.getResponse().getContentAsString(), new TypeReference<Result<String>>() {}); |
||||
logger.info(result.toString()); |
||||
Assert.assertTrue(result != null && result.isSuccess()); |
||||
Assert.assertNotNull(result.getData()); |
||||
logger.info("create environment return result:{}", mvcResult.getResponse().getContentAsString()); |
||||
|
||||
environmentCode = (String)result.getData(); |
||||
} |
||||
|
||||
@Test |
||||
public void testUpdateEnvironment() throws Exception { |
||||
MultiValueMap<String, String> paramsMap = new LinkedMultiValueMap<>(); |
||||
paramsMap.add("code", environmentCode); |
||||
paramsMap.add("name","environment_test_update"); |
||||
paramsMap.add("config","this is config content"); |
||||
paramsMap.add("desc","the test environment update"); |
||||
|
||||
MvcResult mvcResult = mockMvc.perform(post("/environment/update") |
||||
.header(SESSION_ID, sessionId) |
||||
.params(paramsMap)) |
||||
.andExpect(status().isOk()) |
||||
.andExpect(content().contentType(MediaType.APPLICATION_JSON_UTF8)) |
||||
.andReturn(); |
||||
|
||||
Result result = JSONUtils.parseObject(mvcResult.getResponse().getContentAsString(), Result.class); |
||||
logger.info(result.toString()); |
||||
Assert.assertTrue(result != null && result.isSuccess()); |
||||
logger.info("update environment return result:{}", mvcResult.getResponse().getContentAsString()); |
||||
|
||||
} |
||||
|
||||
@Test |
||||
public void testQueryEnvironmentByCode() throws Exception { |
||||
MultiValueMap<String, String> paramsMap = new LinkedMultiValueMap<>(); |
||||
paramsMap.add("environmentCode", environmentCode); |
||||
|
||||
MvcResult mvcResult = mockMvc.perform(get("/environment/query-by-code") |
||||
.header(SESSION_ID, sessionId) |
||||
.params(paramsMap)) |
||||
.andExpect(status().isOk()) |
||||
.andExpect(content().contentType(MediaType.APPLICATION_JSON_UTF8)) |
||||
.andReturn(); |
||||
|
||||
Result result = JSONUtils.parseObject(mvcResult.getResponse().getContentAsString(), Result.class); |
||||
logger.info(result.toString()); |
||||
Assert.assertTrue(result != null && result.isSuccess()); |
||||
logger.info(mvcResult.getResponse().getContentAsString()); |
||||
logger.info("query environment by id :{}, return result:{}", environmentCode, mvcResult.getResponse().getContentAsString()); |
||||
|
||||
} |
||||
|
||||
@Test |
||||
public void testQueryEnvironmentListPaging() throws Exception { |
||||
MultiValueMap<String, String> paramsMap = new LinkedMultiValueMap<>(); |
||||
paramsMap.add("searchVal","test"); |
||||
paramsMap.add("pageSize","2"); |
||||
paramsMap.add("pageNo","2"); |
||||
|
||||
MvcResult mvcResult = mockMvc.perform(get("/environment/list-paging") |
||||
.header(SESSION_ID, sessionId) |
||||
.params(paramsMap)) |
||||
.andExpect(status().isOk()) |
||||
.andExpect(content().contentType(MediaType.APPLICATION_JSON_UTF8)) |
||||
.andReturn(); |
||||
|
||||
Result result = JSONUtils.parseObject(mvcResult.getResponse().getContentAsString(), Result.class); |
||||
logger.info(result.toString()); |
||||
Assert.assertTrue(result != null && result.isSuccess()); |
||||
logger.info("query list-paging environment return result:{}", mvcResult.getResponse().getContentAsString()); |
||||
} |
||||
|
||||
@Test |
||||
public void testQueryAllEnvironmentList() throws Exception { |
||||
MultiValueMap<String, String> paramsMap = new LinkedMultiValueMap<>(); |
||||
|
||||
MvcResult mvcResult = mockMvc.perform(get("/environment/query-environment-list") |
||||
.header(SESSION_ID, sessionId) |
||||
.params(paramsMap)) |
||||
.andExpect(status().isOk()) |
||||
.andExpect(content().contentType(MediaType.APPLICATION_JSON_UTF8)) |
||||
.andReturn(); |
||||
|
||||
Result result = JSONUtils.parseObject(mvcResult.getResponse().getContentAsString(), Result.class); |
||||
logger.info(result.toString()); |
||||
Assert.assertTrue(result != null && result.isSuccess()); |
||||
logger.info("query all environment return result:{}", mvcResult.getResponse().getContentAsString()); |
||||
|
||||
} |
||||
|
||||
@Test |
||||
public void testVerifyEnvironment() throws Exception { |
||||
MultiValueMap<String, String> paramsMap = new LinkedMultiValueMap<>(); |
||||
paramsMap.add("environmentName",environmentName); |
||||
|
||||
MvcResult mvcResult = mockMvc.perform(post("/environment/verify-environment") |
||||
.header(SESSION_ID, sessionId) |
||||
.params(paramsMap)) |
||||
.andExpect(status().isOk()) |
||||
.andExpect(content().contentType(MediaType.APPLICATION_JSON_UTF8)) |
||||
.andReturn(); |
||||
|
||||
Result result = JSONUtils.parseObject(mvcResult.getResponse().getContentAsString(), Result.class); |
||||
logger.info(result.toString()); |
||||
Assert.assertTrue(result.isStatus(Status.ENVIRONMENT_NAME_EXISTS)); |
||||
logger.info("verify environment return result:{}", mvcResult.getResponse().getContentAsString()); |
||||
|
||||
} |
||||
|
||||
private void testDeleteEnvironment() throws Exception { |
||||
Preconditions.checkNotNull(environmentCode); |
||||
|
||||
MultiValueMap<String, String> paramsMap = new LinkedMultiValueMap<>(); |
||||
paramsMap.add("environmentCode", environmentCode); |
||||
|
||||
MvcResult mvcResult = mockMvc.perform(post("/environment/delete") |
||||
.header(SESSION_ID, sessionId) |
||||
.params(paramsMap)) |
||||
.andExpect(status().isOk()) |
||||
.andExpect(content().contentType(MediaType.APPLICATION_JSON_UTF8)) |
||||
.andReturn(); |
||||
|
||||
Result result = JSONUtils.parseObject(mvcResult.getResponse().getContentAsString(), Result.class); |
||||
logger.info(result.toString()); |
||||
Assert.assertTrue(result != null && result.isSuccess()); |
||||
logger.info("delete environment return result:{}", mvcResult.getResponse().getContentAsString()); |
||||
} |
||||
} |
||||
@ -0,0 +1,310 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.api.service; |
||||
|
||||
import org.apache.dolphinscheduler.api.enums.Status; |
||||
import org.apache.dolphinscheduler.api.service.impl.EnvironmentServiceImpl; |
||||
import org.apache.dolphinscheduler.api.utils.PageInfo; |
||||
import org.apache.dolphinscheduler.api.utils.Result; |
||||
import org.apache.dolphinscheduler.common.Constants; |
||||
import org.apache.dolphinscheduler.common.enums.UserType; |
||||
import org.apache.dolphinscheduler.common.utils.CollectionUtils; |
||||
import org.apache.dolphinscheduler.dao.entity.Environment; |
||||
import org.apache.dolphinscheduler.dao.entity.EnvironmentWorkerGroupRelation; |
||||
import org.apache.dolphinscheduler.dao.entity.User; |
||||
import org.apache.dolphinscheduler.dao.mapper.EnvironmentMapper; |
||||
import org.apache.dolphinscheduler.dao.mapper.EnvironmentWorkerGroupRelationMapper; |
||||
import org.apache.dolphinscheduler.dao.mapper.TaskDefinitionMapper; |
||||
|
||||
import java.util.ArrayList; |
||||
import java.util.List; |
||||
import java.util.Map; |
||||
|
||||
import org.assertj.core.util.Lists; |
||||
import org.junit.After; |
||||
import org.junit.Assert; |
||||
import org.junit.Before; |
||||
import org.junit.Test; |
||||
import org.junit.runner.RunWith; |
||||
import org.mockito.InjectMocks; |
||||
import org.mockito.Mock; |
||||
import org.mockito.Mockito; |
||||
import org.mockito.junit.MockitoJUnitRunner; |
||||
import org.slf4j.Logger; |
||||
import org.slf4j.LoggerFactory; |
||||
|
||||
import com.baomidou.mybatisplus.core.conditions.Wrapper; |
||||
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper; |
||||
import com.baomidou.mybatisplus.core.metadata.IPage; |
||||
import com.baomidou.mybatisplus.extension.plugins.pagination.Page; |
||||
|
||||
/** |
||||
* environment service test |
||||
*/ |
||||
@RunWith(MockitoJUnitRunner.class) |
||||
public class EnvironmentServiceTest { |
||||
|
||||
public static final Logger logger = LoggerFactory.getLogger(EnvironmentServiceTest.class); |
||||
|
||||
@InjectMocks |
||||
private EnvironmentServiceImpl environmentService; |
||||
|
||||
@Mock |
||||
private EnvironmentMapper environmentMapper; |
||||
|
||||
@Mock |
||||
private EnvironmentWorkerGroupRelationMapper relationMapper; |
||||
|
||||
@Mock |
||||
private TaskDefinitionMapper taskDefinitionMapper; |
||||
|
||||
public static final String testUserName = "environmentServerTest"; |
||||
|
||||
public static final String environmentName = "Env1"; |
||||
|
||||
public static final String workerGroups = "[\"default\"]"; |
||||
|
||||
@Before |
||||
public void setUp(){ |
||||
} |
||||
|
||||
@After |
||||
public void after(){ |
||||
} |
||||
|
||||
@Test |
||||
public void testCreateEnvironment() { |
||||
User loginUser = getGeneralUser(); |
||||
Map<String, Object> result = environmentService.createEnvironment(loginUser,environmentName,getConfig(),getDesc(),workerGroups); |
||||
logger.info(result.toString()); |
||||
Assert.assertEquals(Status.USER_NO_OPERATION_PERM, result.get(Constants.STATUS)); |
||||
|
||||
loginUser = getAdminUser(); |
||||
result = environmentService.createEnvironment(loginUser,environmentName,"",getDesc(),workerGroups); |
||||
logger.info(result.toString()); |
||||
Assert.assertEquals(Status.ENVIRONMENT_CONFIG_IS_NULL, result.get(Constants.STATUS)); |
||||
|
||||
result = environmentService.createEnvironment(loginUser,"",getConfig(),getDesc(),workerGroups); |
||||
logger.info(result.toString()); |
||||
Assert.assertEquals(Status.ENVIRONMENT_NAME_IS_NULL, result.get(Constants.STATUS)); |
||||
|
||||
result = environmentService.createEnvironment(loginUser,environmentName,getConfig(),getDesc(),"test"); |
||||
logger.info(result.toString()); |
||||
Assert.assertEquals(Status.ENVIRONMENT_WORKER_GROUPS_IS_INVALID, result.get(Constants.STATUS)); |
||||
|
||||
Mockito.when(environmentMapper.queryByEnvironmentName(environmentName)).thenReturn(getEnvironment()); |
||||
result = environmentService.createEnvironment(loginUser,environmentName,getConfig(),getDesc(),workerGroups); |
||||
logger.info(result.toString()); |
||||
Assert.assertEquals(Status.ENVIRONMENT_NAME_EXISTS, result.get(Constants.STATUS)); |
||||
|
||||
Mockito.when(environmentMapper.insert(Mockito.any(Environment.class))).thenReturn(1); |
||||
Mockito.when(relationMapper.insert(Mockito.any(EnvironmentWorkerGroupRelation.class))).thenReturn(1); |
||||
result = environmentService.createEnvironment(loginUser,"testName","test","test",workerGroups); |
||||
logger.info(result.toString()); |
||||
Assert.assertEquals(Status.SUCCESS, result.get(Constants.STATUS)); |
||||
|
||||
} |
||||
|
||||
@Test |
||||
public void testCheckParams() { |
||||
Map<String, Object> result = environmentService.checkParams(environmentName,getConfig(),"test"); |
||||
Assert.assertEquals(Status.ENVIRONMENT_WORKER_GROUPS_IS_INVALID, result.get(Constants.STATUS)); |
||||
} |
||||
|
||||
@Test |
||||
public void testUpdateEnvironmentByCode() { |
||||
User loginUser = getGeneralUser(); |
||||
Map<String, Object> result = environmentService.updateEnvironmentByCode(loginUser,1L,environmentName,getConfig(),getDesc(),workerGroups); |
||||
logger.info(result.toString()); |
||||
Assert.assertEquals(Status.USER_NO_OPERATION_PERM, result.get(Constants.STATUS)); |
||||
|
||||
loginUser = getAdminUser(); |
||||
result = environmentService.updateEnvironmentByCode(loginUser,1L,environmentName,"",getDesc(),workerGroups); |
||||
logger.info(result.toString()); |
||||
Assert.assertEquals(Status.ENVIRONMENT_CONFIG_IS_NULL, result.get(Constants.STATUS)); |
||||
|
||||
result = environmentService.updateEnvironmentByCode(loginUser,1L,"",getConfig(),getDesc(),workerGroups); |
||||
logger.info(result.toString()); |
||||
Assert.assertEquals(Status.ENVIRONMENT_NAME_IS_NULL, result.get(Constants.STATUS)); |
||||
|
||||
result = environmentService.updateEnvironmentByCode(loginUser,1L,environmentName,getConfig(),getDesc(),"test"); |
||||
logger.info(result.toString()); |
||||
Assert.assertEquals(Status.ENVIRONMENT_WORKER_GROUPS_IS_INVALID, result.get(Constants.STATUS)); |
||||
|
||||
Mockito.when(environmentMapper.queryByEnvironmentName(environmentName)).thenReturn(getEnvironment()); |
||||
result = environmentService.updateEnvironmentByCode(loginUser,2L,environmentName,getConfig(),getDesc(),workerGroups); |
||||
logger.info(result.toString()); |
||||
Assert.assertEquals(Status.ENVIRONMENT_NAME_EXISTS, result.get(Constants.STATUS)); |
||||
|
||||
Mockito.when(environmentMapper.update(Mockito.any(Environment.class),Mockito.any(Wrapper.class))).thenReturn(1); |
||||
result = environmentService.updateEnvironmentByCode(loginUser,1L,"testName","test","test",workerGroups); |
||||
logger.info(result.toString()); |
||||
Assert.assertEquals(Status.SUCCESS, result.get(Constants.STATUS)); |
||||
|
||||
} |
||||
|
||||
@Test |
||||
public void testQueryAllEnvironmentList() { |
||||
Mockito.when(environmentMapper.queryAllEnvironmentList()).thenReturn(Lists.newArrayList(getEnvironment())); |
||||
Map<String, Object> result = environmentService.queryAllEnvironmentList(); |
||||
logger.info(result.toString()); |
||||
Assert.assertEquals(Status.SUCCESS,result.get(Constants.STATUS)); |
||||
|
||||
List<Environment> list = (List<Environment>)(result.get(Constants.DATA_LIST)); |
||||
Assert.assertEquals(1,list.size()); |
||||
} |
||||
|
||||
@Test |
||||
public void testQueryEnvironmentListPaging() { |
||||
IPage<Environment> page = new Page<>(1, 10); |
||||
page.setRecords(getList()); |
||||
page.setTotal(1L); |
||||
Mockito.when(environmentMapper.queryEnvironmentListPaging(Mockito.any(Page.class), Mockito.eq(environmentName))).thenReturn(page); |
||||
|
||||
Result result = environmentService.queryEnvironmentListPaging(1, 10, environmentName); |
||||
logger.info(result.toString()); |
||||
PageInfo<Environment> pageInfo = (PageInfo<Environment>) result.getData(); |
||||
Assert.assertTrue(CollectionUtils.isNotEmpty(pageInfo.getTotalList())); |
||||
} |
||||
|
||||
@Test |
||||
public void testQueryEnvironmentByName() { |
||||
Mockito.when(environmentMapper.queryByEnvironmentName(environmentName)).thenReturn(null); |
||||
Map<String, Object> result = environmentService.queryEnvironmentByName(environmentName); |
||||
logger.info(result.toString()); |
||||
Assert.assertEquals(Status.QUERY_ENVIRONMENT_BY_NAME_ERROR,result.get(Constants.STATUS)); |
||||
|
||||
Mockito.when(environmentMapper.queryByEnvironmentName(environmentName)).thenReturn(getEnvironment()); |
||||
result = environmentService.queryEnvironmentByName(environmentName); |
||||
logger.info(result.toString()); |
||||
Assert.assertEquals(Status.SUCCESS,result.get(Constants.STATUS)); |
||||
} |
||||
|
||||
@Test |
||||
public void testQueryEnvironmentByCode() { |
||||
Mockito.when(environmentMapper.queryByEnvironmentCode(1L)).thenReturn(null); |
||||
Map<String, Object> result = environmentService.queryEnvironmentByCode(1L); |
||||
logger.info(result.toString()); |
||||
Assert.assertEquals(Status.QUERY_ENVIRONMENT_BY_CODE_ERROR,result.get(Constants.STATUS)); |
||||
|
||||
Mockito.when(environmentMapper.queryByEnvironmentCode(1L)).thenReturn(getEnvironment()); |
||||
result = environmentService.queryEnvironmentByCode(1L); |
||||
logger.info(result.toString()); |
||||
Assert.assertEquals(Status.SUCCESS,result.get(Constants.STATUS)); |
||||
} |
||||
|
||||
@Test |
||||
public void testDeleteEnvironmentByCode() { |
||||
User loginUser = getGeneralUser(); |
||||
Map<String, Object> result = environmentService.deleteEnvironmentByCode(loginUser,1L); |
||||
logger.info(result.toString()); |
||||
Assert.assertEquals(Status.USER_NO_OPERATION_PERM, result.get(Constants.STATUS)); |
||||
|
||||
loginUser = getAdminUser(); |
||||
Mockito.when(taskDefinitionMapper.selectCount(Mockito.any(LambdaQueryWrapper.class))).thenReturn(1); |
||||
result = environmentService.deleteEnvironmentByCode(loginUser,1L); |
||||
logger.info(result.toString()); |
||||
Assert.assertEquals(Status.DELETE_ENVIRONMENT_RELATED_TASK_EXISTS, result.get(Constants.STATUS)); |
||||
|
||||
Mockito.when(taskDefinitionMapper.selectCount(Mockito.any(LambdaQueryWrapper.class))).thenReturn(0); |
||||
Mockito.when(environmentMapper.deleteByCode(1L)).thenReturn(1); |
||||
result = environmentService.deleteEnvironmentByCode(loginUser,1L); |
||||
logger.info(result.toString()); |
||||
Assert.assertEquals(Status.SUCCESS, result.get(Constants.STATUS)); |
||||
} |
||||
|
||||
@Test |
||||
public void testVerifyEnvironment() { |
||||
Map<String, Object> result = environmentService.verifyEnvironment(""); |
||||
logger.info(result.toString()); |
||||
Assert.assertEquals(Status.ENVIRONMENT_NAME_IS_NULL, result.get(Constants.STATUS)); |
||||
|
||||
Mockito.when(environmentMapper.queryByEnvironmentName(environmentName)).thenReturn(getEnvironment()); |
||||
result = environmentService.verifyEnvironment(environmentName); |
||||
logger.info(result.toString()); |
||||
Assert.assertEquals(Status.ENVIRONMENT_NAME_EXISTS, result.get(Constants.STATUS)); |
||||
} |
||||
|
||||
private Environment getEnvironment() { |
||||
Environment environment = new Environment(); |
||||
environment.setId(1); |
||||
environment.setCode(1L); |
||||
environment.setName(environmentName); |
||||
environment.setConfig(getConfig()); |
||||
environment.setDescription(getDesc()); |
||||
environment.setOperator(1); |
||||
return environment; |
||||
} |
||||
|
||||
/** |
||||
* create an environment description |
||||
*/ |
||||
private String getDesc() { |
||||
return "create an environment to test "; |
||||
} |
||||
|
||||
/** |
||||
* create an environment config |
||||
*/ |
||||
private String getConfig() { |
||||
return "export HADOOP_HOME=/opt/hadoop-2.6.5\n" |
||||
+ "export HADOOP_CONF_DIR=/etc/hadoop/conf\n" |
||||
+ "export SPARK_HOME1=/opt/soft/spark1\n" |
||||
+ "export SPARK_HOME2=/opt/soft/spark2\n" |
||||
+ "export PYTHON_HOME=/opt/soft/python\n" |
||||
+ "export JAVA_HOME=/opt/java/jdk1.8.0_181-amd64\n" |
||||
+ "export HIVE_HOME=/opt/soft/hive\n" |
||||
+ "export FLINK_HOME=/opt/soft/flink\n" |
||||
+ "export DATAX_HOME=/opt/soft/datax\n" |
||||
+ "export YARN_CONF_DIR=\"/etc/hadoop/conf\"\n" |
||||
+ "\n" |
||||
+ "export PATH=$HADOOP_HOME/bin:$SPARK_HOME1/bin:$SPARK_HOME2/bin:$PYTHON_HOME/bin:$JAVA_HOME/bin:$HIVE_HOME/bin:$FLINK_HOME/bin:$DATAX_HOME/bin:$PATH\n" |
||||
+ "\n" |
||||
+ "export HADOOP_CLASSPATH=`hadoop classpath`\n" |
||||
+ "\n" |
||||
+ "#echo \"HADOOP_CLASSPATH=\"$HADOOP_CLASSPATH"; |
||||
} |
||||
|
||||
/** |
||||
* create general user |
||||
*/ |
||||
private User getGeneralUser() { |
||||
User loginUser = new User(); |
||||
loginUser.setUserType(UserType.GENERAL_USER); |
||||
loginUser.setUserName(testUserName); |
||||
loginUser.setId(1); |
||||
return loginUser; |
||||
} |
||||
|
||||
/** |
||||
* create admin user |
||||
*/ |
||||
private User getAdminUser() { |
||||
User loginUser = new User(); |
||||
loginUser.setUserType(UserType.ADMIN_USER); |
||||
loginUser.setUserName(testUserName); |
||||
loginUser.setId(1); |
||||
return loginUser; |
||||
} |
||||
|
||||
private List<Environment> getList() { |
||||
List<Environment> list = new ArrayList<>(); |
||||
list.add(getEnvironment()); |
||||
return list; |
||||
} |
||||
} |
||||
@ -0,0 +1,69 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.api.service; |
||||
|
||||
import org.apache.dolphinscheduler.api.enums.Status; |
||||
import org.apache.dolphinscheduler.api.service.impl.EnvironmentWorkerGroupRelationServiceImpl; |
||||
import org.apache.dolphinscheduler.common.Constants; |
||||
import org.apache.dolphinscheduler.dao.entity.EnvironmentWorkerGroupRelation; |
||||
import org.apache.dolphinscheduler.dao.mapper.EnvironmentWorkerGroupRelationMapper; |
||||
|
||||
import java.util.Map; |
||||
|
||||
import org.assertj.core.util.Lists; |
||||
import org.junit.Assert; |
||||
import org.junit.Test; |
||||
import org.junit.runner.RunWith; |
||||
import org.mockito.InjectMocks; |
||||
import org.mockito.Mock; |
||||
import org.mockito.Mockito; |
||||
import org.mockito.junit.MockitoJUnitRunner; |
||||
import org.slf4j.Logger; |
||||
import org.slf4j.LoggerFactory; |
||||
|
||||
/** |
||||
* environment service test |
||||
*/ |
||||
@RunWith(MockitoJUnitRunner.class) |
||||
public class EnvironmentWorkerGroupRelationServiceTest { |
||||
|
||||
public static final Logger logger = LoggerFactory.getLogger(EnvironmentWorkerGroupRelationServiceTest.class); |
||||
|
||||
@InjectMocks |
||||
private EnvironmentWorkerGroupRelationServiceImpl relationService; |
||||
|
||||
@Mock |
||||
private EnvironmentWorkerGroupRelationMapper relationMapper; |
||||
|
||||
@Test |
||||
public void testQueryEnvironmentWorkerGroupRelation() { |
||||
Mockito.when(relationMapper.queryByEnvironmentCode(1L)).thenReturn(Lists.newArrayList(new EnvironmentWorkerGroupRelation())); |
||||
Map<String, Object> result = relationService.queryEnvironmentWorkerGroupRelation(1L); |
||||
logger.info(result.toString()); |
||||
Assert.assertEquals(Status.SUCCESS,result.get(Constants.STATUS)); |
||||
} |
||||
|
||||
@Test |
||||
public void testQueryAllEnvironmentWorkerGroupRelationList() { |
||||
Mockito.when(relationMapper.selectList(Mockito.any())).thenReturn(Lists.newArrayList(new EnvironmentWorkerGroupRelation())); |
||||
Map<String, Object> result = relationService.queryAllEnvironmentWorkerGroupRelationList(); |
||||
logger.info(result.toString()); |
||||
Assert.assertEquals(Status.SUCCESS,result.get(Constants.STATUS)); |
||||
} |
||||
|
||||
} |
||||
@ -0,0 +1,111 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.common.enums; |
||||
|
||||
import io.netty.channel.Channel; |
||||
|
||||
/** |
||||
* state event |
||||
*/ |
||||
public class StateEvent { |
||||
|
||||
/** |
||||
* origin_pid-origin_task_id-process_instance_id-task_instance_id |
||||
*/ |
||||
private String key; |
||||
|
||||
private StateEventType type; |
||||
|
||||
private ExecutionStatus executionStatus; |
||||
|
||||
private int taskInstanceId; |
||||
|
||||
private int processInstanceId; |
||||
|
||||
private String context; |
||||
|
||||
private Channel channel; |
||||
|
||||
public ExecutionStatus getExecutionStatus() { |
||||
return executionStatus; |
||||
} |
||||
|
||||
public void setExecutionStatus(ExecutionStatus executionStatus) { |
||||
this.executionStatus = executionStatus; |
||||
} |
||||
|
||||
public int getTaskInstanceId() { |
||||
return taskInstanceId; |
||||
} |
||||
|
||||
public int getProcessInstanceId() { |
||||
return processInstanceId; |
||||
} |
||||
|
||||
public void setProcessInstanceId(int processInstanceId) { |
||||
this.processInstanceId = processInstanceId; |
||||
} |
||||
|
||||
public String getContext() { |
||||
return context; |
||||
} |
||||
|
||||
public void setContext(String context) { |
||||
this.context = context; |
||||
} |
||||
|
||||
public void setTaskInstanceId(int taskInstanceId) { |
||||
this.taskInstanceId = taskInstanceId; |
||||
} |
||||
|
||||
public Channel getChannel() { |
||||
return channel; |
||||
} |
||||
|
||||
public void setChannel(Channel channel) { |
||||
this.channel = channel; |
||||
} |
||||
|
||||
@Override |
||||
public String toString() { |
||||
return "State Event :" |
||||
+ "key: " + key |
||||
+ " type: " + type.toString() |
||||
+ " executeStatus: " + executionStatus |
||||
+ " task instance id: " + taskInstanceId |
||||
+ " process instance id: " + processInstanceId |
||||
+ " context: " + context |
||||
; |
||||
} |
||||
|
||||
public String getKey() { |
||||
return key; |
||||
} |
||||
|
||||
public void setKey(String key) { |
||||
this.key = key; |
||||
} |
||||
|
||||
public void setType(StateEventType type) { |
||||
this.type = type; |
||||
} |
||||
|
||||
public StateEventType getType() { |
||||
return this.type; |
||||
} |
||||
} |
||||
@ -0,0 +1,142 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.dao.entity; |
||||
|
||||
import java.util.Date; |
||||
|
||||
import com.baomidou.mybatisplus.annotation.IdType; |
||||
import com.baomidou.mybatisplus.annotation.TableId; |
||||
import com.baomidou.mybatisplus.annotation.TableName; |
||||
import com.fasterxml.jackson.annotation.JsonFormat; |
||||
|
||||
/** |
||||
* Environment |
||||
*/ |
||||
@TableName("t_ds_environment") |
||||
public class Environment { |
||||
|
||||
@TableId(value = "id", type = IdType.AUTO) |
||||
private int id; |
||||
|
||||
/** |
||||
* environment code |
||||
*/ |
||||
private Long code; |
||||
|
||||
/** |
||||
* environment name |
||||
*/ |
||||
private String name; |
||||
|
||||
/** |
||||
* config content |
||||
*/ |
||||
private String config; |
||||
|
||||
private String description; |
||||
|
||||
/** |
||||
* operator user id |
||||
*/ |
||||
private Integer operator; |
||||
|
||||
@JsonFormat(pattern = "yyyy-MM-dd HH:mm:ss", timezone = "GMT+8") |
||||
private Date createTime; |
||||
|
||||
@JsonFormat(pattern = "yyyy-MM-dd HH:mm:ss", timezone = "GMT+8") |
||||
private Date updateTime; |
||||
|
||||
public int getId() { |
||||
return id; |
||||
} |
||||
|
||||
public void setId(int id) { |
||||
this.id = id; |
||||
} |
||||
|
||||
public String getName() { |
||||
return name; |
||||
} |
||||
|
||||
public void setName(String name) { |
||||
this.name = name; |
||||
} |
||||
|
||||
public Long getCode() { |
||||
return this.code; |
||||
} |
||||
|
||||
public void setCode(Long code) { |
||||
this.code = code; |
||||
} |
||||
|
||||
public String getConfig() { |
||||
return this.config; |
||||
} |
||||
|
||||
public void setConfig(String config) { |
||||
this.config = config; |
||||
} |
||||
|
||||
public String getDescription() { |
||||
return this.description; |
||||
} |
||||
|
||||
public void setDescription(String description) { |
||||
this.description = description; |
||||
} |
||||
|
||||
public Integer getOperator() { |
||||
return this.operator; |
||||
} |
||||
|
||||
public void setOperator(Integer operator) { |
||||
this.operator = operator; |
||||
} |
||||
|
||||
public Date getCreateTime() { |
||||
return createTime; |
||||
} |
||||
|
||||
public void setCreateTime(Date createTime) { |
||||
this.createTime = createTime; |
||||
} |
||||
|
||||
public Date getUpdateTime() { |
||||
return updateTime; |
||||
} |
||||
|
||||
public void setUpdateTime(Date updateTime) { |
||||
this.updateTime = updateTime; |
||||
} |
||||
|
||||
@Override |
||||
public String toString() { |
||||
return "Environment{" |
||||
+ "id= " + id |
||||
+ ", code= " + code |
||||
+ ", name= " + name |
||||
+ ", config= " + config |
||||
+ ", description= " + description |
||||
+ ", operator= " + operator |
||||
+ ", createTime= " + createTime |
||||
+ ", updateTime= " + updateTime |
||||
+ "}"; |
||||
} |
||||
|
||||
} |
||||
@ -0,0 +1,117 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.dao.entity; |
||||
|
||||
import java.util.Date; |
||||
|
||||
import com.baomidou.mybatisplus.annotation.IdType; |
||||
import com.baomidou.mybatisplus.annotation.TableId; |
||||
import com.baomidou.mybatisplus.annotation.TableName; |
||||
import com.fasterxml.jackson.annotation.JsonFormat; |
||||
|
||||
/** |
||||
* EnvironmentWorkerGroupRelation |
||||
*/ |
||||
@TableName("t_ds_environment_worker_group_relation") |
||||
public class EnvironmentWorkerGroupRelation { |
||||
|
||||
@TableId(value = "id", type = IdType.AUTO) |
||||
private int id; |
||||
|
||||
/** |
||||
* environment code |
||||
*/ |
||||
private Long environmentCode; |
||||
|
||||
/** |
||||
* worker group id |
||||
*/ |
||||
private String workerGroup; |
||||
|
||||
/** |
||||
* operator user id |
||||
*/ |
||||
private Integer operator; |
||||
|
||||
@JsonFormat(pattern = "yyyy-MM-dd HH:mm:ss", timezone = "GMT+8") |
||||
private Date createTime; |
||||
|
||||
@JsonFormat(pattern = "yyyy-MM-dd HH:mm:ss", timezone = "GMT+8") |
||||
private Date updateTime; |
||||
|
||||
public int getId() { |
||||
return id; |
||||
} |
||||
|
||||
public void setId(int id) { |
||||
this.id = id; |
||||
} |
||||
|
||||
public String getWorkerGroup() { |
||||
return workerGroup; |
||||
} |
||||
|
||||
public void setWorkerGroup(String workerGroup) { |
||||
this.workerGroup = workerGroup; |
||||
} |
||||
|
||||
public Long getEnvironmentCode() { |
||||
return this.environmentCode; |
||||
} |
||||
|
||||
public void setEnvironmentCode(Long environmentCode) { |
||||
this.environmentCode = environmentCode; |
||||
} |
||||
|
||||
public Integer getOperator() { |
||||
return this.operator; |
||||
} |
||||
|
||||
public void setOperator(Integer operator) { |
||||
this.operator = operator; |
||||
} |
||||
|
||||
public Date getCreateTime() { |
||||
return createTime; |
||||
} |
||||
|
||||
public void setCreateTime(Date createTime) { |
||||
this.createTime = createTime; |
||||
} |
||||
|
||||
public Date getUpdateTime() { |
||||
return updateTime; |
||||
} |
||||
|
||||
public void setUpdateTime(Date updateTime) { |
||||
this.updateTime = updateTime; |
||||
} |
||||
|
||||
@Override |
||||
public String toString() { |
||||
return "EnvironmentWorkerGroupRelation{" |
||||
+ "id= " + id |
||||
+ ", environmentCode= " + environmentCode |
||||
+ ", workerGroup= " + workerGroup |
||||
+ ", operator= " + operator |
||||
+ ", createTime= " + createTime |
||||
+ ", updateTime= " + updateTime |
||||
+ "}"; |
||||
} |
||||
|
||||
} |
||||
@ -0,0 +1,71 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.dao.mapper; |
||||
|
||||
import org.apache.dolphinscheduler.dao.entity.Environment; |
||||
|
||||
import org.apache.ibatis.annotations.Param; |
||||
|
||||
import java.util.List; |
||||
|
||||
import com.baomidou.mybatisplus.core.mapper.BaseMapper; |
||||
import com.baomidou.mybatisplus.core.metadata.IPage; |
||||
|
||||
/** |
||||
* environment mapper interface
|
||||
*/ |
||||
public interface EnvironmentMapper extends BaseMapper<Environment> { |
||||
|
||||
/** |
||||
* query environment by name |
||||
* |
||||
* @param name name |
||||
* @return environment |
||||
*/ |
||||
Environment queryByEnvironmentName(@Param("environmentName") String name); |
||||
|
||||
/** |
||||
* query environment by code |
||||
* |
||||
* @param environmentCode environmentCode |
||||
* @return environment |
||||
*/ |
||||
Environment queryByEnvironmentCode(@Param("environmentCode") Long environmentCode); |
||||
|
||||
/** |
||||
* query all environment list |
||||
* @return environment list |
||||
*/ |
||||
List<Environment> queryAllEnvironmentList(); |
||||
|
||||
/** |
||||
* environment page |
||||
* @param page page |
||||
* @param searchName searchName |
||||
* @return environment IPage |
||||
*/ |
||||
IPage<Environment> queryEnvironmentListPaging(IPage<Environment> page, @Param("searchName") String searchName); |
||||
|
||||
/** |
||||
* delete environment by code |
||||
* |
||||
* @param code code |
||||
* @return int |
||||
*/ |
||||
int deleteByCode(@Param("code") Long code); |
||||
} |
||||
@ -0,0 +1,57 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.dao.mapper; |
||||
|
||||
import org.apache.dolphinscheduler.dao.entity.EnvironmentWorkerGroupRelation; |
||||
|
||||
import org.apache.ibatis.annotations.Param; |
||||
|
||||
import java.util.List; |
||||
|
||||
import com.baomidou.mybatisplus.core.mapper.BaseMapper; |
||||
|
||||
/** |
||||
* environment worker group relation mapper interface
|
||||
*/ |
||||
public interface EnvironmentWorkerGroupRelationMapper extends BaseMapper<EnvironmentWorkerGroupRelation> { |
||||
|
||||
/** |
||||
* environment worker group relation by environmentCode |
||||
* |
||||
* @param environmentCode environmentCode |
||||
* @return EnvironmentWorkerGroupRelation list |
||||
*/ |
||||
List<EnvironmentWorkerGroupRelation> queryByEnvironmentCode(@Param("environmentCode") Long environmentCode); |
||||
|
||||
/** |
||||
* environment worker group relation by workerGroupName |
||||
* |
||||
* @param workerGroupName workerGroupName |
||||
* @return EnvironmentWorkerGroupRelation list |
||||
*/ |
||||
List<EnvironmentWorkerGroupRelation> queryByWorkerGroupName(@Param("workerGroupName") String workerGroupName); |
||||
|
||||
/** |
||||
* delete environment worker group relation by processCode |
||||
* |
||||
* @param environmentCode environmentCode |
||||
* @param workerGroupName workerGroupName |
||||
* @return int |
||||
*/ |
||||
int deleteByCode(@Param("environmentCode") Long environmentCode, @Param("workerGroupName") String workerGroupName); |
||||
} |
||||
@ -0,0 +1,55 @@
|
||||
<?xml version="1.0" encoding="UTF-8" ?> |
||||
<!-- |
||||
~ Licensed to the Apache Software Foundation (ASF) under one or more |
||||
~ contributor license agreements. See the NOTICE file distributed with |
||||
~ this work for additional information regarding copyright ownership. |
||||
~ The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
~ (the "License"); you may not use this file except in compliance with |
||||
~ the License. You may obtain a copy of the License at |
||||
~ |
||||
~ http://www.apache.org/licenses/LICENSE-2.0 |
||||
~ |
||||
~ Unless required by applicable law or agreed to in writing, software |
||||
~ distributed under the License is distributed on an "AS IS" BASIS, |
||||
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
~ See the License for the specific language governing permissions and |
||||
~ limitations under the License. |
||||
--> |
||||
|
||||
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd" > |
||||
<mapper namespace="org.apache.dolphinscheduler.dao.mapper.EnvironmentMapper"> |
||||
<sql id="baseSql"> |
||||
id, code, name, config, description, operator, create_time, update_time |
||||
</sql> |
||||
<select id="queryByEnvironmentName" resultType="org.apache.dolphinscheduler.dao.entity.Environment"> |
||||
select |
||||
<include refid="baseSql"/> |
||||
from t_ds_environment |
||||
WHERE name = #{environmentName} |
||||
</select> |
||||
<select id="queryAllEnvironmentList" resultType="org.apache.dolphinscheduler.dao.entity.Environment"> |
||||
select |
||||
<include refid="baseSql"/> |
||||
from t_ds_environment |
||||
order by create_time desc |
||||
</select> |
||||
<select id="queryEnvironmentListPaging" resultType="org.apache.dolphinscheduler.dao.entity.Environment"> |
||||
select |
||||
<include refid="baseSql"/> |
||||
from t_ds_environment |
||||
where 1=1 |
||||
<if test="searchName!=null and searchName != ''"> |
||||
and name like concat('%', #{searchName}, '%') |
||||
</if> |
||||
order by create_time desc |
||||
</select> |
||||
<select id="queryByEnvironmentCode" resultType="org.apache.dolphinscheduler.dao.entity.Environment"> |
||||
select |
||||
<include refid="baseSql"/> |
||||
from t_ds_environment |
||||
where code = #{environmentCode} |
||||
</select> |
||||
<delete id="deleteByCode"> |
||||
delete from t_ds_environment where code = #{code} |
||||
</delete> |
||||
</mapper> |
||||
@ -0,0 +1,40 @@
|
||||
<?xml version="1.0" encoding="UTF-8" ?> |
||||
<!-- |
||||
~ Licensed to the Apache Software Foundation (ASF) under one or more |
||||
~ contributor license agreements. See the NOTICE file distributed with |
||||
~ this work for additional information regarding copyright ownership. |
||||
~ The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
~ (the "License"); you may not use this file except in compliance with |
||||
~ the License. You may obtain a copy of the License at |
||||
~ |
||||
~ http://www.apache.org/licenses/LICENSE-2.0 |
||||
~ |
||||
~ Unless required by applicable law or agreed to in writing, software |
||||
~ distributed under the License is distributed on an "AS IS" BASIS, |
||||
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
~ See the License for the specific language governing permissions and |
||||
~ limitations under the License. |
||||
--> |
||||
|
||||
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd" > |
||||
<mapper namespace="org.apache.dolphinscheduler.dao.mapper.EnvironmentWorkerGroupRelationMapper"> |
||||
<sql id="baseSql"> |
||||
id, environment_code, worker_group, operator, create_time, update_time |
||||
</sql> |
||||
<select id="queryByEnvironmentCode" resultType="org.apache.dolphinscheduler.dao.entity.EnvironmentWorkerGroupRelation"> |
||||
select |
||||
<include refid="baseSql"/> |
||||
from t_ds_environment_worker_group_relation |
||||
WHERE environment_code = #{environmentCode} |
||||
</select> |
||||
<select id="queryByWorkerGroupName" resultType="org.apache.dolphinscheduler.dao.entity.EnvironmentWorkerGroupRelation"> |
||||
select |
||||
<include refid="baseSql"/> |
||||
from t_ds_environment_worker_group_relation |
||||
WHERE worker_group = #{workerGroupName} |
||||
</select> |
||||
<delete id="deleteByCode"> |
||||
delete from t_ds_environment_worker_group_relation |
||||
WHERE environment_code = #{environmentCode} and worker_group = #{workerGroupName} |
||||
</delete> |
||||
</mapper> |
||||
@ -0,0 +1,199 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.dao.mapper; |
||||
|
||||
import org.apache.dolphinscheduler.dao.entity.Environment; |
||||
|
||||
import java.util.Date; |
||||
import java.util.List; |
||||
|
||||
import org.junit.After; |
||||
import org.junit.Assert; |
||||
import org.junit.Before; |
||||
import org.junit.Test; |
||||
import org.junit.runner.RunWith; |
||||
import org.springframework.beans.factory.annotation.Autowired; |
||||
import org.springframework.boot.test.context.SpringBootTest; |
||||
import org.springframework.test.annotation.Rollback; |
||||
import org.springframework.test.context.junit4.SpringRunner; |
||||
import org.springframework.transaction.annotation.Transactional; |
||||
|
||||
import com.baomidou.mybatisplus.core.metadata.IPage; |
||||
import com.baomidou.mybatisplus.extension.plugins.pagination.Page; |
||||
|
||||
@RunWith(SpringRunner.class) |
||||
@SpringBootTest |
||||
@Transactional |
||||
@Rollback(false) |
||||
public class EnvironmentMapperTest { |
||||
|
||||
@Autowired |
||||
EnvironmentMapper environmentMapper; |
||||
|
||||
/** |
||||
* insert |
||||
* |
||||
* @return Environment |
||||
*/ |
||||
private Environment insertOne() { |
||||
//insertOne
|
||||
Environment environment = new Environment(); |
||||
environment.setName("testEnv"); |
||||
environment.setCode(1L); |
||||
environment.setOperator(1); |
||||
environment.setConfig(getConfig()); |
||||
environment.setDescription(getDesc()); |
||||
environment.setCreateTime(new Date()); |
||||
environment.setUpdateTime(new Date()); |
||||
environmentMapper.insert(environment); |
||||
return environment; |
||||
} |
||||
|
||||
@Before |
||||
public void setUp() { |
||||
clearTestData(); |
||||
} |
||||
|
||||
@After |
||||
public void after() { |
||||
clearTestData(); |
||||
} |
||||
|
||||
public void clearTestData() { |
||||
environmentMapper.queryAllEnvironmentList().stream().forEach(environment -> { |
||||
environmentMapper.deleteByCode(environment.getCode()); |
||||
}); |
||||
} |
||||
|
||||
/** |
||||
* test update |
||||
*/ |
||||
@Test |
||||
public void testUpdate() { |
||||
//insertOne
|
||||
Environment environment = insertOne(); |
||||
environment.setDescription("new description info"); |
||||
//update
|
||||
int update = environmentMapper.updateById(environment); |
||||
Assert.assertEquals(update, 1); |
||||
} |
||||
|
||||
/** |
||||
* test delete |
||||
*/ |
||||
@Test |
||||
public void testDelete() { |
||||
Environment environment = insertOne(); |
||||
int delete = environmentMapper.deleteById(environment.getId()); |
||||
Assert.assertEquals(delete, 1); |
||||
} |
||||
|
||||
/** |
||||
* test query |
||||
*/ |
||||
@Test |
||||
public void testQuery() { |
||||
insertOne(); |
||||
//query
|
||||
List<Environment> environments = environmentMapper.selectList(null); |
||||
Assert.assertEquals(environments.size(), 1); |
||||
} |
||||
|
||||
/** |
||||
* test query environment by name |
||||
*/ |
||||
@Test |
||||
public void testQueryByEnvironmentName() { |
||||
Environment entity = insertOne(); |
||||
Environment environment = environmentMapper.queryByEnvironmentName(entity.getName()); |
||||
Assert.assertEquals(entity.toString(),environment.toString()); |
||||
} |
||||
|
||||
/** |
||||
* test query environment by code |
||||
*/ |
||||
@Test |
||||
public void testQueryByEnvironmentCode() { |
||||
Environment entity = insertOne(); |
||||
Environment environment = environmentMapper.queryByEnvironmentCode(entity.getCode()); |
||||
Assert.assertEquals(entity.toString(),environment.toString()); |
||||
} |
||||
|
||||
/** |
||||
* test query all environments |
||||
*/ |
||||
@Test |
||||
public void testQueryAllEnvironmentList() { |
||||
Environment entity = insertOne(); |
||||
List<Environment> environments = environmentMapper.queryAllEnvironmentList(); |
||||
Assert.assertEquals(environments.size(), 1); |
||||
Assert.assertEquals(entity.toString(),environments.get(0).toString()); |
||||
} |
||||
|
||||
/** |
||||
* test query environment list paging |
||||
*/ |
||||
@Test |
||||
public void testQueryEnvironmentListPaging() { |
||||
Environment entity = insertOne(); |
||||
Page<Environment> page = new Page<>(1, 10); |
||||
IPage<Environment> environmentIPage = environmentMapper.queryEnvironmentListPaging(page,""); |
||||
List<Environment> environmentList = environmentIPage.getRecords(); |
||||
Assert.assertEquals(environmentList.size(), 1); |
||||
|
||||
environmentIPage = environmentMapper.queryEnvironmentListPaging(page,"abc"); |
||||
environmentList = environmentIPage.getRecords(); |
||||
Assert.assertEquals(environmentList.size(), 0); |
||||
} |
||||
|
||||
/** |
||||
* test query all environments |
||||
*/ |
||||
@Test |
||||
public void testDeleteByCode() { |
||||
Environment entity = insertOne(); |
||||
int delete = environmentMapper.deleteByCode(entity.getCode()); |
||||
Assert.assertEquals(delete, 1); |
||||
} |
||||
|
||||
private String getDesc() { |
||||
return "create an environment to test "; |
||||
} |
||||
|
||||
/** |
||||
* create an environment config |
||||
*/ |
||||
private String getConfig() { |
||||
return "export HADOOP_HOME=/opt/hadoop-2.6.5\n" |
||||
+ "export HADOOP_CONF_DIR=/etc/hadoop/conf\n" |
||||
+ "export SPARK_HOME1=/opt/soft/spark1\n" |
||||
+ "export SPARK_HOME2=/opt/soft/spark2\n" |
||||
+ "export PYTHON_HOME=/opt/soft/python\n" |
||||
+ "export JAVA_HOME=/opt/java/jdk1.8.0_181-amd64\n" |
||||
+ "export HIVE_HOME=/opt/soft/hive\n" |
||||
+ "export FLINK_HOME=/opt/soft/flink\n" |
||||
+ "export DATAX_HOME=/opt/soft/datax\n" |
||||
+ "export YARN_CONF_DIR=\"/etc/hadoop/conf\"\n" |
||||
+ "\n" |
||||
+ "export PATH=$HADOOP_HOME/bin:$SPARK_HOME1/bin:$SPARK_HOME2/bin:$PYTHON_HOME/bin:$JAVA_HOME/bin:$HIVE_HOME/bin:$FLINK_HOME/bin:$DATAX_HOME/bin:$PATH\n" |
||||
+ "\n" |
||||
+ "export HADOOP_CLASSPATH=`hadoop classpath`\n" |
||||
+ "\n" |
||||
+ "#echo \"HADOOP_CLASSPATH=\"$HADOOP_CLASSPATH"; |
||||
} |
||||
} |
||||
@ -0,0 +1,109 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.dao.mapper; |
||||
|
||||
import org.apache.dolphinscheduler.dao.entity.EnvironmentWorkerGroupRelation; |
||||
|
||||
import java.util.Date; |
||||
import java.util.List; |
||||
|
||||
import org.junit.After; |
||||
import org.junit.Assert; |
||||
import org.junit.Before; |
||||
import org.junit.Test; |
||||
import org.junit.runner.RunWith; |
||||
import org.springframework.beans.factory.annotation.Autowired; |
||||
import org.springframework.boot.test.context.SpringBootTest; |
||||
import org.springframework.test.annotation.Rollback; |
||||
import org.springframework.test.context.junit4.SpringRunner; |
||||
import org.springframework.transaction.annotation.Transactional; |
||||
|
||||
@RunWith(SpringRunner.class) |
||||
@SpringBootTest |
||||
@Transactional |
||||
@Rollback(true) |
||||
public class EnvironmentWorkerGroupRelationMapperTest { |
||||
|
||||
@Autowired |
||||
EnvironmentWorkerGroupRelationMapper environmentWorkerGroupRelationMapper; |
||||
|
||||
@Before |
||||
public void setUp() { |
||||
clearTestData(); |
||||
} |
||||
|
||||
@After |
||||
public void after() { |
||||
clearTestData(); |
||||
} |
||||
|
||||
public void clearTestData() { |
||||
environmentWorkerGroupRelationMapper.selectList(null).stream().forEach(environment -> { |
||||
environmentWorkerGroupRelationMapper.deleteById(environment.getId()); |
||||
}); |
||||
} |
||||
|
||||
/** |
||||
* insert |
||||
* |
||||
* @return ProcessDefinition |
||||
*/ |
||||
private EnvironmentWorkerGroupRelation insertOne() { |
||||
//insertOne
|
||||
EnvironmentWorkerGroupRelation relation = new EnvironmentWorkerGroupRelation(); |
||||
relation.setEnvironmentCode(1L); |
||||
relation.setWorkerGroup("default"); |
||||
relation.setOperator(1); |
||||
relation.setUpdateTime(new Date()); |
||||
relation.setCreateTime(new Date()); |
||||
environmentWorkerGroupRelationMapper.insert(relation); |
||||
return relation; |
||||
} |
||||
|
||||
/** |
||||
* test query |
||||
*/ |
||||
@Test |
||||
public void testQuery() { |
||||
insertOne(); |
||||
//query
|
||||
List<EnvironmentWorkerGroupRelation> relations = environmentWorkerGroupRelationMapper.selectList(null); |
||||
Assert.assertEquals(relations.size(), 1); |
||||
} |
||||
|
||||
@Test |
||||
public void testQueryByEnvironmentCode() { |
||||
EnvironmentWorkerGroupRelation relation = insertOne(); |
||||
List<EnvironmentWorkerGroupRelation> environmentWorkerGroupRelations = environmentWorkerGroupRelationMapper.queryByEnvironmentCode(1L); |
||||
Assert.assertNotEquals(environmentWorkerGroupRelations.size(), 0); |
||||
} |
||||
|
||||
@Test |
||||
public void testQueryByWorkerGroupName() { |
||||
EnvironmentWorkerGroupRelation relation = insertOne(); |
||||
List<EnvironmentWorkerGroupRelation> environmentWorkerGroupRelations = environmentWorkerGroupRelationMapper.queryByWorkerGroupName("default"); |
||||
Assert.assertNotEquals(environmentWorkerGroupRelations.size(), 0); |
||||
} |
||||
|
||||
@Test |
||||
public void testDeleteByCode() { |
||||
EnvironmentWorkerGroupRelation relation = insertOne(); |
||||
int i = environmentWorkerGroupRelationMapper.deleteByCode(1L, "default"); |
||||
Assert.assertNotEquals(i, 0); |
||||
} |
||||
} |
||||
@ -0,0 +1,72 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.remote.command; |
||||
|
||||
import org.apache.dolphinscheduler.common.utils.JSONUtils; |
||||
|
||||
import java.io.Serializable; |
||||
|
||||
/** |
||||
* process host update |
||||
*/ |
||||
public class HostUpdateCommand implements Serializable { |
||||
|
||||
/** |
||||
* task id |
||||
*/ |
||||
private int taskInstanceId; |
||||
|
||||
private String processHost; |
||||
|
||||
public int getTaskInstanceId() { |
||||
return taskInstanceId; |
||||
} |
||||
|
||||
public void setTaskInstanceId(int taskInstanceId) { |
||||
this.taskInstanceId = taskInstanceId; |
||||
} |
||||
|
||||
public String getProcessHost() { |
||||
return processHost; |
||||
} |
||||
|
||||
public void setProcessHost(String processHost) { |
||||
this.processHost = processHost; |
||||
} |
||||
|
||||
/** |
||||
* package request command |
||||
* |
||||
* @return command |
||||
*/ |
||||
public Command convert2Command() { |
||||
Command command = new Command(); |
||||
command.setType(CommandType.PROCESS_HOST_UPDATE_REQUST); |
||||
byte[] body = JSONUtils.toJsonByteArray(this); |
||||
command.setBody(body); |
||||
return command; |
||||
} |
||||
|
||||
@Override |
||||
public String toString() { |
||||
return "HostUpdateCommand{" |
||||
+ "taskInstanceId=" + taskInstanceId |
||||
+ "host=" + processHost |
||||
+ '}'; |
||||
} |
||||
} |
||||
@ -0,0 +1,83 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.remote.command; |
||||
|
||||
import org.apache.dolphinscheduler.common.utils.JSONUtils; |
||||
|
||||
import java.io.Serializable; |
||||
|
||||
public class HostUpdateResponseCommand implements Serializable { |
||||
|
||||
private int taskInstanceId; |
||||
|
||||
private String processHost; |
||||
|
||||
private int status; |
||||
|
||||
public HostUpdateResponseCommand(int taskInstanceId, String processHost, int code) { |
||||
this.taskInstanceId = taskInstanceId; |
||||
this.processHost = processHost; |
||||
this.status = code; |
||||
} |
||||
|
||||
public int getTaskInstanceId() { |
||||
return this.taskInstanceId; |
||||
} |
||||
|
||||
public void setTaskInstanceId(int taskInstanceId) { |
||||
this.taskInstanceId = taskInstanceId; |
||||
} |
||||
|
||||
public String getProcessHost() { |
||||
return this.processHost; |
||||
} |
||||
|
||||
public void setProcessHost(String processHost) { |
||||
this.processHost = processHost; |
||||
} |
||||
|
||||
public int getStatus() { |
||||
return status; |
||||
} |
||||
|
||||
public void setStatus(int status) { |
||||
this.status = status; |
||||
} |
||||
|
||||
/** |
||||
* package request command |
||||
* |
||||
* @return command |
||||
*/ |
||||
public Command convert2Command() { |
||||
Command command = new Command(); |
||||
command.setType(CommandType.PROCESS_HOST_UPDATE_REQUST); |
||||
byte[] body = JSONUtils.toJsonByteArray(this); |
||||
command.setBody(body); |
||||
return command; |
||||
} |
||||
|
||||
@Override |
||||
public String toString() { |
||||
return "HostUpdateResponseCommand{" |
||||
+ "taskInstanceId=" + taskInstanceId |
||||
+ "host=" + processHost |
||||
+ '}'; |
||||
} |
||||
|
||||
} |
||||
@ -0,0 +1,131 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.remote.command; |
||||
|
||||
import org.apache.dolphinscheduler.common.enums.ExecutionStatus; |
||||
import org.apache.dolphinscheduler.common.utils.JSONUtils; |
||||
|
||||
import java.io.Serializable; |
||||
|
||||
/** |
||||
* db task final result response command |
||||
*/ |
||||
public class StateEventChangeCommand implements Serializable { |
||||
|
||||
private String key; |
||||
|
||||
private ExecutionStatus sourceStatus; |
||||
|
||||
private int sourceProcessInstanceId; |
||||
|
||||
private int sourceTaskInstanceId; |
||||
|
||||
private int destProcessInstanceId; |
||||
|
||||
private int destTaskInstanceId; |
||||
|
||||
public StateEventChangeCommand() { |
||||
super(); |
||||
} |
||||
|
||||
public StateEventChangeCommand(int sourceProcessInstanceId, int sourceTaskInstanceId, |
||||
ExecutionStatus sourceStatus, |
||||
int destProcessInstanceId, |
||||
int destTaskInstanceId |
||||
) { |
||||
this.key = String.format("%d-%d-%d-%d", |
||||
sourceProcessInstanceId, |
||||
sourceTaskInstanceId, |
||||
destProcessInstanceId, |
||||
destTaskInstanceId); |
||||
|
||||
this.sourceStatus = sourceStatus; |
||||
this.sourceProcessInstanceId = sourceProcessInstanceId; |
||||
this.sourceTaskInstanceId = sourceTaskInstanceId; |
||||
this.destProcessInstanceId = destProcessInstanceId; |
||||
this.destTaskInstanceId = destTaskInstanceId; |
||||
} |
||||
|
||||
public String getKey() { |
||||
return key; |
||||
} |
||||
|
||||
public void setKey(String key) { |
||||
this.key = key; |
||||
} |
||||
|
||||
/** |
||||
* package response command |
||||
* |
||||
* @return command |
||||
*/ |
||||
public Command convert2Command() { |
||||
Command command = new Command(); |
||||
command.setType(CommandType.STATE_EVENT_REQUEST); |
||||
byte[] body = JSONUtils.toJsonByteArray(this); |
||||
command.setBody(body); |
||||
return command; |
||||
} |
||||
|
||||
@Override |
||||
public String toString() { |
||||
return "StateEventResponseCommand{" |
||||
+ "key=" + key |
||||
+ '}'; |
||||
} |
||||
|
||||
public ExecutionStatus getSourceStatus() { |
||||
return sourceStatus; |
||||
} |
||||
|
||||
public void setSourceStatus(ExecutionStatus sourceStatus) { |
||||
this.sourceStatus = sourceStatus; |
||||
} |
||||
|
||||
public int getSourceProcessInstanceId() { |
||||
return sourceProcessInstanceId; |
||||
} |
||||
|
||||
public void setSourceProcessInstanceId(int sourceProcessInstanceId) { |
||||
this.sourceProcessInstanceId = sourceProcessInstanceId; |
||||
} |
||||
|
||||
public int getSourceTaskInstanceId() { |
||||
return sourceTaskInstanceId; |
||||
} |
||||
|
||||
public void setSourceTaskInstanceId(int sourceTaskInstanceId) { |
||||
this.sourceTaskInstanceId = sourceTaskInstanceId; |
||||
} |
||||
|
||||
public int getDestProcessInstanceId() { |
||||
return destProcessInstanceId; |
||||
} |
||||
|
||||
public void setDestProcessInstanceId(int destProcessInstanceId) { |
||||
this.destProcessInstanceId = destProcessInstanceId; |
||||
} |
||||
|
||||
public int getDestTaskInstanceId() { |
||||
return destTaskInstanceId; |
||||
} |
||||
|
||||
public void setDestTaskInstanceId(int destTaskInstanceId) { |
||||
this.destTaskInstanceId = destTaskInstanceId; |
||||
} |
||||
} |
||||
@ -0,0 +1,78 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.remote.command; |
||||
|
||||
import org.apache.dolphinscheduler.common.utils.JSONUtils; |
||||
|
||||
import java.io.Serializable; |
||||
|
||||
/** |
||||
* db task final result response command |
||||
*/ |
||||
public class StateEventResponseCommand implements Serializable { |
||||
|
||||
private String key; |
||||
private int status; |
||||
|
||||
public StateEventResponseCommand() { |
||||
super(); |
||||
} |
||||
|
||||
public StateEventResponseCommand(int status, String key) { |
||||
this.status = status; |
||||
this.key = key; |
||||
} |
||||
|
||||
public int getStatus() { |
||||
return status; |
||||
} |
||||
|
||||
public void setStatus(int status) { |
||||
this.status = status; |
||||
} |
||||
|
||||
public String getKey() { |
||||
return key; |
||||
} |
||||
|
||||
public void setKey(String key) { |
||||
this.key = key; |
||||
} |
||||
|
||||
/** |
||||
* package response command |
||||
* |
||||
* @return command |
||||
*/ |
||||
public Command convert2Command() { |
||||
Command command = new Command(); |
||||
command.setType(CommandType.DB_TASK_RESPONSE); |
||||
byte[] body = JSONUtils.toJsonByteArray(this); |
||||
command.setBody(body); |
||||
return command; |
||||
} |
||||
|
||||
@Override |
||||
public String toString() { |
||||
return "StateEventResponseCommand{" |
||||
+ "key=" + key |
||||
+ ", status=" + status |
||||
+ '}'; |
||||
} |
||||
|
||||
} |
||||
@ -0,0 +1,125 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.remote.processor; |
||||
|
||||
import static org.apache.dolphinscheduler.common.Constants.SLEEP_TIME_MILLIS; |
||||
|
||||
import org.apache.dolphinscheduler.remote.NettyRemotingClient; |
||||
import org.apache.dolphinscheduler.remote.command.Command; |
||||
import org.apache.dolphinscheduler.remote.config.NettyClientConfig; |
||||
import org.apache.dolphinscheduler.remote.utils.Host; |
||||
|
||||
import java.util.concurrent.ConcurrentHashMap; |
||||
|
||||
import org.slf4j.Logger; |
||||
import org.slf4j.LoggerFactory; |
||||
import org.springframework.stereotype.Service; |
||||
|
||||
import io.netty.channel.Channel; |
||||
|
||||
/** |
||||
* task callback service |
||||
*/ |
||||
@Service |
||||
public class StateEventCallbackService { |
||||
|
||||
private final Logger logger = LoggerFactory.getLogger(StateEventCallbackService.class); |
||||
private static final int[] RETRY_BACKOFF = {1, 2, 3, 5, 10, 20, 40, 100, 100, 100, 100, 200, 200, 200}; |
||||
|
||||
/** |
||||
* remote channels |
||||
*/ |
||||
private static final ConcurrentHashMap<String, NettyRemoteChannel> REMOTE_CHANNELS = new ConcurrentHashMap<>(); |
||||
|
||||
/** |
||||
* netty remoting client |
||||
*/ |
||||
private final NettyRemotingClient nettyRemotingClient; |
||||
|
||||
public StateEventCallbackService() { |
||||
final NettyClientConfig clientConfig = new NettyClientConfig(); |
||||
this.nettyRemotingClient = new NettyRemotingClient(clientConfig); |
||||
} |
||||
|
||||
/** |
||||
* add callback channel |
||||
* |
||||
* @param channel channel |
||||
*/ |
||||
public void addRemoteChannel(String host, NettyRemoteChannel channel) { |
||||
REMOTE_CHANNELS.put(host, channel); |
||||
} |
||||
|
||||
/** |
||||
* get callback channel |
||||
* |
||||
* @param host |
||||
* @return callback channel |
||||
*/ |
||||
private NettyRemoteChannel newRemoteChannel(Host host) { |
||||
Channel newChannel; |
||||
NettyRemoteChannel nettyRemoteChannel = REMOTE_CHANNELS.get(host.getAddress()); |
||||
if (nettyRemoteChannel != null) { |
||||
if (nettyRemoteChannel.isActive()) { |
||||
return nettyRemoteChannel; |
||||
} |
||||
} |
||||
newChannel = nettyRemotingClient.getChannel(host); |
||||
if (newChannel != null) { |
||||
return newRemoteChannel(newChannel, host.getAddress()); |
||||
} |
||||
return null; |
||||
} |
||||
|
||||
public int pause(int ntries) { |
||||
return SLEEP_TIME_MILLIS * RETRY_BACKOFF[ntries % RETRY_BACKOFF.length]; |
||||
} |
||||
|
||||
private NettyRemoteChannel newRemoteChannel(Channel newChannel, long opaque, String host) { |
||||
NettyRemoteChannel remoteChannel = new NettyRemoteChannel(newChannel, opaque); |
||||
addRemoteChannel(host, remoteChannel); |
||||
return remoteChannel; |
||||
} |
||||
|
||||
private NettyRemoteChannel newRemoteChannel(Channel newChannel, String host) { |
||||
NettyRemoteChannel remoteChannel = new NettyRemoteChannel(newChannel); |
||||
addRemoteChannel(host, remoteChannel); |
||||
return remoteChannel; |
||||
} |
||||
|
||||
/** |
||||
* remove callback channels |
||||
*/ |
||||
public void remove(String host) { |
||||
REMOTE_CHANNELS.remove(host); |
||||
} |
||||
|
||||
/** |
||||
* send result |
||||
* |
||||
* @param command command |
||||
*/ |
||||
public void sendResult(String address, int port, Command command) { |
||||
logger.info("send result, host:{}, command:{}", address, command.toString()); |
||||
Host host = new Host(address, port); |
||||
NettyRemoteChannel nettyRemoteChannel = newRemoteChannel(host); |
||||
if (nettyRemoteChannel != null) { |
||||
nettyRemoteChannel.writeAndFlush(command); |
||||
} |
||||
} |
||||
} |
||||
@ -1,175 +0,0 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
package org.apache.dolphinscheduler.server.master.future; |
||||
|
||||
|
||||
import org.apache.dolphinscheduler.remote.command.Command; |
||||
import org.slf4j.Logger; |
||||
import org.slf4j.LoggerFactory; |
||||
|
||||
import java.util.Iterator; |
||||
import java.util.LinkedList; |
||||
import java.util.List; |
||||
import java.util.Map; |
||||
import java.util.concurrent.ConcurrentHashMap; |
||||
import java.util.concurrent.CountDownLatch; |
||||
import java.util.concurrent.TimeUnit; |
||||
import java.util.concurrent.atomic.AtomicReference; |
||||
|
||||
/** |
||||
* task future |
||||
*/ |
||||
public class TaskFuture { |
||||
|
||||
private final static Logger LOGGER = LoggerFactory.getLogger(TaskFuture.class); |
||||
|
||||
private final static ConcurrentHashMap<Long,TaskFuture> FUTURE_TABLE = new ConcurrentHashMap<>(256); |
||||
|
||||
/** |
||||
* request unique identification |
||||
*/ |
||||
private final long opaque; |
||||
|
||||
/** |
||||
* timeout |
||||
*/ |
||||
private final long timeoutMillis; |
||||
|
||||
private final CountDownLatch latch = new CountDownLatch(1); |
||||
|
||||
private final long beginTimestamp = System.currentTimeMillis(); |
||||
|
||||
/** |
||||
* response command |
||||
*/ |
||||
private AtomicReference<Command> responseCommandReference = new AtomicReference<>(); |
||||
|
||||
private volatile boolean sendOk = true; |
||||
|
||||
private AtomicReference<Throwable> causeReference; |
||||
|
||||
public TaskFuture(long opaque, long timeoutMillis) { |
||||
this.opaque = opaque; |
||||
this.timeoutMillis = timeoutMillis; |
||||
FUTURE_TABLE.put(opaque, this); |
||||
} |
||||
|
||||
/** |
||||
* wait for response |
||||
* @return command |
||||
* @throws InterruptedException if error throws InterruptedException |
||||
*/ |
||||
public Command waitResponse() throws InterruptedException { |
||||
this.latch.await(timeoutMillis, TimeUnit.MILLISECONDS); |
||||
return this.responseCommandReference.get(); |
||||
} |
||||
|
||||
/** |
||||
* put response |
||||
* |
||||
* @param responseCommand responseCommand |
||||
*/ |
||||
public void putResponse(final Command responseCommand) { |
||||
responseCommandReference.set(responseCommand); |
||||
this.latch.countDown(); |
||||
FUTURE_TABLE.remove(opaque); |
||||
} |
||||
|
||||
/** |
||||
* whether timeout |
||||
* @return timeout |
||||
*/ |
||||
public boolean isTimeout() { |
||||
long diff = System.currentTimeMillis() - this.beginTimestamp; |
||||
return diff > this.timeoutMillis; |
||||
} |
||||
|
||||
public static void notify(final Command responseCommand){ |
||||
TaskFuture taskFuture = FUTURE_TABLE.remove(responseCommand.getOpaque()); |
||||
if(taskFuture != null){ |
||||
taskFuture.putResponse(responseCommand); |
||||
} |
||||
} |
||||
|
||||
|
||||
public boolean isSendOK() { |
||||
return sendOk; |
||||
} |
||||
|
||||
public void setSendOk(boolean sendOk) { |
||||
this.sendOk = sendOk; |
||||
} |
||||
|
||||
public void setCause(Throwable cause) { |
||||
causeReference.set(cause); |
||||
} |
||||
|
||||
public Throwable getCause() { |
||||
return causeReference.get(); |
||||
} |
||||
|
||||
public long getOpaque() { |
||||
return opaque; |
||||
} |
||||
|
||||
public long getTimeoutMillis() { |
||||
return timeoutMillis; |
||||
} |
||||
|
||||
public long getBeginTimestamp() { |
||||
return beginTimestamp; |
||||
} |
||||
|
||||
public Command getResponseCommand() { |
||||
return responseCommandReference.get(); |
||||
} |
||||
|
||||
public void setResponseCommand(Command responseCommand) { |
||||
responseCommandReference.set(responseCommand); |
||||
} |
||||
|
||||
|
||||
/** |
||||
* scan future table |
||||
*/ |
||||
public static void scanFutureTable(){ |
||||
final List<TaskFuture> futureList = new LinkedList<>(); |
||||
Iterator<Map.Entry<Long, TaskFuture>> it = FUTURE_TABLE.entrySet().iterator(); |
||||
while (it.hasNext()) { |
||||
Map.Entry<Long, TaskFuture> next = it.next(); |
||||
TaskFuture future = next.getValue(); |
||||
if ((future.getBeginTimestamp() + future.getTimeoutMillis() + 1000) <= System.currentTimeMillis()) { |
||||
futureList.add(future); |
||||
it.remove(); |
||||
LOGGER.warn("remove timeout request : {}", future); |
||||
} |
||||
} |
||||
} |
||||
|
||||
@Override |
||||
public String toString() { |
||||
return "TaskFuture{" + |
||||
"opaque=" + opaque + |
||||
", timeoutMillis=" + timeoutMillis + |
||||
", latch=" + latch + |
||||
", beginTimestamp=" + beginTimestamp + |
||||
", responseCommand=" + responseCommandReference.get() + |
||||
", sendOk=" + sendOk + |
||||
", cause=" + causeReference.get() + |
||||
'}'; |
||||
} |
||||
} |
||||
@ -0,0 +1,42 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.server.master.processor; |
||||
|
||||
import org.apache.dolphinscheduler.common.utils.JSONUtils; |
||||
import org.apache.dolphinscheduler.common.utils.Preconditions; |
||||
import org.apache.dolphinscheduler.remote.command.Command; |
||||
import org.apache.dolphinscheduler.remote.command.CommandType; |
||||
import org.apache.dolphinscheduler.remote.processor.NettyRequestProcessor; |
||||
|
||||
import org.slf4j.Logger; |
||||
import org.slf4j.LoggerFactory; |
||||
|
||||
import io.netty.channel.Channel; |
||||
|
||||
public class HostUpdateResponseProcessor implements NettyRequestProcessor { |
||||
|
||||
private final Logger logger = LoggerFactory.getLogger(HostUpdateResponseProcessor.class); |
||||
|
||||
@Override |
||||
public void process(Channel channel, Command command) { |
||||
Preconditions.checkArgument(CommandType.PROCESS_HOST_UPDATE_RESPONSE == command.getType(), String.format("invalid command type : %s", command.getType())); |
||||
|
||||
HostUpdateResponseProcessor responseCommand = JSONUtils.parseObject(command.getBody(), HostUpdateResponseProcessor.class); |
||||
logger.info("received process host response command : {}", responseCommand); |
||||
} |
||||
} |
||||
@ -0,0 +1,74 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.server.master.processor; |
||||
|
||||
import org.apache.dolphinscheduler.common.enums.ExecutionStatus; |
||||
import org.apache.dolphinscheduler.common.enums.StateEvent; |
||||
import org.apache.dolphinscheduler.common.enums.StateEventType; |
||||
import org.apache.dolphinscheduler.common.utils.JSONUtils; |
||||
import org.apache.dolphinscheduler.common.utils.Preconditions; |
||||
import org.apache.dolphinscheduler.remote.command.Command; |
||||
import org.apache.dolphinscheduler.remote.command.CommandType; |
||||
import org.apache.dolphinscheduler.remote.command.StateEventChangeCommand; |
||||
import org.apache.dolphinscheduler.remote.processor.NettyRequestProcessor; |
||||
import org.apache.dolphinscheduler.server.master.processor.queue.StateEventResponseService; |
||||
import org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread; |
||||
import org.apache.dolphinscheduler.service.bean.SpringApplicationContext; |
||||
|
||||
import java.util.concurrent.ConcurrentHashMap; |
||||
|
||||
import org.slf4j.Logger; |
||||
import org.slf4j.LoggerFactory; |
||||
|
||||
import io.netty.channel.Channel; |
||||
|
||||
/** |
||||
* handle state event received from master/api |
||||
*/ |
||||
public class StateEventProcessor implements NettyRequestProcessor { |
||||
|
||||
private final Logger logger = LoggerFactory.getLogger(StateEventProcessor.class); |
||||
|
||||
private StateEventResponseService stateEventResponseService; |
||||
|
||||
public StateEventProcessor() { |
||||
stateEventResponseService = SpringApplicationContext.getBean(StateEventResponseService.class); |
||||
} |
||||
|
||||
public void init(ConcurrentHashMap<Integer, WorkflowExecuteThread> processInstanceExecMaps) { |
||||
this.stateEventResponseService.init(processInstanceExecMaps); |
||||
} |
||||
|
||||
@Override |
||||
public void process(Channel channel, Command command) { |
||||
Preconditions.checkArgument(CommandType.STATE_EVENT_REQUEST == command.getType(), String.format("invalid command type: %s", command.getType())); |
||||
|
||||
StateEventChangeCommand stateEventChangeCommand = JSONUtils.parseObject(command.getBody(), StateEventChangeCommand.class); |
||||
StateEvent stateEvent = new StateEvent(); |
||||
stateEvent.setExecutionStatus(ExecutionStatus.RUNNING_EXECUTION); |
||||
stateEvent.setKey(stateEventChangeCommand.getKey()); |
||||
stateEvent.setProcessInstanceId(stateEventChangeCommand.getDestProcessInstanceId()); |
||||
stateEvent.setTaskInstanceId(stateEventChangeCommand.getDestTaskInstanceId()); |
||||
StateEventType type = stateEvent.getTaskInstanceId() == 0 ? StateEventType.PROCESS_STATE_CHANGE : StateEventType.TASK_STATE_CHANGE; |
||||
stateEvent.setType(type); |
||||
|
||||
logger.info("received command : {}", stateEvent.toString()); |
||||
stateEventResponseService.addResponse(stateEvent); |
||||
} |
||||
|
||||
} |
||||
@ -0,0 +1,149 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.server.master.processor.queue; |
||||
|
||||
import org.apache.dolphinscheduler.common.enums.ExecutionStatus; |
||||
import org.apache.dolphinscheduler.common.enums.StateEvent; |
||||
import org.apache.dolphinscheduler.common.thread.Stopper; |
||||
import org.apache.dolphinscheduler.remote.command.StateEventResponseCommand; |
||||
import org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread; |
||||
|
||||
import java.util.ArrayList; |
||||
import java.util.List; |
||||
import java.util.concurrent.BlockingQueue; |
||||
import java.util.concurrent.ConcurrentHashMap; |
||||
import java.util.concurrent.LinkedBlockingQueue; |
||||
|
||||
import javax.annotation.PostConstruct; |
||||
import javax.annotation.PreDestroy; |
||||
|
||||
import org.slf4j.Logger; |
||||
import org.slf4j.LoggerFactory; |
||||
import org.springframework.stereotype.Component; |
||||
|
||||
import io.netty.channel.Channel; |
||||
|
||||
/** |
||||
* task manager |
||||
*/ |
||||
@Component |
||||
public class StateEventResponseService { |
||||
|
||||
/** |
||||
* logger |
||||
*/ |
||||
private final Logger logger = LoggerFactory.getLogger(StateEventResponseService.class); |
||||
|
||||
/** |
||||
* attemptQueue |
||||
*/ |
||||
private final BlockingQueue<StateEvent> eventQueue = new LinkedBlockingQueue<>(5000); |
||||
|
||||
/** |
||||
* task response worker |
||||
*/ |
||||
private Thread responseWorker; |
||||
|
||||
private ConcurrentHashMap<Integer, WorkflowExecuteThread> processInstanceMapper; |
||||
|
||||
public void init(ConcurrentHashMap<Integer, WorkflowExecuteThread> processInstanceMapper) { |
||||
if (this.processInstanceMapper == null) { |
||||
this.processInstanceMapper = processInstanceMapper; |
||||
} |
||||
} |
||||
|
||||
@PostConstruct |
||||
public void start() { |
||||
this.responseWorker = new StateEventResponseWorker(); |
||||
this.responseWorker.setName("StateEventResponseWorker"); |
||||
this.responseWorker.start(); |
||||
} |
||||
|
||||
@PreDestroy |
||||
public void stop() { |
||||
this.responseWorker.interrupt(); |
||||
if (!eventQueue.isEmpty()) { |
||||
List<StateEvent> remainEvents = new ArrayList<>(eventQueue.size()); |
||||
eventQueue.drainTo(remainEvents); |
||||
for (StateEvent event : remainEvents) { |
||||
this.persist(event); |
||||
} |
||||
} |
||||
} |
||||
|
||||
/** |
||||
* put task to attemptQueue |
||||
*/ |
||||
public void addResponse(StateEvent stateEvent) { |
||||
try { |
||||
eventQueue.put(stateEvent); |
||||
} catch (InterruptedException e) { |
||||
logger.error("put state event : {} error :{}", stateEvent, e); |
||||
Thread.currentThread().interrupt(); |
||||
} |
||||
} |
||||
|
||||
/** |
||||
* task worker thread |
||||
*/ |
||||
class StateEventResponseWorker extends Thread { |
||||
|
||||
@Override |
||||
public void run() { |
||||
|
||||
while (Stopper.isRunning()) { |
||||
try { |
||||
// if not task , blocking here
|
||||
StateEvent stateEvent = eventQueue.take(); |
||||
persist(stateEvent); |
||||
} catch (InterruptedException e) { |
||||
logger.warn("persist task error", e); |
||||
Thread.currentThread().interrupt(); |
||||
} |
||||
} |
||||
logger.info("StateEventResponseWorker stopped"); |
||||
} |
||||
} |
||||
|
||||
private void writeResponse(StateEvent stateEvent, ExecutionStatus status) { |
||||
Channel channel = stateEvent.getChannel(); |
||||
if (channel != null) { |
||||
StateEventResponseCommand command = new StateEventResponseCommand(status.getCode(), stateEvent.getKey()); |
||||
channel.writeAndFlush(command.convert2Command()); |
||||
} |
||||
} |
||||
|
||||
private void persist(StateEvent stateEvent) { |
||||
try { |
||||
if (!this.processInstanceMapper.containsKey(stateEvent.getProcessInstanceId())) { |
||||
writeResponse(stateEvent, ExecutionStatus.FAILURE); |
||||
return; |
||||
} |
||||
|
||||
WorkflowExecuteThread workflowExecuteThread = this.processInstanceMapper.get(stateEvent.getProcessInstanceId()); |
||||
workflowExecuteThread.addStateEvent(stateEvent); |
||||
writeResponse(stateEvent, ExecutionStatus.SUCCESS); |
||||
} catch (Exception e) { |
||||
logger.error("persist event queue error:", stateEvent.toString(), e); |
||||
} |
||||
} |
||||
|
||||
public BlockingQueue<StateEvent> getEventQueue() { |
||||
return eventQueue; |
||||
} |
||||
} |
||||
@ -0,0 +1,195 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.server.master.runner; |
||||
|
||||
import org.apache.dolphinscheduler.common.enums.ExecutionStatus; |
||||
import org.apache.dolphinscheduler.common.enums.StateEvent; |
||||
import org.apache.dolphinscheduler.common.enums.StateEventType; |
||||
import org.apache.dolphinscheduler.common.thread.Stopper; |
||||
import org.apache.dolphinscheduler.common.thread.ThreadUtils; |
||||
import org.apache.dolphinscheduler.common.utils.NetUtils; |
||||
import org.apache.dolphinscheduler.common.utils.StringUtils; |
||||
import org.apache.dolphinscheduler.dao.entity.ProcessInstance; |
||||
import org.apache.dolphinscheduler.dao.entity.TaskInstance; |
||||
import org.apache.dolphinscheduler.remote.command.StateEventChangeCommand; |
||||
import org.apache.dolphinscheduler.remote.processor.StateEventCallbackService; |
||||
import org.apache.dolphinscheduler.server.master.config.MasterConfig; |
||||
import org.apache.dolphinscheduler.service.bean.SpringApplicationContext; |
||||
import org.apache.dolphinscheduler.service.process.ProcessService; |
||||
|
||||
import java.util.Map; |
||||
import java.util.concurrent.ConcurrentHashMap; |
||||
import java.util.concurrent.ExecutorService; |
||||
|
||||
import org.slf4j.Logger; |
||||
import org.slf4j.LoggerFactory; |
||||
import org.springframework.beans.factory.annotation.Autowired; |
||||
import org.springframework.stereotype.Service; |
||||
|
||||
import com.google.common.util.concurrent.FutureCallback; |
||||
import com.google.common.util.concurrent.Futures; |
||||
import com.google.common.util.concurrent.ListenableFuture; |
||||
import com.google.common.util.concurrent.ListeningExecutorService; |
||||
import com.google.common.util.concurrent.MoreExecutors; |
||||
|
||||
@Service |
||||
public class EventExecuteService extends Thread { |
||||
|
||||
private static final Logger logger = LoggerFactory.getLogger(EventExecuteService.class); |
||||
|
||||
|
||||
/** |
||||
* dolphinscheduler database interface
|
||||
*/ |
||||
@Autowired |
||||
private ProcessService processService; |
||||
|
||||
@Autowired |
||||
private MasterConfig masterConfig; |
||||
|
||||
private ExecutorService eventExecService; |
||||
|
||||
/** |
||||
* |
||||
*/ |
||||
private StateEventCallbackService stateEventCallbackService; |
||||
|
||||
|
||||
private ConcurrentHashMap<Integer, WorkflowExecuteThread> processInstanceExecMaps; |
||||
private ConcurrentHashMap<String, WorkflowExecuteThread> eventHandlerMap = new ConcurrentHashMap(); |
||||
ListeningExecutorService listeningExecutorService; |
||||
|
||||
public void init(ConcurrentHashMap<Integer, WorkflowExecuteThread> processInstanceExecMaps) { |
||||
|
||||
eventExecService = ThreadUtils.newDaemonFixedThreadExecutor("MasterEventExecution", masterConfig.getMasterExecThreads()); |
||||
|
||||
this.processInstanceExecMaps = processInstanceExecMaps; |
||||
|
||||
listeningExecutorService = MoreExecutors.listeningDecorator(eventExecService); |
||||
this.stateEventCallbackService = SpringApplicationContext.getBean(StateEventCallbackService.class); |
||||
|
||||
} |
||||
|
||||
@Override |
||||
public synchronized void start() { |
||||
super.setName("EventServiceStarted"); |
||||
super.start(); |
||||
} |
||||
|
||||
public void close() { |
||||
eventExecService.shutdown(); |
||||
logger.info("event service stopped..."); |
||||
} |
||||
|
||||
@Override |
||||
public void run() { |
||||
logger.info("Event service started"); |
||||
while (Stopper.isRunning()) { |
||||
try { |
||||
eventHandler(); |
||||
|
||||
} catch (Exception e) { |
||||
logger.error("Event service thread error", e); |
||||
} |
||||
} |
||||
} |
||||
|
||||
private void eventHandler() { |
||||
for (WorkflowExecuteThread workflowExecuteThread : this.processInstanceExecMaps.values()) { |
||||
if (workflowExecuteThread.eventSize() == 0 |
||||
|| StringUtils.isEmpty(workflowExecuteThread.getKey()) |
||||
|| eventHandlerMap.containsKey(workflowExecuteThread.getKey())) { |
||||
continue; |
||||
} |
||||
int processInstanceId = workflowExecuteThread.getProcessInstance().getId(); |
||||
logger.info("handle process instance : {} events, count:{}", |
||||
processInstanceId, |
||||
workflowExecuteThread.eventSize()); |
||||
logger.info("already exists handler process size:{}", this.eventHandlerMap.size()); |
||||
eventHandlerMap.put(workflowExecuteThread.getKey(), workflowExecuteThread); |
||||
ListenableFuture future = this.listeningExecutorService.submit(workflowExecuteThread); |
||||
FutureCallback futureCallback = new FutureCallback() { |
||||
@Override |
||||
public void onSuccess(Object o) { |
||||
if (workflowExecuteThread.workFlowFinish()) { |
||||
processInstanceExecMaps.remove(processInstanceId); |
||||
notifyProcessChanged(); |
||||
logger.info("process instance {} finished.", processInstanceId); |
||||
} |
||||
if (workflowExecuteThread.getProcessInstance().getId() != processInstanceId) { |
||||
processInstanceExecMaps.remove(processInstanceId); |
||||
processInstanceExecMaps.put(workflowExecuteThread.getProcessInstance().getId(), workflowExecuteThread); |
||||
|
||||
} |
||||
eventHandlerMap.remove(workflowExecuteThread.getKey()); |
||||
} |
||||
|
||||
private void notifyProcessChanged() { |
||||
Map<ProcessInstance, TaskInstance> fatherMaps |
||||
= processService.notifyProcessList(processInstanceId, 0); |
||||
|
||||
for (ProcessInstance processInstance : fatherMaps.keySet()) { |
||||
String address = NetUtils.getAddr(masterConfig.getListenPort()); |
||||
if (processInstance.getHost().equalsIgnoreCase(address)) { |
||||
notifyMyself(processInstance, fatherMaps.get(processInstance)); |
||||
} else { |
||||
notifyProcess(processInstance, fatherMaps.get(processInstance)); |
||||
} |
||||
} |
||||
} |
||||
|
||||
private void notifyMyself(ProcessInstance processInstance, TaskInstance taskInstance) { |
||||
logger.info("notify process {} task {} state change", processInstance.getId(), taskInstance.getId()); |
||||
if (!processInstanceExecMaps.containsKey(processInstance.getId())) { |
||||
return; |
||||
} |
||||
WorkflowExecuteThread workflowExecuteThreadNotify = processInstanceExecMaps.get(processInstance.getId()); |
||||
StateEvent stateEvent = new StateEvent(); |
||||
stateEvent.setTaskInstanceId(taskInstance.getId()); |
||||
stateEvent.setType(StateEventType.TASK_STATE_CHANGE); |
||||
stateEvent.setProcessInstanceId(processInstance.getId()); |
||||
stateEvent.setExecutionStatus(ExecutionStatus.RUNNING_EXECUTION); |
||||
workflowExecuteThreadNotify.addStateEvent(stateEvent); |
||||
} |
||||
|
||||
private void notifyProcess(ProcessInstance processInstance, TaskInstance taskInstance) { |
||||
String host = processInstance.getHost(); |
||||
if (StringUtils.isEmpty(host)) { |
||||
logger.info("process {} host is empty, cannot notify task {} now.", |
||||
processInstance.getId(), taskInstance.getId()); |
||||
return; |
||||
} |
||||
String address = host.split(":")[0]; |
||||
int port = Integer.parseInt(host.split(":")[1]); |
||||
logger.info("notify process {} task {} state change, host:{}", |
||||
processInstance.getId(), taskInstance.getId(), host); |
||||
StateEventChangeCommand stateEventChangeCommand = new StateEventChangeCommand( |
||||
processInstanceId, 0, workflowExecuteThread.getProcessInstance().getState(), processInstance.getId(), taskInstance.getId() |
||||
); |
||||
|
||||
stateEventCallbackService.sendResult(address, port, stateEventChangeCommand.convert2Command()); |
||||
} |
||||
|
||||
@Override |
||||
public void onFailure(Throwable throwable) { |
||||
} |
||||
}; |
||||
Futures.addCallback(future, futureCallback, this.listeningExecutorService); |
||||
} |
||||
} |
||||
} |
||||
@ -0,0 +1,154 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.server.master.runner; |
||||
|
||||
import org.apache.dolphinscheduler.common.Constants; |
||||
import org.apache.dolphinscheduler.common.enums.ExecutionStatus; |
||||
import org.apache.dolphinscheduler.common.enums.StateEvent; |
||||
import org.apache.dolphinscheduler.common.enums.StateEventType; |
||||
import org.apache.dolphinscheduler.common.enums.TimeoutFlag; |
||||
import org.apache.dolphinscheduler.common.thread.Stopper; |
||||
import org.apache.dolphinscheduler.common.utils.DateUtils; |
||||
import org.apache.dolphinscheduler.dao.entity.ProcessInstance; |
||||
import org.apache.dolphinscheduler.dao.entity.TaskInstance; |
||||
|
||||
import org.apache.hadoop.util.ThreadUtil; |
||||
|
||||
import java.util.concurrent.ConcurrentHashMap; |
||||
|
||||
import org.slf4j.Logger; |
||||
import org.slf4j.LoggerFactory; |
||||
|
||||
/** |
||||
* 1. timeout check wheel |
||||
* 2. dependent task check wheel |
||||
*/ |
||||
public class StateWheelExecuteThread extends Thread { |
||||
|
||||
private static final Logger logger = LoggerFactory.getLogger(StateWheelExecuteThread.class); |
||||
|
||||
ConcurrentHashMap<Integer, ProcessInstance> processInstanceCheckList; |
||||
ConcurrentHashMap<Integer, TaskInstance> taskInstanceCheckList; |
||||
private ConcurrentHashMap<Integer, WorkflowExecuteThread> processInstanceExecMaps; |
||||
|
||||
private int stateCheckIntervalSecs; |
||||
|
||||
public StateWheelExecuteThread(ConcurrentHashMap<Integer, ProcessInstance> processInstances, |
||||
ConcurrentHashMap<Integer, TaskInstance> taskInstances, |
||||
ConcurrentHashMap<Integer, WorkflowExecuteThread> processInstanceExecMaps, |
||||
int stateCheckIntervalSecs) { |
||||
this.processInstanceCheckList = processInstances; |
||||
this.taskInstanceCheckList = taskInstances; |
||||
this.processInstanceExecMaps = processInstanceExecMaps; |
||||
this.stateCheckIntervalSecs = stateCheckIntervalSecs; |
||||
} |
||||
|
||||
@Override |
||||
public void run() { |
||||
|
||||
logger.info("state wheel thread start"); |
||||
while (Stopper.isRunning()) { |
||||
try { |
||||
checkProcess(); |
||||
checkTask(); |
||||
} catch (Exception e) { |
||||
logger.error("state wheel thread check error:", e); |
||||
} |
||||
ThreadUtil.sleepAtLeastIgnoreInterrupts(stateCheckIntervalSecs); |
||||
} |
||||
} |
||||
|
||||
public boolean addProcess(ProcessInstance processInstance) { |
||||
this.processInstanceCheckList.put(processInstance.getId(), processInstance); |
||||
return true; |
||||
} |
||||
|
||||
public boolean addTask(TaskInstance taskInstance) { |
||||
this.taskInstanceCheckList.put(taskInstance.getId(), taskInstance); |
||||
return true; |
||||
} |
||||
|
||||
private void checkTask() { |
||||
if (taskInstanceCheckList.isEmpty()) { |
||||
return; |
||||
} |
||||
|
||||
for (TaskInstance taskInstance : this.taskInstanceCheckList.values()) { |
||||
if (TimeoutFlag.OPEN == taskInstance.getTaskDefine().getTimeoutFlag()) { |
||||
long timeRemain = DateUtils.getRemainTime(taskInstance.getStartTime(), taskInstance.getTaskDefine().getTimeout() * Constants.SEC_2_MINUTES_TIME_UNIT); |
||||
if (0 <= timeRemain && processTimeout(taskInstance)) { |
||||
taskInstanceCheckList.remove(taskInstance.getId()); |
||||
return; |
||||
} |
||||
} |
||||
if (taskInstance.isSubProcess() || taskInstance.isDependTask()) { |
||||
processDependCheck(taskInstance); |
||||
} |
||||
} |
||||
} |
||||
|
||||
private void checkProcess() { |
||||
if (processInstanceCheckList.isEmpty()) { |
||||
return; |
||||
} |
||||
for (ProcessInstance processInstance : this.processInstanceCheckList.values()) { |
||||
|
||||
long timeRemain = DateUtils.getRemainTime(processInstance.getStartTime(), processInstance.getTimeout() * Constants.SEC_2_MINUTES_TIME_UNIT); |
||||
if (0 <= timeRemain && processTimeout(processInstance)) { |
||||
processInstanceCheckList.remove(processInstance.getId()); |
||||
} |
||||
} |
||||
} |
||||
|
||||
private void putEvent(StateEvent stateEvent) { |
||||
|
||||
if (!processInstanceExecMaps.containsKey(stateEvent.getProcessInstanceId())) { |
||||
return; |
||||
} |
||||
WorkflowExecuteThread workflowExecuteThread = this.processInstanceExecMaps.get(stateEvent.getProcessInstanceId()); |
||||
workflowExecuteThread.addStateEvent(stateEvent); |
||||
} |
||||
|
||||
private boolean processDependCheck(TaskInstance taskInstance) { |
||||
StateEvent stateEvent = new StateEvent(); |
||||
stateEvent.setType(StateEventType.TASK_STATE_CHANGE); |
||||
stateEvent.setProcessInstanceId(taskInstance.getProcessInstanceId()); |
||||
stateEvent.setTaskInstanceId(taskInstance.getId()); |
||||
stateEvent.setExecutionStatus(ExecutionStatus.RUNNING_EXECUTION); |
||||
putEvent(stateEvent); |
||||
return true; |
||||
} |
||||
|
||||
private boolean processTimeout(TaskInstance taskInstance) { |
||||
StateEvent stateEvent = new StateEvent(); |
||||
stateEvent.setType(StateEventType.TASK_TIMEOUT); |
||||
stateEvent.setProcessInstanceId(taskInstance.getProcessInstanceId()); |
||||
stateEvent.setTaskInstanceId(taskInstance.getId()); |
||||
putEvent(stateEvent); |
||||
return true; |
||||
} |
||||
|
||||
private boolean processTimeout(ProcessInstance processInstance) { |
||||
StateEvent stateEvent = new StateEvent(); |
||||
stateEvent.setType(StateEventType.PROCESS_TIMEOUT); |
||||
stateEvent.setProcessInstanceId(processInstance.getId()); |
||||
putEvent(stateEvent); |
||||
return true; |
||||
} |
||||
|
||||
} |
||||
@ -0,0 +1,112 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.server.master.runner.task; |
||||
|
||||
import org.apache.dolphinscheduler.dao.entity.ProcessInstance; |
||||
import org.apache.dolphinscheduler.dao.entity.TaskInstance; |
||||
|
||||
import org.slf4j.Logger; |
||||
import org.slf4j.LoggerFactory; |
||||
|
||||
public abstract class BaseTaskProcessor implements ITaskProcessor { |
||||
|
||||
protected Logger logger = LoggerFactory.getLogger(getClass()); |
||||
|
||||
protected boolean killed = false; |
||||
|
||||
protected boolean paused = false; |
||||
|
||||
protected boolean timeout = false; |
||||
|
||||
protected TaskInstance taskInstance = null; |
||||
|
||||
protected ProcessInstance processInstance; |
||||
|
||||
/** |
||||
* pause task, common tasks donot need this. |
||||
* |
||||
* @return |
||||
*/ |
||||
protected abstract boolean pauseTask(); |
||||
|
||||
/** |
||||
* kill task, all tasks need to realize this function |
||||
* |
||||
* @return |
||||
*/ |
||||
protected abstract boolean killTask(); |
||||
|
||||
/** |
||||
* task timeout process |
||||
* @return |
||||
*/ |
||||
protected abstract boolean taskTimeout(); |
||||
|
||||
@Override |
||||
public void run() { |
||||
} |
||||
|
||||
@Override |
||||
public boolean action(TaskAction taskAction) { |
||||
|
||||
switch (taskAction) { |
||||
case STOP: |
||||
return stop(); |
||||
case PAUSE: |
||||
return pause(); |
||||
case TIMEOUT: |
||||
return timeout(); |
||||
default: |
||||
logger.error("unknown task action: {}", taskAction.toString()); |
||||
|
||||
} |
||||
return false; |
||||
} |
||||
|
||||
protected boolean timeout() { |
||||
if (timeout) { |
||||
return true; |
||||
} |
||||
timeout = taskTimeout(); |
||||
return timeout; |
||||
} |
||||
|
||||
/** |
||||
* @return |
||||
*/ |
||||
protected boolean pause() { |
||||
if (paused) { |
||||
return true; |
||||
} |
||||
paused = pauseTask(); |
||||
return paused; |
||||
} |
||||
|
||||
protected boolean stop() { |
||||
if (killed) { |
||||
return true; |
||||
} |
||||
killed = killTask(); |
||||
return killed; |
||||
} |
||||
|
||||
@Override |
||||
public String getType() { |
||||
return null; |
||||
} |
||||
} |
||||
@ -0,0 +1,33 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.server.master.runner.task; |
||||
|
||||
import org.apache.dolphinscheduler.common.Constants; |
||||
|
||||
public class CommonTaskProcessFactory implements ITaskProcessFactory { |
||||
@Override |
||||
public String type() { |
||||
return Constants.COMMON_TASK_TYPE; |
||||
|
||||
} |
||||
|
||||
@Override |
||||
public ITaskProcessor create() { |
||||
return new CommonTaskProcessor(); |
||||
} |
||||
} |
||||
@ -0,0 +1,175 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.server.master.runner.task; |
||||
|
||||
import org.apache.dolphinscheduler.common.Constants; |
||||
import org.apache.dolphinscheduler.common.enums.ExecutionStatus; |
||||
import org.apache.dolphinscheduler.common.utils.JSONUtils; |
||||
import org.apache.dolphinscheduler.dao.entity.ProcessInstance; |
||||
import org.apache.dolphinscheduler.dao.entity.TaskInstance; |
||||
import org.apache.dolphinscheduler.remote.command.TaskKillRequestCommand; |
||||
import org.apache.dolphinscheduler.remote.utils.Host; |
||||
import org.apache.dolphinscheduler.server.master.config.MasterConfig; |
||||
import org.apache.dolphinscheduler.server.master.dispatch.context.ExecutionContext; |
||||
import org.apache.dolphinscheduler.server.master.dispatch.enums.ExecutorType; |
||||
import org.apache.dolphinscheduler.server.master.dispatch.exceptions.ExecuteException; |
||||
import org.apache.dolphinscheduler.server.master.dispatch.executor.NettyExecutorManager; |
||||
import org.apache.dolphinscheduler.service.bean.SpringApplicationContext; |
||||
import org.apache.dolphinscheduler.service.process.ProcessService; |
||||
import org.apache.dolphinscheduler.service.queue.TaskPriority; |
||||
import org.apache.dolphinscheduler.service.queue.TaskPriorityQueue; |
||||
import org.apache.dolphinscheduler.service.queue.TaskPriorityQueueImpl; |
||||
|
||||
import java.util.Date; |
||||
|
||||
import org.slf4j.Logger; |
||||
import org.slf4j.LoggerFactory; |
||||
import org.springframework.beans.factory.annotation.Autowired; |
||||
|
||||
/** |
||||
* common task processor |
||||
*/ |
||||
public class CommonTaskProcessor extends BaseTaskProcessor { |
||||
|
||||
@Autowired |
||||
private TaskPriorityQueue taskUpdateQueue; |
||||
|
||||
@Autowired |
||||
MasterConfig masterConfig; |
||||
|
||||
@Autowired |
||||
NettyExecutorManager nettyExecutorManager; |
||||
|
||||
/** |
||||
* logger of MasterBaseTaskExecThread |
||||
*/ |
||||
protected Logger logger = LoggerFactory.getLogger(getClass()); |
||||
|
||||
protected ProcessService processService = SpringApplicationContext.getBean(ProcessService.class); |
||||
|
||||
@Override |
||||
public boolean submit(TaskInstance task, ProcessInstance processInstance, int maxRetryTimes, int commitInterval) { |
||||
this.processInstance = processInstance; |
||||
this.taskInstance = processService.submitTask(task, maxRetryTimes, commitInterval); |
||||
|
||||
if (this.taskInstance == null) { |
||||
return false; |
||||
} |
||||
dispatchTask(taskInstance, processInstance); |
||||
return true; |
||||
} |
||||
|
||||
@Override |
||||
public ExecutionStatus taskState() { |
||||
return this.taskInstance.getState(); |
||||
} |
||||
|
||||
@Override |
||||
public void run() { |
||||
} |
||||
|
||||
@Override |
||||
protected boolean taskTimeout() { |
||||
return true; |
||||
} |
||||
|
||||
/** |
||||
* common task cannot be paused |
||||
*/ |
||||
@Override |
||||
protected boolean pauseTask() { |
||||
return true; |
||||
} |
||||
|
||||
@Override |
||||
public String getType() { |
||||
return Constants.COMMON_TASK_TYPE; |
||||
} |
||||
|
||||
private boolean dispatchTask(TaskInstance taskInstance, ProcessInstance processInstance) { |
||||
|
||||
try { |
||||
if (taskUpdateQueue == null) { |
||||
this.initQueue(); |
||||
} |
||||
if (taskInstance.getState().typeIsFinished()) { |
||||
logger.info(String.format("submit task , but task [%s] state [%s] is already finished. ", taskInstance.getName(), taskInstance.getState().toString())); |
||||
return true; |
||||
} |
||||
// task cannot be submitted because its execution state is RUNNING or DELAY.
|
||||
if (taskInstance.getState() == ExecutionStatus.RUNNING_EXECUTION |
||||
|| taskInstance.getState() == ExecutionStatus.DELAY_EXECUTION) { |
||||
logger.info("submit task, but the status of the task {} is already running or delayed.", taskInstance.getName()); |
||||
return true; |
||||
} |
||||
logger.info("task ready to submit: {}", taskInstance); |
||||
|
||||
TaskPriority taskPriority = new TaskPriority(processInstance.getProcessInstancePriority().getCode(), |
||||
processInstance.getId(), taskInstance.getProcessInstancePriority().getCode(), |
||||
taskInstance.getId(), org.apache.dolphinscheduler.common.Constants.DEFAULT_WORKER_GROUP); |
||||
taskUpdateQueue.put(taskPriority); |
||||
logger.info(String.format("master submit success, task : %s", taskInstance.getName())); |
||||
return true; |
||||
} catch (Exception e) { |
||||
logger.error("submit task Exception: ", e); |
||||
logger.error("task error : %s", JSONUtils.toJsonString(taskInstance)); |
||||
return false; |
||||
} |
||||
} |
||||
|
||||
public void initQueue() { |
||||
this.taskUpdateQueue = SpringApplicationContext.getBean(TaskPriorityQueueImpl.class); |
||||
} |
||||
|
||||
@Override |
||||
public boolean killTask() { |
||||
|
||||
try { |
||||
taskInstance = processService.findTaskInstanceById(taskInstance.getId()); |
||||
if (taskInstance == null) { |
||||
return true; |
||||
} |
||||
if (taskInstance.getState().typeIsFinished()) { |
||||
return true; |
||||
} |
||||
if (null == taskInstance.getHost() || taskInstance.getHost().isEmpty()) { |
||||
taskInstance.setState(ExecutionStatus.KILL); |
||||
taskInstance.setEndTime(new Date()); |
||||
processService.updateTaskInstance(taskInstance); |
||||
return true; |
||||
} |
||||
|
||||
TaskKillRequestCommand killCommand = new TaskKillRequestCommand(); |
||||
killCommand.setTaskInstanceId(taskInstance.getId()); |
||||
|
||||
ExecutionContext executionContext = new ExecutionContext(killCommand.convert2Command(), ExecutorType.WORKER); |
||||
|
||||
Host host = Host.of(taskInstance.getHost()); |
||||
executionContext.setHost(host); |
||||
|
||||
nettyExecutorManager.executeDirectly(executionContext); |
||||
} catch (ExecuteException e) { |
||||
logger.error("kill task error:", e); |
||||
return false; |
||||
} |
||||
|
||||
logger.info("master kill taskInstance name :{} taskInstance id:{}", |
||||
taskInstance.getName(), taskInstance.getId()); |
||||
return true; |
||||
} |
||||
} |
||||
@ -0,0 +1,32 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.server.master.runner.task; |
||||
|
||||
import org.apache.dolphinscheduler.common.enums.TaskType; |
||||
|
||||
public class ConditionTaskProcessFactory implements ITaskProcessFactory { |
||||
@Override |
||||
public String type() { |
||||
return TaskType.CONDITIONS.getDesc(); |
||||
} |
||||
|
||||
@Override |
||||
public ITaskProcessor create() { |
||||
return new ConditionTaskProcessor(); |
||||
} |
||||
} |
||||
@ -0,0 +1,210 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.server.master.runner.task; |
||||
|
||||
import org.apache.dolphinscheduler.common.Constants; |
||||
import org.apache.dolphinscheduler.common.enums.DependResult; |
||||
import org.apache.dolphinscheduler.common.enums.ExecutionStatus; |
||||
import org.apache.dolphinscheduler.common.enums.TaskTimeoutStrategy; |
||||
import org.apache.dolphinscheduler.common.enums.TaskType; |
||||
import org.apache.dolphinscheduler.common.model.DependentItem; |
||||
import org.apache.dolphinscheduler.common.model.DependentTaskModel; |
||||
import org.apache.dolphinscheduler.common.task.dependent.DependentParameters; |
||||
import org.apache.dolphinscheduler.common.utils.DependentUtils; |
||||
import org.apache.dolphinscheduler.common.utils.LoggerUtils; |
||||
import org.apache.dolphinscheduler.common.utils.NetUtils; |
||||
import org.apache.dolphinscheduler.dao.entity.ProcessInstance; |
||||
import org.apache.dolphinscheduler.dao.entity.TaskDefinition; |
||||
import org.apache.dolphinscheduler.dao.entity.TaskInstance; |
||||
import org.apache.dolphinscheduler.server.master.config.MasterConfig; |
||||
import org.apache.dolphinscheduler.server.utils.LogUtils; |
||||
import org.apache.dolphinscheduler.service.bean.SpringApplicationContext; |
||||
import org.apache.dolphinscheduler.service.process.ProcessService; |
||||
|
||||
import java.util.ArrayList; |
||||
import java.util.Date; |
||||
import java.util.List; |
||||
import java.util.Map; |
||||
import java.util.concurrent.ConcurrentHashMap; |
||||
|
||||
import org.slf4j.LoggerFactory; |
||||
|
||||
/** |
||||
* condition task processor |
||||
*/ |
||||
public class ConditionTaskProcessor extends BaseTaskProcessor { |
||||
|
||||
/** |
||||
* dependent parameters |
||||
*/ |
||||
private DependentParameters dependentParameters; |
||||
|
||||
ProcessInstance processInstance; |
||||
|
||||
/** |
||||
* condition result |
||||
*/ |
||||
private DependResult conditionResult = DependResult.WAITING; |
||||
|
||||
/** |
||||
* complete task map |
||||
*/ |
||||
private Map<String, ExecutionStatus> completeTaskList = new ConcurrentHashMap<>(); |
||||
|
||||
protected ProcessService processService = SpringApplicationContext.getBean(ProcessService.class); |
||||
MasterConfig masterConfig = SpringApplicationContext.getBean(MasterConfig.class); |
||||
|
||||
private TaskDefinition taskDefinition; |
||||
|
||||
@Override |
||||
public boolean submit(TaskInstance task, ProcessInstance processInstance, int masterTaskCommitRetryTimes, int masterTaskCommitInterval) { |
||||
this.processInstance = processInstance; |
||||
this.taskInstance = processService.submitTask(task, masterTaskCommitRetryTimes, masterTaskCommitInterval); |
||||
|
||||
if (this.taskInstance == null) { |
||||
return false; |
||||
} |
||||
taskDefinition = processService.findTaskDefinition( |
||||
taskInstance.getTaskCode(), taskInstance.getTaskDefinitionVersion() |
||||
); |
||||
|
||||
logger = LoggerFactory.getLogger(LoggerUtils.buildTaskId(LoggerUtils.TASK_LOGGER_INFO_PREFIX, |
||||
processInstance.getProcessDefinitionCode(), |
||||
processInstance.getProcessDefinitionVersion(), |
||||
taskInstance.getProcessInstanceId(), |
||||
taskInstance.getId())); |
||||
String threadLoggerInfoName = String.format(Constants.TASK_LOG_INFO_FORMAT, processService.formatTaskAppId(this.taskInstance)); |
||||
Thread.currentThread().setName(threadLoggerInfoName); |
||||
initTaskParameters(); |
||||
logger.info("dependent task start"); |
||||
endTask(); |
||||
return true; |
||||
} |
||||
|
||||
@Override |
||||
public ExecutionStatus taskState() { |
||||
return this.taskInstance.getState(); |
||||
} |
||||
|
||||
@Override |
||||
public void run() { |
||||
if (conditionResult.equals(DependResult.WAITING)) { |
||||
setConditionResult(); |
||||
} else { |
||||
endTask(); |
||||
} |
||||
} |
||||
|
||||
@Override |
||||
protected boolean pauseTask() { |
||||
this.taskInstance.setState(ExecutionStatus.PAUSE); |
||||
this.taskInstance.setEndTime(new Date()); |
||||
processService.saveTaskInstance(taskInstance); |
||||
return true; |
||||
} |
||||
|
||||
@Override |
||||
protected boolean taskTimeout() { |
||||
TaskTimeoutStrategy taskTimeoutStrategy = |
||||
taskDefinition.getTimeoutNotifyStrategy(); |
||||
if (taskTimeoutStrategy == TaskTimeoutStrategy.WARN) { |
||||
return true; |
||||
} |
||||
logger.info("condition task {} timeout, strategy {} ", |
||||
taskInstance.getId(), taskTimeoutStrategy.getDescp()); |
||||
conditionResult = DependResult.FAILED; |
||||
endTask(); |
||||
return true; |
||||
} |
||||
|
||||
@Override |
||||
protected boolean killTask() { |
||||
this.taskInstance.setState(ExecutionStatus.KILL); |
||||
this.taskInstance.setEndTime(new Date()); |
||||
processService.saveTaskInstance(taskInstance); |
||||
return true; |
||||
} |
||||
|
||||
@Override |
||||
public String getType() { |
||||
return TaskType.CONDITIONS.getDesc(); |
||||
} |
||||
|
||||
private void initTaskParameters() { |
||||
taskInstance.setLogPath(LogUtils.getTaskLogPath(processInstance.getProcessDefinitionCode(), |
||||
processInstance.getProcessDefinitionVersion(), |
||||
taskInstance.getProcessInstanceId(), |
||||
taskInstance.getId())); |
||||
this.taskInstance.setHost(NetUtils.getAddr(masterConfig.getListenPort())); |
||||
taskInstance.setState(ExecutionStatus.RUNNING_EXECUTION); |
||||
taskInstance.setStartTime(new Date()); |
||||
this.processService.saveTaskInstance(taskInstance); |
||||
this.dependentParameters = taskInstance.getDependency(); |
||||
} |
||||
|
||||
private void setConditionResult() { |
||||
|
||||
List<TaskInstance> taskInstances = processService.findValidTaskListByProcessId(taskInstance.getProcessInstanceId()); |
||||
for (TaskInstance task : taskInstances) { |
||||
completeTaskList.putIfAbsent(task.getName(), task.getState()); |
||||
} |
||||
|
||||
List<DependResult> modelResultList = new ArrayList<>(); |
||||
for (DependentTaskModel dependentTaskModel : dependentParameters.getDependTaskList()) { |
||||
List<DependResult> itemDependResult = new ArrayList<>(); |
||||
for (DependentItem item : dependentTaskModel.getDependItemList()) { |
||||
itemDependResult.add(getDependResultForItem(item)); |
||||
} |
||||
DependResult modelResult = DependentUtils.getDependResultForRelation(dependentTaskModel.getRelation(), itemDependResult); |
||||
modelResultList.add(modelResult); |
||||
} |
||||
conditionResult = DependentUtils.getDependResultForRelation(dependentParameters.getRelation(), modelResultList); |
||||
logger.info("the conditions task depend result : {}", conditionResult); |
||||
} |
||||
|
||||
/** |
||||
* depend result for depend item |
||||
*/ |
||||
private DependResult getDependResultForItem(DependentItem item) { |
||||
|
||||
DependResult dependResult = DependResult.SUCCESS; |
||||
if (!completeTaskList.containsKey(item.getDepTasks())) { |
||||
logger.info("depend item: {} have not completed yet.", item.getDepTasks()); |
||||
dependResult = DependResult.FAILED; |
||||
return dependResult; |
||||
} |
||||
ExecutionStatus executionStatus = completeTaskList.get(item.getDepTasks()); |
||||
if (executionStatus != item.getStatus()) { |
||||
logger.info("depend item : {} expect status: {}, actual status: {}", item.getDepTasks(), item.getStatus(), executionStatus); |
||||
dependResult = DependResult.FAILED; |
||||
} |
||||
logger.info("dependent item complete {} {},{}", |
||||
Constants.DEPENDENT_SPLIT, item.getDepTasks(), dependResult); |
||||
return dependResult; |
||||
} |
||||
|
||||
/** |
||||
* |
||||
*/ |
||||
private void endTask() { |
||||
ExecutionStatus status = (conditionResult == DependResult.SUCCESS) ? ExecutionStatus.SUCCESS : ExecutionStatus.FAILURE; |
||||
taskInstance.setState(status); |
||||
taskInstance.setEndTime(new Date()); |
||||
processService.updateTaskInstance(taskInstance); |
||||
} |
||||
} |
||||
@ -0,0 +1,33 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.server.master.runner.task; |
||||
|
||||
import org.apache.dolphinscheduler.common.enums.TaskType; |
||||
|
||||
public class DependentTaskProcessFactory implements ITaskProcessFactory { |
||||
|
||||
@Override |
||||
public String type() { |
||||
return TaskType.DEPENDENT.getDesc(); |
||||
} |
||||
|
||||
@Override |
||||
public ITaskProcessor create() { |
||||
return new DependentTaskProcessor(); |
||||
} |
||||
} |
||||
@ -0,0 +1,220 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.server.master.runner.task; |
||||
|
||||
import static org.apache.dolphinscheduler.common.Constants.DEPENDENT_SPLIT; |
||||
|
||||
import org.apache.dolphinscheduler.common.enums.DependResult; |
||||
import org.apache.dolphinscheduler.common.enums.ExecutionStatus; |
||||
import org.apache.dolphinscheduler.common.enums.TaskTimeoutStrategy; |
||||
import org.apache.dolphinscheduler.common.enums.TaskType; |
||||
import org.apache.dolphinscheduler.common.model.DependentTaskModel; |
||||
import org.apache.dolphinscheduler.common.task.dependent.DependentParameters; |
||||
import org.apache.dolphinscheduler.common.utils.DependentUtils; |
||||
import org.apache.dolphinscheduler.common.utils.NetUtils; |
||||
import org.apache.dolphinscheduler.dao.entity.ProcessInstance; |
||||
import org.apache.dolphinscheduler.dao.entity.TaskDefinition; |
||||
import org.apache.dolphinscheduler.dao.entity.TaskInstance; |
||||
import org.apache.dolphinscheduler.server.master.config.MasterConfig; |
||||
import org.apache.dolphinscheduler.server.utils.DependentExecute; |
||||
import org.apache.dolphinscheduler.server.utils.LogUtils; |
||||
import org.apache.dolphinscheduler.service.bean.SpringApplicationContext; |
||||
import org.apache.dolphinscheduler.service.process.ProcessService; |
||||
|
||||
import java.util.ArrayList; |
||||
import java.util.Date; |
||||
import java.util.HashMap; |
||||
import java.util.List; |
||||
import java.util.Map; |
||||
|
||||
import com.fasterxml.jackson.annotation.JsonFormat; |
||||
|
||||
/** |
||||
* dependent task processor |
||||
*/ |
||||
public class DependentTaskProcessor extends BaseTaskProcessor { |
||||
|
||||
private DependentParameters dependentParameters; |
||||
|
||||
/** |
||||
* dependent task list |
||||
*/ |
||||
private List<DependentExecute> dependentTaskList = new ArrayList<>(); |
||||
|
||||
/** |
||||
* depend item result map |
||||
* save the result to log file |
||||
*/ |
||||
private Map<String, DependResult> dependResultMap = new HashMap<>(); |
||||
|
||||
/** |
||||
* dependent date |
||||
*/ |
||||
@JsonFormat(pattern = "yyyy-MM-dd HH:mm:ss", timezone = "GMT+8") |
||||
private Date dependentDate; |
||||
|
||||
DependResult result; |
||||
|
||||
ProcessInstance processInstance; |
||||
TaskDefinition taskDefinition; |
||||
|
||||
protected ProcessService processService = SpringApplicationContext.getBean(ProcessService.class); |
||||
MasterConfig masterConfig = SpringApplicationContext.getBean(MasterConfig.class); |
||||
|
||||
boolean allDependentItemFinished; |
||||
|
||||
@Override |
||||
public boolean submit(TaskInstance task, ProcessInstance processInstance, int masterTaskCommitRetryTimes, int masterTaskCommitInterval) { |
||||
this.processInstance = processInstance; |
||||
this.taskInstance = task; |
||||
this.taskInstance = processService.submitTask(task, masterTaskCommitRetryTimes, masterTaskCommitInterval); |
||||
|
||||
if (this.taskInstance == null) { |
||||
return false; |
||||
} |
||||
taskDefinition = processService.findTaskDefinition( |
||||
taskInstance.getTaskCode(), taskInstance.getTaskDefinitionVersion() |
||||
); |
||||
taskInstance.setLogPath(LogUtils.getTaskLogPath(processInstance.getProcessDefinitionCode(), |
||||
processInstance.getProcessDefinitionVersion(), |
||||
taskInstance.getProcessInstanceId(), |
||||
taskInstance.getId())); |
||||
taskInstance.setHost(NetUtils.getAddr(masterConfig.getListenPort())); |
||||
taskInstance.setState(ExecutionStatus.RUNNING_EXECUTION); |
||||
taskInstance.setStartTime(new Date()); |
||||
processService.updateTaskInstance(taskInstance); |
||||
initDependParameters(); |
||||
return true; |
||||
} |
||||
|
||||
@Override |
||||
public ExecutionStatus taskState() { |
||||
return this.taskInstance.getState(); |
||||
} |
||||
|
||||
@Override |
||||
public void run() { |
||||
if (!allDependentItemFinished) { |
||||
allDependentItemFinished = allDependentTaskFinish(); |
||||
} |
||||
if (allDependentItemFinished) { |
||||
getTaskDependResult(); |
||||
endTask(); |
||||
} |
||||
} |
||||
|
||||
@Override |
||||
protected boolean taskTimeout() { |
||||
TaskTimeoutStrategy taskTimeoutStrategy = |
||||
taskDefinition.getTimeoutNotifyStrategy(); |
||||
if (TaskTimeoutStrategy.FAILED != taskTimeoutStrategy |
||||
&& TaskTimeoutStrategy.WARNFAILED != taskTimeoutStrategy) { |
||||
return true; |
||||
} |
||||
logger.info("dependent task {} timeout, strategy {} ", |
||||
taskInstance.getId(), taskTimeoutStrategy.getDescp()); |
||||
result = DependResult.FAILED; |
||||
endTask(); |
||||
return true; |
||||
} |
||||
|
||||
/** |
||||
* init dependent parameters |
||||
*/ |
||||
private void initDependParameters() { |
||||
this.dependentParameters = taskInstance.getDependency(); |
||||
for (DependentTaskModel taskModel : dependentParameters.getDependTaskList()) { |
||||
this.dependentTaskList.add(new DependentExecute(taskModel.getDependItemList(), taskModel.getRelation())); |
||||
} |
||||
if (processInstance.getScheduleTime() != null) { |
||||
this.dependentDate = this.processInstance.getScheduleTime(); |
||||
} else { |
||||
this.dependentDate = new Date(); |
||||
} |
||||
} |
||||
|
||||
@Override |
||||
protected boolean pauseTask() { |
||||
this.taskInstance.setState(ExecutionStatus.PAUSE); |
||||
this.taskInstance.setEndTime(new Date()); |
||||
processService.saveTaskInstance(taskInstance); |
||||
return true; |
||||
} |
||||
|
||||
@Override |
||||
protected boolean killTask() { |
||||
this.taskInstance.setState(ExecutionStatus.KILL); |
||||
this.taskInstance.setEndTime(new Date()); |
||||
processService.saveTaskInstance(taskInstance); |
||||
return true; |
||||
} |
||||
|
||||
/** |
||||
* judge all dependent tasks finish |
||||
* |
||||
* @return whether all dependent tasks finish |
||||
*/ |
||||
private boolean allDependentTaskFinish() { |
||||
boolean finish = true; |
||||
for (DependentExecute dependentExecute : dependentTaskList) { |
||||
for (Map.Entry<String, DependResult> entry : dependentExecute.getDependResultMap().entrySet()) { |
||||
if (!dependResultMap.containsKey(entry.getKey())) { |
||||
dependResultMap.put(entry.getKey(), entry.getValue()); |
||||
//save depend result to log
|
||||
logger.info("dependent item complete {} {},{}", DEPENDENT_SPLIT, entry.getKey(), entry.getValue()); |
||||
} |
||||
} |
||||
if (!dependentExecute.finish(dependentDate)) { |
||||
finish = false; |
||||
} |
||||
} |
||||
return finish; |
||||
} |
||||
|
||||
/** |
||||
* get dependent result |
||||
* |
||||
* @return DependResult |
||||
*/ |
||||
private DependResult getTaskDependResult() { |
||||
List<DependResult> dependResultList = new ArrayList<>(); |
||||
for (DependentExecute dependentExecute : dependentTaskList) { |
||||
DependResult dependResult = dependentExecute.getModelDependResult(dependentDate); |
||||
dependResultList.add(dependResult); |
||||
} |
||||
result = DependentUtils.getDependResultForRelation(this.dependentParameters.getRelation(), dependResultList); |
||||
logger.info("dependent task completed, dependent result:{}", result); |
||||
return result; |
||||
} |
||||
|
||||
/** |
||||
* |
||||
*/ |
||||
private void endTask() { |
||||
ExecutionStatus status; |
||||
status = (result == DependResult.SUCCESS) ? ExecutionStatus.SUCCESS : ExecutionStatus.FAILURE; |
||||
taskInstance.setState(status); |
||||
taskInstance.setEndTime(new Date()); |
||||
processService.saveTaskInstance(taskInstance); |
||||
} |
||||
|
||||
@Override |
||||
public String getType() { |
||||
return TaskType.DEPENDENT.getDesc(); |
||||
} |
||||
} |
||||
@ -0,0 +1,25 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.server.master.runner.task; |
||||
|
||||
public interface ITaskProcessFactory { |
||||
|
||||
String type(); |
||||
|
||||
ITaskProcessor create(); |
||||
} |
||||
@ -0,0 +1,39 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.server.master.runner.task; |
||||
|
||||
import org.apache.dolphinscheduler.common.enums.ExecutionStatus; |
||||
import org.apache.dolphinscheduler.dao.entity.ProcessInstance; |
||||
import org.apache.dolphinscheduler.dao.entity.TaskInstance; |
||||
|
||||
/** |
||||
* interface of task processor in master |
||||
*/ |
||||
public interface ITaskProcessor { |
||||
|
||||
void run(); |
||||
|
||||
boolean action(TaskAction taskAction); |
||||
|
||||
String getType(); |
||||
|
||||
boolean submit(TaskInstance taskInstance, ProcessInstance processInstance, int masterTaskCommitRetryTimes, int masterTaskCommitInterval); |
||||
|
||||
ExecutionStatus taskState(); |
||||
|
||||
} |
||||
@ -0,0 +1,32 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.server.master.runner.task; |
||||
|
||||
import org.apache.dolphinscheduler.common.enums.TaskType; |
||||
|
||||
public class SubTaskProcessFactory implements ITaskProcessFactory { |
||||
@Override |
||||
public String type() { |
||||
return TaskType.SUB_PROCESS.getDesc(); |
||||
} |
||||
|
||||
@Override |
||||
public ITaskProcessor create() { |
||||
return new SubTaskProcessor(); |
||||
} |
||||
} |
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in new issue