From 3b495d2fb0871db0fb2945ed79f60bb9059446b5 Mon Sep 17 00:00:00 2001 From: JinyLeeChina <42576980+JinyLeeChina@users.noreply.github.com> Date: Mon, 23 Aug 2021 16:23:40 +0800 Subject: [PATCH] [Feature][JsonSplit-api] merge code from dev to json2 (#6023) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * [BUG-#5678][Registry]fix registry init node miss (#5686) * [Improvement][UI] Update the update time after the user information is successfully modified (#5684) * improve edit the userinfo success, but the updatetime is not the latest. * Improved shell task execution result log information, adding process.waitFor() and process.exitValue() information to the original log (#5691) Co-authored-by: shenglm * [Feature-#5565][Master Worker-Server] Global Param passed by sense dependencies (#5603) * add globalParams new plan with varPool * add unit test * add python task varPoolParams Co-authored-by: wangxj * Issue robot translation judgment changed to Chinese (#5694) Co-authored-by: chenxingchun <438044805@qq.com> * the update function should use post instead of get (#5703) * enhance form verify (#5696) * checkState only supports %s not {} (#5711) * [Fix-5701]When deleting a user, the accessToken associated with the user should also be deleted (#5697) * update * fix the codestyle error * fix the compile error * support rollback * [Fix-5699][UI] Fix update user error in user information (#5700) * [Improvement] the automatically generated spi service name in alert-plugin is wrong (#5676) * bug fix the auto generated spi service can't be recongized * include a new method * [Improvement-5622][project management] Modify the title (#5723) * [Fix-5714] When updating the existing alarm instance, the creation time should't be updated (#5715) * add a new init method. * [Fix#5758] There are some problems in the api documentation that need to be improved (#5759) * add the necessary parameters * openapi improve * fix code style error * [FIX-#5721][master-server] Global params parameter missing (#5757) Co-authored-by: wangxj * [Fix-5738][UI] The cancel button in the pop-up dialog of `batch copy` and `batch move` doesn't work. (#5739) * Update relatedItems.vue * Update relatedItems.vue * [Improvement#5741][Worker] Improve task process status log (#5776) * [Improvement-5773][server] need to support two parameters related to task (#5774) * add some new parameter for task * restore official properties * improve imports * modify a variable's name Co-authored-by: jiang hua * [FIX-5786][Improvement][Server] When the Worker turns down, the MasterServer cannot handle the Remove event correctly and throws NPE * [Improvement][Worker] Task log may be lost #5775 (#5783) * [Imporvement #5725][CheckStyle] upgrade checkstyle file (#5789) * [Imporvement #5725][CheckStyle] upgrade checkstyle file Upgrade checkstyle.xml to support checkstyle version 8.24+ * change ci checkstyle version * [Fix-5795][Improvement][Server] The starttime field in the HttpTask log is not displayed as expected. (#5796) * improve timestamp format make the startime in the log of httptask to be easier to read. * fix bad code smell and update the note. * [Imporvement #5621][job instance] start-time and end-time (#5621) (#5797) ·the list of workflow instances is sorted by start time and end time ·This closes #5621 * fix (#5803) Co-authored-by: shuangbofu * fix: Remove duplicate "registryClient.close" method calls (#5805) Co-authored-by: wen-hemin * [Improvement][SPI] support load single plugin (#5794) change load operation of 'registry.plugin.dir' * [Improvement][Api Module] refactor registry client, remove spring annotation (#5814) * fix: refactor registry client, remove spring annotation * fix UT * fix UT * fix checkstyle * fix UT * fix UT * fix UT * fix: Rename RegistryCenterUtils method name Co-authored-by: wen-hemin * [Fix-5699][UI] Fix update user error in user information introduced by #5700 (#5735) * [Fix-5726] When we used the UI page, we found some problems such as parameter validation, parameter update shows success but actually work (#5727) * enhance the validation in UI * enchance form verifaction * simplify disable condition * fix: Remove unused class (#5833) Co-authored-by: wen-hemin * [fix-5737] [Bug][Datasource] datsource other param check error (#5835) Co-authored-by: wanggang * [Fix-5719][K8s] Fix Ingress tls: got map expected array On TLS enabled On Kubernetes [Fix-5719][K8s] Fix Ingress tls: got map expected array On TLS enabled On Kubernetes * [Fix-5825][BUG][WEB] the resource tree in the process definition of latest dev branch can't display correctly (#5826) * resoures-shows-error * fix codestyle error * add license header for new js * fix codesmell * [Improvement-5852][server] Support two parameters related to task for the rest of type of tasks. (#5867) * provide two system parameters to support the rest of type of tasks * provide two system parameters to support the rest of type of tasks * improve test conversion * [Improvement][Fix-5769][UI]When we try to delete the existing dag, the console in web browser would shows exception (#5770) * fix bug * cache the this variable * Avoid self name * fix code style compile error * [Fix-5781][UT] Fix test coverage in sonar (#5817) * build(UT): make jacoco running in offline-instrumentation issue: #5781 * build(UT): remove the jacoco agent dependency in microbench issue: #5781 * [Fix-5808][Server] When we try to transfer data using datax between different types of data sources, the worker will exit with ClassCastException (#5809) * bug fix * fix bug * simplify the code format * add a new parameter to make it easier to understand. * [Fix-5830][Improvement][UI] Improve the selection style in dag edit dialog (#5829) * improve the selection style * update another file * remove unnecessary css part. * [Fix-5904][upgrade]fix dev branch upgrade mysql sql script error (#5821) * fix dev branch upgrade mysql sql script error. * Update naming convention. * [Improvement][Api Module] refactor DataSourceParam and DependentParam, remove spring annotation (#5832) * fix: refactor api utils class, remove spring annotation. * fix: Optimization comments Co-authored-by: wen-hemin * correct the wrong annotion from zk queue implemented to java priority blocking queue (#5906) Co-authored-by: ywang46 * Add a Gitter chat badge to README.md (#5883) * Add Gitter badge * Update README.md Co-authored-by: David * ci: improve maven connection in CI builds (#5924) issue: #5921 * [Improvement][Master]fix typo (#5934) ·fix typo in MasterBaseTaskExecThread * [Fix-5886][server] Enhanced scheduler delete check (#5936) * Add:Name verification remove the first and last spaces. * Update: wrong word: 'WAITTING' ->'WAITING' * Add: Strengthen verification Co-authored-by: Squid <2824638304@qq.com> * [Improvement-5880][api] Optimized data structure of pagination query API results (#5895) * [5880][refactor]Optimized data structure of pagination query API results - refactor PageInfo and delete returnDataListPaging in API - modify the related Controller and Service and the corresponding Test * Merge branch 'dev' of github.com:apache/dolphinscheduler into dev  Conflicts:  dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ProcessDefinitionServiceImpl.java Co-authored-by: 蔡泽华 * [IMPROVEMENT]fix mysql comment error (#5959) * [Improvement][Api]fix typo (#5960) * [Imporvement #5621][job instance] start-time and end-time (#5621) ·the list of workflow instances is sorted by start time and end time ·This closes #5621 * [FIX-5975]queryLastRunningProcess sql in ProcessInstanceMapper.xml (#5980) * [NEW FEATURE][FIX-4385] compensation task add the ability to configure parallelism (#5912) * update * web improved * improve the ui * add the ability to configure the parallelism * update variables * enhance the ut and add necessary note * fix code style * fix code style issue * ensure the complation task in parallel mode can run the right numbers of tasks. * [Improvement][dao]When I search for the keyword description, the web UI shows empty (#5952) * [Bug][WorkerServer] SqlTask NullPointerException #5549 * [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428 * [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428 * [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428 * [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428 * [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428 * [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428 * [Improvement][dao]When I search for the keyword description, the web UI shows empty #5428 * fix the readme typing issue (#5998) * Fix unchecked type conversions * Use indentation level reported by checkstyle * Reorganize CI workflows to fasten the wasted time and resources (#6011) * Add standalone server module to make it easier to develop (#6022) * fix ut Co-authored-by: Kirs Co-authored-by: kyoty Co-authored-by: ji04xiaogang Co-authored-by: shenglm Co-authored-by: wangxj3 <857234426@qq.com> Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com> Co-authored-by: chenxingchun <438044805@qq.com> Co-authored-by: Shiwen Cheng Co-authored-by: Jianchao Wang Co-authored-by: Tanvi Moharir <74228962+tanvimoharir@users.noreply.github.com> Co-authored-by: Hua Jiang Co-authored-by: jiang hua Co-authored-by: Wenjun Ruan <861923274@qq.com> Co-authored-by: Tandoy <56899730+Tandoy@users.noreply.github.com> Co-authored-by: 傅双波 <786183073@qq.com> Co-authored-by: shuangbofu Co-authored-by: wen-hemin <39549317+wen-hemin@users.noreply.github.com> Co-authored-by: wen-hemin Co-authored-by: geosmart Co-authored-by: wanggang Co-authored-by: AzureCN Co-authored-by: 深刻 Co-authored-by: zhuangchong <37063904+zhuangchong@users.noreply.github.com> Co-authored-by: Yao WANG Co-authored-by: ywang46 Co-authored-by: The Gitter Badger Co-authored-by: David Co-authored-by: Squidyu <1297554122@qq.com> Co-authored-by: Squid <2824638304@qq.com> Co-authored-by: soreak <60459867+soreak@users.noreply.github.com> Co-authored-by: 蔡泽华 Co-authored-by: yimaixinchen Co-authored-by: atai-555 <74188560+atai-555@users.noreply.github.com> Co-authored-by: didiaode18 <563646039@qq.com> Co-authored-by: Roy Co-authored-by: lyxell Co-authored-by: Wenjun Ruan Co-authored-by: kezhenxu94 Co-authored-by: JinyLeeChina <297062848@qq.com> --- .github/actions/reviewdog-setup | 1 + .github/actions/sanity-check/action.yml | 53 + .../workflows/{ci_backend.yml => backend.yml} | 38 +- .github/workflows/{ci_e2e.yml => e2e.yml} | 22 +- .../{ci_frontend.yml => frontend.yml} | 31 +- .../workflows/{ci_ut.yml => unit-test.yml} | 100 +- .gitmodules | 6 + .licenserc.yaml | 5 + README.md | 5 +- docker/build/hooks/build | 4 +- docker/build/hooks/build.bat | 4 +- .../dolphinscheduler-alert-email/pom.xml | 13 +- .../api/controller/ExecutorController.java | 7 +- .../api/service/ExecutorService.java | 3 +- .../api/service/impl/ExecutorServiceImpl.java | 37 +- .../api/service/impl/TenantServiceImpl.java | 4 +- .../api/service/impl/UsersServiceImpl.java | 4 +- .../resources/i18n/messages_en_US.properties | 1 + .../resources/i18n/messages_zh_CN.properties | 1 + .../controller/ExecutorControllerTest.java | 49 +- .../api/service/ExecutorServiceTest.java | 14 +- .../dolphinscheduler/common/enums/DbType.java | 66 +- .../dolphinscheduler/common/graph/DAG.java | 12 +- .../common/utils/PropertyUtils.java | 18 +- .../datasource/SpringConnectionFactory.java | 1 + .../dao/mapper/ProcessDefinitionMapper.xml | 4 +- .../dao/mapper/ProcessInstanceMapper.xml | 4 +- .../dao/mapper/ProjectMapper.xml | 4 +- .../dao/mapper/WorkFlowLineageMapper.xml | 3 +- .../ResourceProcessDefinitionUtilsTest.java | 2 +- dolphinscheduler-dist/pom.xml | 7 +- dolphinscheduler-dist/release-docs/LICENSE | 3 +- .../executor/NettyExecutorManager.java | 2 +- .../runner/MasterBaseTaskExecThread.java | 2 +- .../server/worker/task/sql/SqlTask.java | 14 +- .../server/master/MasterExecThreadTest.java | 6 +- .../server/worker/task/http/HttpTaskTest.java | 4 +- .../service/quartz/ProcessScheduleJob.java | 4 +- dolphinscheduler-standalone-server/pom.xml | 52 + .../server/StandaloneServer.java | 82 ++ .../src/main/resources/registry.properties | 22 + .../definition/pages/list/_source/start.vue | 68 +- .../src/js/module/i18n/locale/en_US.js | 4 + .../src/js/module/i18n/locale/zh_CN.js | 3 + install.sh | 103 -- pom.xml | 18 +- script/dolphinscheduler-daemon.sh | 6 +- .../mysql/dolphinscheduler_ddl.sql | 4 +- sql/dolphinscheduler_h2.sql | 943 ++++++++++++++++++ style/checkstyle-suppressions.xml | 24 - style/checkstyle.xml | 8 +- tools/dependencies/known-dependencies.txt | 1 + 52 files changed, 1500 insertions(+), 396 deletions(-) create mode 160000 .github/actions/reviewdog-setup create mode 100644 .github/actions/sanity-check/action.yml rename .github/workflows/{ci_backend.yml => backend.yml} (63%) rename .github/workflows/{ci_e2e.yml => e2e.yml} (89%) rename .github/workflows/{ci_frontend.yml => frontend.yml} (67%) rename .github/workflows/{ci_ut.yml => unit-test.yml} (52%) create mode 100644 dolphinscheduler-standalone-server/pom.xml create mode 100644 dolphinscheduler-standalone-server/src/main/java/org/apache/dolphinscheduler/server/StandaloneServer.java create mode 100644 dolphinscheduler-standalone-server/src/main/resources/registry.properties delete mode 100755 install.sh create mode 100644 sql/dolphinscheduler_h2.sql delete mode 100644 style/checkstyle-suppressions.xml diff --git a/.github/actions/reviewdog-setup b/.github/actions/reviewdog-setup new file mode 160000 index 0000000000..2fc905b187 --- /dev/null +++ b/.github/actions/reviewdog-setup @@ -0,0 +1 @@ +Subproject commit 2fc905b1875f2e6b91c4201a4dc6eaa21b86547e diff --git a/.github/actions/sanity-check/action.yml b/.github/actions/sanity-check/action.yml new file mode 100644 index 0000000000..a1d03a33c3 --- /dev/null +++ b/.github/actions/sanity-check/action.yml @@ -0,0 +1,53 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +name: "Sanity Check" + +description: | + Action to perform some very basic lightweight checks, like code styles, license headers, etc., + and fail fast to avoid wasting resources running heavyweight checks, like unit tests, e2e tests. + +inputs: + token: + description: 'The GitHub API token' + required: false + +runs: + using: "composite" + steps: + - name: Check License Header + uses: apache/skywalking-eyes@a63f4afcc287dfb3727ecc45a4afc55a5e69c15f + + - uses: ./.github/actions/reviewdog-setup + with: + reviewdog_version: v0.10.2 + + - shell: bash + run: ./mvnw -B -q checkstyle:checkstyle-aggregate + + - shell: bash + env: + REVIEWDOG_GITHUB_API_TOKEN: ${{ inputs.token }} + run: | + if [[ -n "${{ inputs.token }}" ]]; then + reviewdog -f=checkstyle \ + -reporter="github-pr-check" \ + -filter-mode="added" \ + -fail-on-error="true" < target/checkstyle-result.xml + fi diff --git a/.github/workflows/ci_backend.yml b/.github/workflows/backend.yml similarity index 63% rename from .github/workflows/ci_backend.yml rename to .github/workflows/backend.yml index e19336eab3..55475b2fe4 100644 --- a/.github/workflows/ci_backend.yml +++ b/.github/workflows/backend.yml @@ -19,8 +19,10 @@ name: Backend on: push: + branches: + - dev paths: - - '.github/workflows/ci_backend.yml' + - '.github/workflows/backend.yml' - 'package.xml' - 'pom.xml' - 'dolphinscheduler-alert/**' @@ -31,7 +33,7 @@ on: - 'dolphinscheduler-server/**' pull_request: paths: - - '.github/workflows/ci_backend.yml' + - '.github/workflows/backend.yml' - 'package.xml' - 'pom.xml' - 'dolphinscheduler-alert/**' @@ -41,20 +43,34 @@ on: - 'dolphinscheduler-rpc/**' - 'dolphinscheduler-server/**' +concurrency: + group: backend-${{ github.event.pull_request.number || github.ref }} + cancel-in-progress: true + jobs: - Compile-check: + build: + name: Build runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 with: - submodule: true - - name: Check License Header - uses: apache/skywalking-eyes@ec88b7d850018c8983f87729ea88549e100c5c82 - - name: Set up JDK 1.8 - uses: actions/setup-java@v1 + submodules: true + - name: Sanity Check + uses: ./.github/actions/sanity-check + with: + token: ${{ secrets.GITHUB_TOKEN }} # We only need to pass this token in one workflow + - uses: actions/cache@v2 with: - java-version: 1.8 - - name: Compile - run: mvn -B clean compile install -Prelease -Dmaven.test.skip=true + path: ~/.m2/repository + key: ${{ runner.os }}-maven + - name: Build and Package + run: | + ./mvnw -B clean install \ + -Prelease \ + -Dmaven.test.skip=true \ + -Dcheckstyle.skip=true \ + -Dhttp.keepAlive=false \ + -Dmaven.wagon.http.pool=false \ + -Dmaven.wagon.httpconnectionManager.ttlSeconds=120 - name: Check dependency license run: tools/dependencies/check-LICENSE.sh diff --git a/.github/workflows/ci_e2e.yml b/.github/workflows/e2e.yml similarity index 89% rename from .github/workflows/ci_e2e.yml rename to .github/workflows/e2e.yml index 009b3fb151..2fbbffa8bd 100644 --- a/.github/workflows/ci_e2e.yml +++ b/.github/workflows/e2e.yml @@ -20,26 +20,26 @@ env: DOCKER_DIR: ./docker LOG_DIR: /tmp/dolphinscheduler -name: e2e Test +name: Test -jobs: +concurrency: + group: e2e-${{ github.event.pull_request.number || github.ref }} + cancel-in-progress: true - build: - name: Test +jobs: + test: + name: E2E runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 with: - submodule: true - - name: Check License Header - uses: apache/skywalking-eyes@ec88b7d850018c8983f87729ea88549e100c5c82 + submodules: true + - name: Sanity Check + uses: ./.github/actions/sanity-check - uses: actions/cache@v1 with: path: ~/.m2/repository - key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }} - restore-keys: | - ${{ runner.os }}-maven- + key: ${{ runner.os }}-maven - name: Build Image run: | sh ./docker/build/hooks/build diff --git a/.github/workflows/ci_frontend.yml b/.github/workflows/frontend.yml similarity index 67% rename from .github/workflows/ci_frontend.yml rename to .github/workflows/frontend.yml index afa0c8d672..4ab1e0d6c5 100644 --- a/.github/workflows/ci_frontend.yml +++ b/.github/workflows/frontend.yml @@ -19,31 +19,44 @@ name: Frontend on: push: + branches: + - dev paths: - - '.github/workflows/ci_frontend.yml' + - '.github/workflows/frontend.yml' - 'dolphinscheduler-ui/**' pull_request: paths: - - '.github/workflows/ci_frontend.yml' + - '.github/workflows/frontend.yml' - 'dolphinscheduler-ui/**' +defaults: + run: + working-directory: dolphinscheduler-ui + +concurrency: + group: frontend-${{ github.event.pull_request.number || github.ref }} + cancel-in-progress: true + jobs: - Compile-check: + build: + name: Build runs-on: ${{ matrix.os }} strategy: matrix: - os: [ubuntu-latest, macos-latest] + os: [ ubuntu-latest, macos-latest ] steps: - uses: actions/checkout@v2 with: - submodule: true + submodules: true + - if: matrix.os == 'ubuntu-latest' + name: Sanity Check + uses: ./.github/actions/sanity-check - name: Set up Node.js - uses: actions/setup-node@v1 + uses: actions/setup-node@v2 with: - version: 8 - - name: Compile + node-version: 8 + - name: Compile and Build run: | - cd dolphinscheduler-ui npm install node-sass --unsafe-perm npm install npm run lint diff --git a/.github/workflows/ci_ut.yml b/.github/workflows/unit-test.yml similarity index 52% rename from .github/workflows/ci_ut.yml rename to .github/workflows/unit-test.yml index 0246aaf80e..3087806894 100644 --- a/.github/workflows/ci_ut.yml +++ b/.github/workflows/unit-test.yml @@ -15,103 +15,91 @@ # limitations under the License. # +name: Test + on: pull_request: + paths-ignore: + - '**/*.md' + - 'dolphinscheduler-ui' push: + paths-ignore: + - '**/*.md' + - 'dolphinscheduler-ui' branches: - dev + env: LOG_DIR: /tmp/dolphinscheduler -name: Unit Test +concurrency: + group: unit-test-${{ github.event.pull_request.number || github.ref }} + cancel-in-progress: true jobs: - - build: - name: Build + unit-test: + name: Unit Test runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 with: - submodule: true - - name: Check License Header - uses: apache/skywalking-eyes@ec88b7d850018c8983f87729ea88549e100c5c82 - env: - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # Only enable review / suggestion here - - uses: actions/cache@v1 + submodules: true + - name: Sanity Check + uses: ./.github/actions/sanity-check + - name: Set up JDK 1.8 + uses: actions/setup-java@v2 + with: + java-version: 8 + distribution: 'adopt' + - uses: actions/cache@v2 with: path: ~/.m2/repository - key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }} - restore-keys: | - ${{ runner.os }}-maven- + key: ${{ runner.os }}-maven - name: Bootstrap database run: | sed -i "/image: bitnami\/postgresql/a\ ports:\n - 5432:5432" $(pwd)/docker/docker-swarm/docker-compose.yml sed -i "/image: bitnami\/zookeeper/a\ ports:\n - 2181:2181" $(pwd)/docker/docker-swarm/docker-compose.yml docker-compose -f $(pwd)/docker/docker-swarm/docker-compose.yml up -d dolphinscheduler-zookeeper dolphinscheduler-postgresql until docker logs docker-swarm_dolphinscheduler-postgresql_1 2>&1 | grep 'listening on IPv4 address'; do echo "waiting for postgresql ready ..."; sleep 1; done - docker run --rm --network docker-swarm_dolphinscheduler -v $(pwd)/sql/dolphinscheduler_postgre.sql:/docker-entrypoint-initdb.d/dolphinscheduler_postgre.sql bitnami/postgresql:latest bash -c "PGPASSWORD=root psql -h docker-swarm_dolphinscheduler-postgresql_1 -U root -d dolphinscheduler -v ON_ERROR_STOP=1 -f /docker-entrypoint-initdb.d/dolphinscheduler_postgre.sql" - - name: Set up JDK 1.8 - uses: actions/setup-java@v1 - with: - java-version: 1.8 - - name: Git fetch unshallow - run: | - git fetch --unshallow - git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*" - git fetch origin - - name: Compile - run: | - export MAVEN_OPTS='-Dmaven.repo.local=.m2/repository -XX:+TieredCompilation -XX:TieredStopAtLevel=1 -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -Xmx5g' - mvn clean verify -B -Dmaven.test.skip=false + docker run --rm --network docker-swarm_dolphinscheduler -v $(pwd)/sql/dolphinscheduler_postgre.sql:/docker-entrypoint-initdb.d/dolphinscheduler_postgre.sql bitnami/postgresql:11.11.0 bash -c "PGPASSWORD=root psql -h docker-swarm_dolphinscheduler-postgresql_1 -U root -d dolphinscheduler -v ON_ERROR_STOP=1 -f /docker-entrypoint-initdb.d/dolphinscheduler_postgre.sql" + + - name: Run Unit tests + run: ./mvnw clean verify -B -Dmaven.test.skip=false - name: Upload coverage report to codecov - run: | - CODECOV_TOKEN="09c2663f-b091-4258-8a47-c981827eb29a" bash <(curl -s https://codecov.io/bash) + run: CODECOV_TOKEN="09c2663f-b091-4258-8a47-c981827eb29a" bash <(curl -s https://codecov.io/bash) + # Set up JDK 11 for SonarCloud. - - name: Set up JDK 1.11 - uses: actions/setup-java@v1 + - name: Set up JDK 11 + uses: actions/setup-java@v2 with: - java-version: 1.11 + java-version: 11 + distribution: 'adopt' - name: Run SonarCloud Analysis run: > - mvn --batch-mode verify sonar:sonar + ./mvnw --batch-mode verify sonar:sonar -Dsonar.coverage.jacoco.xmlReportPaths=target/site/jacoco/jacoco.xml -Dmaven.test.skip=true + -Dcheckstyle.skip=true -Dsonar.host.url=https://sonarcloud.io -Dsonar.organization=apache -Dsonar.core.codeCoveragePlugin=jacoco -Dsonar.projectKey=apache-dolphinscheduler -Dsonar.login=e4058004bc6be89decf558ac819aa1ecbee57682 -Dsonar.exclusions=dolphinscheduler-ui/src/**/i18n/locale/*.js,dolphinscheduler-microbench/src/**/* + -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.httpconnectionManager.ttlSeconds=120 env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }} + - name: Collect logs + continue-on-error: true run: | mkdir -p ${LOG_DIR} docker-compose -f $(pwd)/docker/docker-swarm/docker-compose.yml logs dolphinscheduler-postgresql > ${LOG_DIR}/db.txt - continue-on-error: true - Checkstyle: - name: Check code style - runs-on: ubuntu-latest - steps: - - name: Checkout - uses: actions/checkout@v2 + - name: Upload logs + uses: actions/upload-artifact@v2 + continue-on-error: true with: - submodule: true - - name: check code style - env: - WORKDIR: ./ - REVIEWDOG_GITHUB_API_TOKEN: ${{ secrets.GITHUB_TOKEN }} - CHECKSTYLE_CONFIG: style/checkstyle.xml - REVIEWDOG_VERSION: v0.10.2 - run: | - wget -O - -q https://github.com/checkstyle/checkstyle/releases/download/checkstyle-8.43/checkstyle-8.43-all.jar > /opt/checkstyle.jar - wget -O - -q https://raw.githubusercontent.com/reviewdog/reviewdog/master/install.sh | sh -s -- -b /opt ${REVIEWDOG_VERSION} - java -jar /opt/checkstyle.jar "${WORKDIR}" -c "${CHECKSTYLE_CONFIG}" -f xml \ - | /opt/reviewdog -f=checkstyle \ - -reporter="${INPUT_REPORTER:-github-pr-check}" \ - -filter-mode="${INPUT_FILTER_MODE:-added}" \ - -fail-on-error="${INPUT_FAIL_ON_ERROR:-false}" + name: unit-test-logs + path: ${LOG_DIR} diff --git a/.gitmodules b/.gitmodules index 11414db08c..64a562af13 100644 --- a/.gitmodules +++ b/.gitmodules @@ -21,3 +21,9 @@ [submodule ".github/actions/lable-on-issue"] path = .github/actions/lable-on-issue url = https://github.com/xingchun-chen/labeler +[submodule ".github/actions/translate-on-issue"] + path = .github/actions/translate-on-issue + url = https://github.com/xingchun-chen/translation-helper.git +[submodule ".github/actions/reviewdog-setup"] + path = .github/actions/reviewdog-setup + url = https://github.com/reviewdog/action-setup diff --git a/.licenserc.yaml b/.licenserc.yaml index 8f69da5608..44a776ee59 100644 --- a/.licenserc.yaml +++ b/.licenserc.yaml @@ -40,5 +40,10 @@ header: - '**/.gitignore' - '**/LICENSE' - '**/NOTICE' + - '**/node_modules/**' + - '.github/actions/comment-on-issue/**' + - '.github/actions/lable-on-issue/**' + - '.github/actions/reviewdog-setup/**' + - '.github/actions/translate-on-issue/**' comment: on-failure diff --git a/README.md b/README.md index 9a4d7a81ad..5e304fde68 100644 --- a/README.md +++ b/README.md @@ -8,6 +8,7 @@ Dolphin Scheduler Official Website [![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=apache-dolphinscheduler&metric=alert_status)](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler) [![Twitter Follow](https://img.shields.io/twitter/follow/dolphinschedule.svg?style=social&label=Follow)](https://twitter.com/dolphinschedule) [![Slack Status](https://img.shields.io/badge/slack-join_chat-white.svg?logo=slack&style=social)](https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-omtdhuio-_JISsxYhiVsltmC5h38yfw) +[![Join the chat at https://gitter.im/apache-dolphinscheduler/community](https://badges.gitter.im/apache-dolphinscheduler/community.svg)](https://gitter.im/apache-dolphinscheduler/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) @@ -91,11 +92,11 @@ We would like to express our deep gratitude to all the open-source projects used You are very welcome to communicate with the developers and users of Dolphin Scheduler. There are two ways to find them: 1. Join the Slack channel by [this invitation link](https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-omtdhuio-_JISsxYhiVsltmC5h38yfw). -2. Follow the [Twitter account of Dolphin Scheduler](https://twitter.com/dolphinschedule) and get the latest news on time. +2. Follow the [Twitter account of DolphinScheduler](https://twitter.com/dolphinschedule) and get the latest news on time. ### Contributor over time -[![Contributor over time](https://contributor-graph-api.apiseven.com/contributors-svg?chart=contributorOverTime&repo=apache/dolphinscheduler)](https://www.apiseven.com/en/contributor-graph?chart=contributorOverTime&repo=apache/dolphinscheduler) +[![Contributor over time](https://contributor-graph-api.apiseven.com/contributors-svg?chart=contributorOverTime&repo=apache/dolphinscheduler)](https://www.apiseven.com/en/contributor-graph?chart=contributorOverTime&repo=apache/dolphinscheduler) ## How to Contribute diff --git a/docker/build/hooks/build b/docker/build/hooks/build index 0590761eeb..70ea260dea 100755 --- a/docker/build/hooks/build +++ b/docker/build/hooks/build @@ -39,8 +39,8 @@ echo "Repo: $DOCKER_REPO" echo -e "Current Directory is $(pwd)\n" # maven package(Project Directory) -echo -e "mvn -B clean compile package -Prelease -Dmaven.test.skip=true" -mvn -B clean compile package -Prelease -Dmaven.test.skip=true +echo -e "./mvnw -B clean package -Prelease -Dmaven.test.skip=true -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.httpconnectionManager.ttlSeconds=120" +./mvnw -B clean package -Prelease -Dmaven.test.skip=true -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.httpconnectionManager.ttlSeconds=120 # mv dolphinscheduler-bin.tar.gz file to docker/build directory echo -e "mv $(pwd)/dolphinscheduler-dist/target/apache-dolphinscheduler-${VERSION}-bin.tar.gz $(pwd)/docker/build/\n" diff --git a/docker/build/hooks/build.bat b/docker/build/hooks/build.bat index d4d538b5dc..6aa3726ceb 100644 --- a/docker/build/hooks/build.bat +++ b/docker/build/hooks/build.bat @@ -39,8 +39,8 @@ echo "Repo: %DOCKER_REPO%" echo "Current Directory is %cd%" :: maven package(Project Directory) -echo "call mvn clean compile package -Prelease" -call mvn clean compile package -Prelease -DskipTests=true +echo "mvn clean package -Prelease -DskipTests=true -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.httpconnectionManager.ttlSeconds=120" +call mvn clean package -Prelease -DskipTests=true -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.httpconnectionManager.ttlSeconds=120 if "%errorlevel%"=="1" goto :mvnFailed :: move dolphinscheduler-bin.tar.gz file to docker/build directory diff --git a/dolphinscheduler-alert-plugin/dolphinscheduler-alert-email/pom.xml b/dolphinscheduler-alert-plugin/dolphinscheduler-alert-email/pom.xml index 74dedf4e0f..079185cf0c 100644 --- a/dolphinscheduler-alert-plugin/dolphinscheduler-alert-email/pom.xml +++ b/dolphinscheduler-alert-plugin/dolphinscheduler-alert-email/pom.xml @@ -31,17 +31,6 @@ dolphinscheduler-plugin - - - com.fasterxml.jackson.core - jackson-annotations - provided - - - com.fasterxml.jackson.core - jackson-databind - provided - org.apache.commons commons-collections4 @@ -131,4 +120,4 @@ dolphinscheduler-alert-email-${project.version} - \ No newline at end of file + diff --git a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ExecutorController.java b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ExecutorController.java index 605a960879..762c52579d 100644 --- a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ExecutorController.java +++ b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ExecutorController.java @@ -99,6 +99,7 @@ public class ExecutorController extends BaseController { @ApiImplicitParam(name = "processInstancePriority", value = "PROCESS_INSTANCE_PRIORITY", required = true, dataType = "Priority"), @ApiImplicitParam(name = "workerGroup", value = "WORKER_GROUP", dataType = "String", example = "default"), @ApiImplicitParam(name = "timeout", value = "TIMEOUT", dataType = "Int", example = "100"), + @ApiImplicitParam(name = "expectedParallelismNumber", value = "EXPECTED_PARALLELISM_NUMBER", dataType = "Int", example = "8") }) @PostMapping(value = "start-process-instance") @ResponseStatus(HttpStatus.OK) @@ -118,7 +119,8 @@ public class ExecutorController extends BaseController { @RequestParam(value = "processInstancePriority", required = false) Priority processInstancePriority, @RequestParam(value = "workerGroup", required = false, defaultValue = "default") String workerGroup, @RequestParam(value = "timeout", required = false) Integer timeout, - @RequestParam(value = "startParams", required = false) String startParams) { + @RequestParam(value = "startParams", required = false) String startParams, + @RequestParam(value = "timeout", required = false) Integer expectedParallelismNumber) { if (timeout == null) { timeout = Constants.MAX_TASK_TIMEOUT; @@ -128,8 +130,7 @@ public class ExecutorController extends BaseController { startParamMap = JSONUtils.toMap(startParams); } Map result = execService.execProcessInstance(loginUser, projectCode, processDefinitionCode, scheduleTime, execType, failureStrategy, - startNodeList, taskDependType, warningType, - warningGroupId, runMode, processInstancePriority, workerGroup, timeout, startParamMap); + startNodeList, taskDependType, warningType, warningGroupId, runMode, processInstancePriority, workerGroup, timeout, startParamMap, expectedParallelismNumber); return returnDataList(result); } diff --git a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/ExecutorService.java b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/ExecutorService.java index ac850fff89..88acff3546 100644 --- a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/ExecutorService.java +++ b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/ExecutorService.java @@ -52,6 +52,7 @@ public interface ExecutorService { * @param runMode run mode * @param timeout timeout * @param startParams the global param values which pass to new process instance + * @param expectedParallelismNumber the expected parallelism number when execute complement in parallel mode * @return execute process instance code */ Map execProcessInstance(User loginUser, long projectCode, @@ -60,7 +61,7 @@ public interface ExecutorService { TaskDependType taskDependType, WarningType warningType, int warningGroupId, RunMode runMode, Priority processInstancePriority, String workerGroup, Integer timeout, - Map startParams); + Map startParams, Integer expectedParallelismNumber); /** * check whether the process definition can be executed diff --git a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java index 28c8c67ef2..35f9330139 100644 --- a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java +++ b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java @@ -51,6 +51,7 @@ import org.apache.dolphinscheduler.dao.entity.Schedule; import org.apache.dolphinscheduler.dao.entity.Tenant; import org.apache.dolphinscheduler.dao.entity.User; import org.apache.dolphinscheduler.dao.mapper.ProcessDefinitionMapper; +import org.apache.dolphinscheduler.dao.mapper.ProcessInstanceMapper; import org.apache.dolphinscheduler.dao.mapper.ProjectMapper; import org.apache.dolphinscheduler.service.process.ProcessService; import org.apache.dolphinscheduler.service.quartz.cron.CronUtils; @@ -89,6 +90,11 @@ public class ExecutorServiceImpl extends BaseServiceImpl implements ExecutorServ @Autowired private MonitorService monitorService; + + @Autowired + private ProcessInstanceMapper processInstanceMapper; + + @Autowired private ProcessService processService; @@ -100,7 +106,7 @@ public class ExecutorServiceImpl extends BaseServiceImpl implements ExecutorServ * @param processDefinitionCode process definition code * @param cronTime cron time * @param commandType command type - * @param failureStrategy failuer strategy + * @param failureStrategy failure strategy * @param startNodeList start nodelist * @param taskDependType node dependency type * @param warningType warning type @@ -110,6 +116,7 @@ public class ExecutorServiceImpl extends BaseServiceImpl implements ExecutorServ * @param runMode run mode * @param timeout timeout * @param startParams the global param values which pass to new process instance + * @param expectedParallelismNumber the expected parallelism number when execute complement in parallel mode * @return execute process instance code */ @Override @@ -119,7 +126,7 @@ public class ExecutorServiceImpl extends BaseServiceImpl implements ExecutorServ TaskDependType taskDependType, WarningType warningType, int warningGroupId, RunMode runMode, Priority processInstancePriority, String workerGroup, Integer timeout, - Map startParams) { + Map startParams, Integer expectedParallelismNumber) { Project project = projectMapper.queryByCode(projectCode); //check user access for project Map result = projectService.checkProjectAndAuth(loginUser, project, projectCode); @@ -156,7 +163,7 @@ public class ExecutorServiceImpl extends BaseServiceImpl implements ExecutorServ */ int create = this.createCommand(commandType, processDefinition.getCode(), taskDependType, failureStrategy, startNodeList, cronTime, warningType, loginUser.getId(), - warningGroupId, runMode, processInstancePriority, workerGroup, startParams); + warningGroupId, runMode, processInstancePriority, workerGroup, startParams, expectedParallelismNumber); if (create > 0) { processDefinition.setWarningGroupId(warningGroupId); @@ -485,7 +492,7 @@ public class ExecutorServiceImpl extends BaseServiceImpl implements ExecutorServ String startNodeList, String schedule, WarningType warningType, int executorId, int warningGroupId, RunMode runMode, Priority processInstancePriority, String workerGroup, - Map startParams) { + Map startParams, Integer expectedParallelismNumber) { /** * instantiate command schedule instance @@ -542,21 +549,31 @@ public class ExecutorServiceImpl extends BaseServiceImpl implements ExecutorServ return processService.createCommand(command); } else if (runMode == RunMode.RUN_MODE_PARALLEL) { List schedules = processService.queryReleaseSchedulerListByProcessDefinitionCode(processDefineCode); - List listDate = new LinkedList<>(); + LinkedList listDate = new LinkedList<>(); if (!CollectionUtils.isEmpty(schedules)) { for (Schedule item : schedules) { listDate.addAll(CronUtils.getSelfFireDateList(start, end, item.getCrontab())); } } if (!CollectionUtils.isEmpty(listDate)) { - // loop by schedule date - for (Date date : listDate) { - cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.dateToString(date)); - cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, DateUtils.dateToString(date)); + int effectThreadsCount = expectedParallelismNumber == null ? listDate.size() : Math.min(listDate.size(), expectedParallelismNumber); + logger.info("In parallel mode, current expectedParallelismNumber:{}", effectThreadsCount); + + int chunkSize = listDate.size() / effectThreadsCount; + listDate.addFirst(start); + listDate.addLast(end); + + for (int i = 0; i < effectThreadsCount; i++) { + int rangeStart = i == 0 ? i : (i * chunkSize); + int rangeEnd = i == effectThreadsCount - 1 ? listDate.size() - 1 + : rangeStart + chunkSize + 1; + cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.dateToString(listDate.get(rangeStart))); + cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, DateUtils.dateToString(listDate.get(rangeEnd))); command.setCommandParam(JSONUtils.toJsonString(cmdParam)); processService.createCommand(command); } - return listDate.size(); + + return effectThreadsCount; } else { // loop by day int runCunt = 0; diff --git a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/TenantServiceImpl.java b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/TenantServiceImpl.java index a362131f16..c690ed38b0 100644 --- a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/TenantServiceImpl.java +++ b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/TenantServiceImpl.java @@ -155,8 +155,8 @@ public class TenantServiceImpl extends BaseServiceImpl implements TenantService * updateProcessInstance tenant * * @param loginUser login user - * @param id tennat id - * @param tenantCode tennat code + * @param id tenant id + * @param tenantCode tenant code * @param queueId queue id * @param desc description * @return update result code diff --git a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/UsersServiceImpl.java b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/UsersServiceImpl.java index fd19570450..3b4d78f593 100644 --- a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/UsersServiceImpl.java +++ b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/UsersServiceImpl.java @@ -313,7 +313,7 @@ public class UsersServiceImpl extends BaseServiceImpl implements UsersService { * * @param loginUser login user * @param pageNo page number - * @param searchVal search avlue + * @param searchVal search value * @param pageSize page size * @return user list page */ @@ -347,7 +347,7 @@ public class UsersServiceImpl extends BaseServiceImpl implements UsersService { * @param userName user name * @param userPassword user password * @param email email - * @param tenantId tennat id + * @param tenantId tenant id * @param phone phone * @param queue queue * @return update result code diff --git a/dolphinscheduler-api/src/main/resources/i18n/messages_en_US.properties b/dolphinscheduler-api/src/main/resources/i18n/messages_en_US.properties index a7c8b84324..db705be6ab 100644 --- a/dolphinscheduler-api/src/main/resources/i18n/messages_en_US.properties +++ b/dolphinscheduler-api/src/main/resources/i18n/messages_en_US.properties @@ -171,6 +171,7 @@ PROCESS_INSTANCE_START_TIME=process instance start time PROCESS_INSTANCE_END_TIME=process instance end time PROCESS_INSTANCE_SIZE=process instance size PROCESS_INSTANCE_PRIORITY=process instance priority +EXPECTED_PARALLELISM_NUMBER=custom parallelism to set the complement task threads UPDATE_SCHEDULE_NOTES=update schedule SCHEDULE_ID=schedule id ONLINE_SCHEDULE_NOTES=online schedule diff --git a/dolphinscheduler-api/src/main/resources/i18n/messages_zh_CN.properties b/dolphinscheduler-api/src/main/resources/i18n/messages_zh_CN.properties index acc5be8631..ec88f74fb5 100644 --- a/dolphinscheduler-api/src/main/resources/i18n/messages_zh_CN.properties +++ b/dolphinscheduler-api/src/main/resources/i18n/messages_zh_CN.properties @@ -157,6 +157,7 @@ RECEIVERS=收件人 RECEIVERS_CC=收件人(抄送) WORKER_GROUP_ID=Worker Server分组ID PROCESS_INSTANCE_PRIORITY=流程实例优先级 +EXPECTED_PARALLELISM_NUMBER=补数任务自定义并行度 UPDATE_SCHEDULE_NOTES=更新定时 SCHEDULE_ID=定时ID ONLINE_SCHEDULE_NOTES=定时上线 diff --git a/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/controller/ExecutorControllerTest.java b/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/controller/ExecutorControllerTest.java index 1bf10ae942..5751e019d4 100644 --- a/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/controller/ExecutorControllerTest.java +++ b/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/controller/ExecutorControllerTest.java @@ -22,23 +22,16 @@ import static org.springframework.test.web.servlet.result.MockMvcResultMatchers. import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status; import org.apache.dolphinscheduler.api.enums.ExecuteType; -import org.apache.dolphinscheduler.api.enums.Status; -import org.apache.dolphinscheduler.api.service.ExecutorService; import org.apache.dolphinscheduler.api.utils.Result; -import org.apache.dolphinscheduler.common.Constants; import org.apache.dolphinscheduler.common.enums.FailureStrategy; import org.apache.dolphinscheduler.common.enums.WarningType; import org.apache.dolphinscheduler.common.utils.JSONUtils; -import java.util.HashMap; -import java.util.Map; - import org.junit.Assert; +import org.junit.Ignore; import org.junit.Test; -import org.mockito.Mockito; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import org.springframework.boot.test.mock.mockito.MockBean; import org.springframework.http.MediaType; import org.springframework.test.web.servlet.MvcResult; import org.springframework.util.LinkedMultiValueMap; @@ -51,79 +44,65 @@ public class ExecutorControllerTest extends AbstractControllerTest { private static Logger logger = LoggerFactory.getLogger(ExecutorControllerTest.class); - @MockBean - private ExecutorService executorService; - + @Ignore @Test public void testStartProcessInstance() throws Exception { - Map resultData = new HashMap<>(); - resultData.put(Constants.STATUS, Status.SUCCESS); - Mockito.when(executorService.execProcessInstance(Mockito.any(), Mockito.anyLong(), Mockito.anyLong(), Mockito.any(), - Mockito.any(), Mockito.any(), Mockito.any(), Mockito.any(), Mockito.any(), Mockito.anyInt(), - Mockito.any(), Mockito.any(), Mockito.any(), Mockito.anyInt(), Mockito.any())).thenReturn(resultData); - MultiValueMap paramsMap = new LinkedMultiValueMap<>(); - paramsMap.add("processDefinitionCode", "1"); + paramsMap.add("processDefinitionId", "40"); paramsMap.add("scheduleTime", ""); paramsMap.add("failureStrategy", String.valueOf(FailureStrategy.CONTINUE)); paramsMap.add("startNodeList", ""); paramsMap.add("taskDependType", ""); paramsMap.add("execType", ""); paramsMap.add("warningType", String.valueOf(WarningType.NONE)); - paramsMap.add("warningGroupId", "1"); + paramsMap.add("warningGroupId", ""); paramsMap.add("receivers", ""); paramsMap.add("receiversCc", ""); paramsMap.add("runMode", ""); paramsMap.add("processInstancePriority", ""); - paramsMap.add("workerGroupId", "1"); + paramsMap.add("workerGroupId", ""); paramsMap.add("timeout", ""); - MvcResult mvcResult = mockMvc.perform(post("/projects/{projectCode}/executors/start-process-instance", 1L) + MvcResult mvcResult = mockMvc.perform(post("/projects/{projectName}/executors/start-process-instance", "cxc_1113") .header("sessionId", sessionId) .params(paramsMap)) .andExpect(status().isOk()) .andExpect(content().contentType(MediaType.APPLICATION_JSON_UTF8)) .andReturn(); Result result = JSONUtils.parseObject(mvcResult.getResponse().getContentAsString(), Result.class); - Assert.assertEquals(Status.SUCCESS.getCode(), result.getCode().intValue()); + Assert.assertTrue(result != null && result.isSuccess()); logger.info(mvcResult.getResponse().getContentAsString()); } + @Ignore @Test public void testExecute() throws Exception { - Map resultData = new HashMap<>(); - resultData.put(Constants.STATUS, Status.SUCCESS); - Mockito.when(executorService.execute(Mockito.any(), Mockito.anyLong(), Mockito.anyInt(), Mockito.any())).thenReturn(resultData); - MultiValueMap paramsMap = new LinkedMultiValueMap<>(); paramsMap.add("processInstanceId", "40"); paramsMap.add("executeType", String.valueOf(ExecuteType.NONE)); - MvcResult mvcResult = mockMvc.perform(post("/projects/{projectCode}/executors/execute", 1L) + MvcResult mvcResult = mockMvc.perform(post("/projects/{projectName}/executors/execute", "cxc_1113") .header("sessionId", sessionId) .params(paramsMap)) .andExpect(status().isOk()) .andExpect(content().contentType(MediaType.APPLICATION_JSON_UTF8)) .andReturn(); Result result = JSONUtils.parseObject(mvcResult.getResponse().getContentAsString(), Result.class); - Assert.assertEquals(Status.SUCCESS.getCode(), result.getCode().intValue()); + Assert.assertTrue(result != null && result.isSuccess()); logger.info(mvcResult.getResponse().getContentAsString()); } @Test - public void testStartCheck() throws Exception { - Map resultData = new HashMap<>(); - resultData.put(Constants.STATUS, Status.SUCCESS); - Mockito.when(executorService.startCheckByProcessDefinedCode(Mockito.anyLong())).thenReturn(resultData); + public void testStartCheckProcessDefinition() throws Exception { - MvcResult mvcResult = mockMvc.perform(post("/projects/{projectCode}/executors/start-check", 1L) + MvcResult mvcResult = mockMvc.perform(post("/projects/{projectName}/executors/start-check", "cxc_1113") .header(SESSION_ID, sessionId) - .param("processDefinitionCode", "1")) + .param("processDefinitionId", "40")) .andExpect(status().isOk()) .andExpect(content().contentType(MediaType.APPLICATION_JSON_UTF8)) .andReturn(); Result result = JSONUtils.parseObject(mvcResult.getResponse().getContentAsString(), Result.class); - Assert.assertEquals(Status.SUCCESS.getCode(), result.getCode().intValue()); + Assert.assertTrue(result != null && result.isSuccess()); logger.info(mvcResult.getResponse().getContentAsString()); } diff --git a/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/service/ExecutorServiceTest.java b/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/service/ExecutorServiceTest.java index 4c1b3e4c63..e962be5257 100644 --- a/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/service/ExecutorServiceTest.java +++ b/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/service/ExecutorServiceTest.java @@ -158,7 +158,7 @@ public class ExecutorServiceTest { null, null, null, null, 0, RunMode.RUN_MODE_SERIAL, - Priority.LOW, Constants.DEFAULT_WORKER_GROUP, 110, null); + Priority.LOW, Constants.DEFAULT_WORKER_GROUP, 110, null, 0); Assert.assertEquals(Status.SUCCESS, result.get(Constants.STATUS)); verify(processService, times(1)).createCommand(any(Command.class)); @@ -176,7 +176,7 @@ public class ExecutorServiceTest { null, "n1,n2", null, null, 0, RunMode.RUN_MODE_SERIAL, - Priority.LOW, Constants.DEFAULT_WORKER_GROUP, 110, null); + Priority.LOW, Constants.DEFAULT_WORKER_GROUP, 110, null, 0); Assert.assertEquals(Status.SUCCESS, result.get(Constants.STATUS)); verify(processService, times(1)).createCommand(any(Command.class)); @@ -194,7 +194,7 @@ public class ExecutorServiceTest { null, null, null, null, 0, RunMode.RUN_MODE_SERIAL, - Priority.LOW, Constants.DEFAULT_WORKER_GROUP, 110, null); + Priority.LOW, Constants.DEFAULT_WORKER_GROUP, 110, null, 0); Assert.assertEquals(Status.START_PROCESS_INSTANCE_ERROR, result.get(Constants.STATUS)); verify(processService, times(0)).createCommand(any(Command.class)); } @@ -211,7 +211,7 @@ public class ExecutorServiceTest { null, null, null, null, 0, RunMode.RUN_MODE_SERIAL, - Priority.LOW, Constants.DEFAULT_WORKER_GROUP, 110, null); + Priority.LOW, Constants.DEFAULT_WORKER_GROUP, 110, null, 0); Assert.assertEquals(Status.SUCCESS, result.get(Constants.STATUS)); verify(processService, times(1)).createCommand(any(Command.class)); } @@ -228,7 +228,7 @@ public class ExecutorServiceTest { null, null, null, null, 0, RunMode.RUN_MODE_PARALLEL, - Priority.LOW, Constants.DEFAULT_WORKER_GROUP, 110, null); + Priority.LOW, Constants.DEFAULT_WORKER_GROUP, 110, null, 0); Assert.assertEquals(Status.SUCCESS, result.get(Constants.STATUS)); verify(processService, times(31)).createCommand(any(Command.class)); @@ -246,7 +246,7 @@ public class ExecutorServiceTest { null, null, null, null, 0, RunMode.RUN_MODE_PARALLEL, - Priority.LOW, Constants.DEFAULT_WORKER_GROUP, 110, null); + Priority.LOW, Constants.DEFAULT_WORKER_GROUP, 110, null, 15); Assert.assertEquals(Status.SUCCESS, result.get(Constants.STATUS)); verify(processService, times(15)).createCommand(any(Command.class)); @@ -261,7 +261,7 @@ public class ExecutorServiceTest { null, null, null, null, 0, RunMode.RUN_MODE_PARALLEL, - Priority.LOW, Constants.DEFAULT_WORKER_GROUP, 110, null); + Priority.LOW, Constants.DEFAULT_WORKER_GROUP, 110, null, 0); Assert.assertEquals(result.get(Constants.STATUS), Status.MASTER_NOT_EXISTS); } diff --git a/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/enums/DbType.java b/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/enums/DbType.java index 46d59d11fc..b994afb5f5 100644 --- a/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/enums/DbType.java +++ b/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/enums/DbType.java @@ -14,65 +14,45 @@ * See the License for the specific language governing permissions and * limitations under the License. */ + package org.apache.dolphinscheduler.common.enums; -import com.baomidou.mybatisplus.annotation.EnumValue; +import static java.util.stream.Collectors.toMap; -import java.util.HashMap; +import java.util.Arrays; +import java.util.Map; -/** - * data base types - */ -public enum DbType { - /** - * 0 mysql - * 1 postgresql - * 2 hive - * 3 spark - * 4 clickhouse - * 5 oracle - * 6 sqlserver - * 7 db2 - * 8 presto - */ - MYSQL(0, "mysql"), - POSTGRESQL(1, "postgresql"), - HIVE(2, "hive"), - SPARK(3, "spark"), - CLICKHOUSE(4, "clickhouse"), - ORACLE(5, "oracle"), - SQLSERVER(6, "sqlserver"), - DB2(7, "db2"), - PRESTO(8, "presto"); +import com.baomidou.mybatisplus.annotation.EnumValue; +import com.google.common.base.Functions; - DbType(int code, String descp) { +public enum DbType { + MYSQL(0), + POSTGRESQL(1), + HIVE(2), + SPARK(3), + CLICKHOUSE(4), + ORACLE(5), + SQLSERVER(6), + DB2(7), + PRESTO(8), + H2(9); + + DbType(int code) { this.code = code; - this.descp = descp; } @EnumValue private final int code; - private final String descp; public int getCode() { return code; } - public String getDescp() { - return descp; - } - - - private static HashMap DB_TYPE_MAP =new HashMap<>(); - - static { - for (DbType dbType:DbType.values()){ - DB_TYPE_MAP.put(dbType.getCode(),dbType); - } - } + private static final Map DB_TYPE_MAP = + Arrays.stream(DbType.values()).collect(toMap(DbType::getCode, Functions.identity())); - public static DbType of(int type){ - if(DB_TYPE_MAP.containsKey(type)){ + public static DbType of(int type) { + if (DB_TYPE_MAP.containsKey(type)) { return DB_TYPE_MAP.get(type); } return null; diff --git a/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/graph/DAG.java b/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/graph/DAG.java index deaf80fa04..397f32ed6e 100644 --- a/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/graph/DAG.java +++ b/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/graph/DAG.java @@ -432,13 +432,11 @@ public class DAG { * @return all neighbor nodes of the node */ private Set getNeighborNodes(Node node, final Map> edges) { - final Map neighborEdges = edges.get(node); - - if (neighborEdges == null) { - return Collections.EMPTY_MAP.keySet(); - } - - return neighborEdges.keySet(); + final Map neighborEdges = edges.get(node); + if (neighborEdges == null) { + return Collections.emptySet(); + } + return neighborEdges.keySet(); } diff --git a/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/PropertyUtils.java b/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/PropertyUtils.java index 065d7bc2ea..53a97d9755 100644 --- a/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/PropertyUtils.java +++ b/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/PropertyUtils.java @@ -34,15 +34,7 @@ import java.util.Set; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -/** - * property utils - * single instance - */ public class PropertyUtils { - - /** - * logger - */ private static final Logger logger = LoggerFactory.getLogger(PropertyUtils.class); private static final Properties properties = new Properties(); @@ -55,9 +47,6 @@ public class PropertyUtils { loadPropertyFile(COMMON_PROPERTIES_PATH); } - /** - * init properties - */ public static synchronized void loadPropertyFile(String... propertyFiles) { for (String fileName : propertyFiles) { try (InputStream fis = PropertyUtils.class.getResourceAsStream(fileName);) { @@ -68,6 +57,13 @@ public class PropertyUtils { System.exit(1); } } + + // Override from system properties + System.getProperties().forEach((k, v) -> { + final String key = String.valueOf(k); + logger.info("Overriding property from system property: {}", key); + PropertyUtils.setValue(key, String.valueOf(v)); + }); } /** diff --git a/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/datasource/SpringConnectionFactory.java b/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/datasource/SpringConnectionFactory.java index a58955da19..ca4a7e20bd 100644 --- a/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/datasource/SpringConnectionFactory.java +++ b/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/datasource/SpringConnectionFactory.java @@ -166,6 +166,7 @@ public class SpringConnectionFactory { Properties properties = new Properties(); properties.setProperty("MySQL", "mysql"); properties.setProperty("PostgreSQL", "pg"); + properties.setProperty("h2", "h2"); databaseIdProvider.setProperties(properties); return databaseIdProvider; } diff --git a/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ProcessDefinitionMapper.xml b/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ProcessDefinitionMapper.xml index 7817cba7d4..9f76dd13dc 100644 --- a/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ProcessDefinitionMapper.xml +++ b/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ProcessDefinitionMapper.xml @@ -77,7 +77,9 @@ left join t_ds_user tu on td.user_id = tu.id where td.project_code = #{projectCode} - and td.name like concat('%', #{searchVal}, '%') + AND (td.name like concat('%', #{searchVal}, '%') + OR td.description like concat('%', #{searchVal}, '%') + ) and td.user_id = #{userId} diff --git a/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ProcessInstanceMapper.xml b/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ProcessInstanceMapper.xml index 5880434746..db56301990 100644 --- a/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ProcessInstanceMapper.xml +++ b/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ProcessInstanceMapper.xml @@ -191,8 +191,8 @@ - and (schedule_time = ]]> #{startTime} and schedule_time #{endTime} - or start_time = ]]> #{startTime} and start_time #{endTime}) + and ((schedule_time = ]]> #{startTime} and schedule_time #{endTime}) + or (start_time = ]]> #{startTime} and start_time #{endTime})) order by start_time desc limit 1 diff --git a/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ProjectMapper.xml b/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ProjectMapper.xml index 2fc077c7e6..d1cc5f7d62 100644 --- a/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ProjectMapper.xml +++ b/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ProjectMapper.xml @@ -88,7 +88,9 @@ ) - and p.name like concat('%', #{searchName}, '%') + AND (p.name LIKE concat('%', #{searchName}, '%') + OR p.description LIKE concat('%', #{searchName}, '%') + ) order by p.create_time desc diff --git a/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/WorkFlowLineageMapper.xml b/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/WorkFlowLineageMapper.xml index eeddaf56e8..7d1dbfb40f 100644 --- a/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/WorkFlowLineageMapper.xml +++ b/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/WorkFlowLineageMapper.xml @@ -64,8 +64,7 @@ and project_code = #{projectCode} - select tepd.id as work_flow_id,tepd.name as work_flow_name, "" as source_work_flow_id, tepd.release_state as work_flow_publish_status, diff --git a/dolphinscheduler-dao/src/test/java/org/apache/dolphinscheduler/dao/utils/ResourceProcessDefinitionUtilsTest.java b/dolphinscheduler-dao/src/test/java/org/apache/dolphinscheduler/dao/utils/ResourceProcessDefinitionUtilsTest.java index 482aa6ea42..67828d7343 100644 --- a/dolphinscheduler-dao/src/test/java/org/apache/dolphinscheduler/dao/utils/ResourceProcessDefinitionUtilsTest.java +++ b/dolphinscheduler-dao/src/test/java/org/apache/dolphinscheduler/dao/utils/ResourceProcessDefinitionUtilsTest.java @@ -31,7 +31,7 @@ public class ResourceProcessDefinitionUtilsTest { @Test public void getResourceProcessDefinitionMapTest(){ List> mapList = new ArrayList<>(); - Map map = new HashMap(); + Map map = new HashMap<>(); map.put("code",1L); map.put("resource_ids","1,2,3"); mapList.add(map); diff --git a/dolphinscheduler-dist/pom.xml b/dolphinscheduler-dist/pom.xml index 33a711cb89..e999a498ab 100644 --- a/dolphinscheduler-dist/pom.xml +++ b/dolphinscheduler-dist/pom.xml @@ -37,6 +37,11 @@ dolphinscheduler-server + + org.apache.dolphinscheduler + dolphinscheduler-standalone-server + + org.apache.dolphinscheduler dolphinscheduler-api @@ -377,4 +382,4 @@ - \ No newline at end of file + diff --git a/dolphinscheduler-dist/release-docs/LICENSE b/dolphinscheduler-dist/release-docs/LICENSE index 2308359cdf..19da58bec9 100644 --- a/dolphinscheduler-dist/release-docs/LICENSE +++ b/dolphinscheduler-dist/release-docs/LICENSE @@ -249,6 +249,7 @@ The text of each license is also included at licenses/LICENSE-[project].txt. curator-client 4.3.0: https://mvnrepository.com/artifact/org.apache.curator/curator-client/4.3.0, Apache 2.0 curator-framework 4.3.0: https://mvnrepository.com/artifact/org.apache.curator/curator-framework/4.3.0, Apache 2.0 curator-recipes 4.3.0: https://mvnrepository.com/artifact/org.apache.curator/curator-recipes/4.3.0, Apache 2.0 + curator-test 2.12.0: https://mvnrepository.com/artifact/org.apache.curator/curator-test/2.12.0, Apache 2.0 datanucleus-api-jdo 4.2.1: https://mvnrepository.com/artifact/org.datanucleus/datanucleus-api-jdo/4.2.1, Apache 2.0 datanucleus-core 4.1.6: https://mvnrepository.com/artifact/org.datanucleus/datanucleus-core/4.1.6, Apache 2.0 datanucleus-rdbms 4.1.7: https://mvnrepository.com/artifact/org.datanucleus/datanucleus-rdbms/4.1.7, Apache 2.0 @@ -557,4 +558,4 @@ Apache 2.0 licenses ======================================== BSD licenses ======================================== - d3 3.5.17: https://github.com/d3/d3 BSD-3-Clause \ No newline at end of file + d3 3.5.17: https://github.com/d3/d3 BSD-3-Clause diff --git a/dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/dispatch/executor/NettyExecutorManager.java b/dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/dispatch/executor/NettyExecutorManager.java index bb1e314f6e..91c954a6ce 100644 --- a/dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/dispatch/executor/NettyExecutorManager.java +++ b/dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/dispatch/executor/NettyExecutorManager.java @@ -178,7 +178,7 @@ public class NettyExecutorManager extends AbstractExecutorManager{ * @return nodes */ private Set getAllNodes(ExecutionContext context){ - Set nodes = Collections.EMPTY_SET; + Set nodes = Collections.emptySet(); /** * executor type */ diff --git a/dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterBaseTaskExecThread.java b/dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterBaseTaskExecThread.java index a784e120ab..cfd8a9a0d0 100644 --- a/dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterBaseTaskExecThread.java +++ b/dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterBaseTaskExecThread.java @@ -191,7 +191,7 @@ public class MasterBaseTaskExecThread implements Callable { } /** - * dispatcht task + * dispatch task * * @param taskInstance taskInstance * @return whether submit task success diff --git a/dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/task/sql/SqlTask.java b/dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/task/sql/SqlTask.java index 9dd8b516ed..3c4b3ab273 100644 --- a/dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/task/sql/SqlTask.java +++ b/dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/task/sql/SqlTask.java @@ -42,6 +42,8 @@ import org.apache.dolphinscheduler.server.worker.task.AbstractTask; import org.apache.dolphinscheduler.service.alert.AlertClientService; import org.apache.dolphinscheduler.service.bean.SpringApplicationContext; +import org.apache.commons.collections.MapUtils; + import java.sql.Connection; import java.sql.PreparedStatement; import java.sql.ResultSet; @@ -271,11 +273,11 @@ public class SqlTask extends AbstractTask { public String setNonQuerySqlReturn(String updateResult, List properties) { String result = null; - for (Property info :properties) { + for (Property info : properties) { if (Direct.OUT == info.getDirect()) { - List> updateRL = new ArrayList<>(); - Map updateRM = new HashMap<>(); - updateRM.put(info.getProp(),updateResult); + List> updateRL = new ArrayList<>(); + Map updateRM = new HashMap<>(); + updateRM.put(info.getProp(), updateResult); updateRL.add(updateRM); result = JSONUtils.toJsonString(updateRL); break; @@ -490,6 +492,10 @@ public class SqlTask extends AbstractTask { public void printReplacedSql(String content, String formatSql, String rgex, Map sqlParamsMap) { //parameter print style logger.info("after replace sql , preparing : {}", formatSql); + if (MapUtils.isEmpty(sqlParamsMap)) { + logger.info("sqlParamsMap should not be Empty"); + return; + } StringBuilder logPrint = new StringBuilder("replaced sql , parameters:"); if (sqlParamsMap == null) { logger.info("printReplacedSql: sqlParamsMap is null."); diff --git a/dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/master/MasterExecThreadTest.java b/dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/master/MasterExecThreadTest.java index 196fb54e76..b8eb7ff075 100644 --- a/dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/master/MasterExecThreadTest.java +++ b/dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/master/MasterExecThreadTest.java @@ -101,8 +101,8 @@ public class MasterExecThreadTest { cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, "2020-01-20 23:00:00"); Mockito.when(processInstance.getCommandParam()).thenReturn(JSONUtils.toJsonString(cmdParam)); ProcessDefinition processDefinition = new ProcessDefinition(); - processDefinition.setGlobalParamMap(Collections.EMPTY_MAP); - processDefinition.setGlobalParamList(Collections.EMPTY_LIST); + processDefinition.setGlobalParamMap(Collections.emptyMap()); + processDefinition.setGlobalParamList(Collections.emptyList()); Mockito.when(processInstance.getProcessDefinition()).thenReturn(processDefinition); Mockito.when(processInstance.getProcessDefinitionCode()).thenReturn(processDefinitionCode); @@ -257,7 +257,7 @@ public class MasterExecThreadTest { } private List zeroSchedulerList() { - return Collections.EMPTY_LIST; + return Collections.emptyList(); } private List oneSchedulerList() { diff --git a/dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/worker/task/http/HttpTaskTest.java b/dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/worker/task/http/HttpTaskTest.java index f0d5d79d00..04b2a0ddcf 100644 --- a/dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/worker/task/http/HttpTaskTest.java +++ b/dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/worker/task/http/HttpTaskTest.java @@ -55,8 +55,6 @@ import org.springframework.context.ApplicationContext; public class HttpTaskTest { private static final Logger logger = LoggerFactory.getLogger(HttpTaskTest.class); - - private HttpTask httpTask; private ProcessService processService; @@ -168,7 +166,7 @@ public class HttpTaskTest { } catch (IOException e) { e.printStackTrace(); - }; + } } @Test diff --git a/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/quartz/ProcessScheduleJob.java b/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/quartz/ProcessScheduleJob.java index eacd8bcf09..1de5c56fd0 100644 --- a/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/quartz/ProcessScheduleJob.java +++ b/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/quartz/ProcessScheduleJob.java @@ -75,8 +75,8 @@ public class ProcessScheduleJob implements Job { // query schedule Schedule schedule = getProcessService().querySchedule(scheduleId); - if (schedule == null) { - logger.warn("process schedule does not exist in db，delete schedule job in quartz, projectId:{}, scheduleId:{}", projectId, scheduleId); + if (schedule == null || ReleaseState.OFFLINE == schedule.getReleaseState()) { + logger.warn("process schedule does not exist in db or process schedule offline，delete schedule job in quartz, projectId:{}, scheduleId:{}", projectId, scheduleId); deleteJob(projectId, scheduleId); return; } diff --git a/dolphinscheduler-standalone-server/pom.xml b/dolphinscheduler-standalone-server/pom.xml new file mode 100644 index 0000000000..505a3b56e2 --- /dev/null +++ b/dolphinscheduler-standalone-server/pom.xml @@ -0,0 +1,52 @@ + + + + + dolphinscheduler + org.apache.dolphinscheduler + 1.3.6-SNAPSHOT + + 4.0.0 + + dolphinscheduler-standalone-server + + + + org.apache.dolphinscheduler + dolphinscheduler-server + + + org.apache.dolphinscheduler + dolphinscheduler-api + + + org.apache.curator + curator-test + ${curator.test} + + + org.javassist + javassist + + + + + + diff --git a/dolphinscheduler-standalone-server/src/main/java/org/apache/dolphinscheduler/server/StandaloneServer.java b/dolphinscheduler-standalone-server/src/main/java/org/apache/dolphinscheduler/server/StandaloneServer.java new file mode 100644 index 0000000000..3b92b7f7cb --- /dev/null +++ b/dolphinscheduler-standalone-server/src/main/java/org/apache/dolphinscheduler/server/StandaloneServer.java @@ -0,0 +1,82 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.dolphinscheduler.server; + +import static org.apache.dolphinscheduler.common.Constants.SPRING_DATASOURCE_DRIVER_CLASS_NAME; +import static org.apache.dolphinscheduler.common.Constants.SPRING_DATASOURCE_PASSWORD; +import static org.apache.dolphinscheduler.common.Constants.SPRING_DATASOURCE_URL; +import static org.apache.dolphinscheduler.common.Constants.SPRING_DATASOURCE_USERNAME; + +import org.apache.dolphinscheduler.api.ApiApplicationServer; +import org.apache.dolphinscheduler.common.utils.ScriptRunner; +import org.apache.dolphinscheduler.dao.datasource.ConnectionFactory; +import org.apache.dolphinscheduler.server.master.MasterServer; +import org.apache.dolphinscheduler.server.worker.WorkerServer; + +import org.apache.curator.test.TestingServer; + +import java.io.FileReader; +import java.nio.file.Files; +import java.nio.file.Path; + +import javax.sql.DataSource; + +import org.h2.tools.Server; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.boot.autoconfigure.SpringBootApplication; +import org.springframework.boot.builder.SpringApplicationBuilder; + +@SpringBootApplication +public class StandaloneServer { + private static final Logger LOGGER = LoggerFactory.getLogger(StandaloneServer.class); + + public static void main(String[] args) throws Exception { + System.setProperty("spring.profiles.active", "api"); + + final Path temp = Files.createTempDirectory("dolphinscheduler_"); + LOGGER.info("H2 database directory: {}", temp); + System.setProperty( + SPRING_DATASOURCE_DRIVER_CLASS_NAME, + org.h2.Driver.class.getName() + ); + System.setProperty( + SPRING_DATASOURCE_URL, + String.format("jdbc:h2:tcp://localhost/%s", temp.toAbsolutePath()) + ); + System.setProperty(SPRING_DATASOURCE_USERNAME, "sa"); + System.setProperty(SPRING_DATASOURCE_PASSWORD, ""); + + Server.createTcpServer("-ifNotExists").start(); + + final DataSource ds = ConnectionFactory.getInstance().getDataSource(); + final ScriptRunner runner = new ScriptRunner(ds.getConnection(), true, true); + runner.runScript(new FileReader("sql/dolphinscheduler_h2.sql")); + + final TestingServer server = new TestingServer(true); + System.setProperty("registry.servers", server.getConnectString()); + + Thread.currentThread().setName("Standalone-Server"); + + new SpringApplicationBuilder( + ApiApplicationServer.class, + MasterServer.class, + WorkerServer.class + ).run(args); + } +} diff --git a/dolphinscheduler-standalone-server/src/main/resources/registry.properties b/dolphinscheduler-standalone-server/src/main/resources/registry.properties new file mode 100644 index 0000000000..3f557ce033 --- /dev/null +++ b/dolphinscheduler-standalone-server/src/main/resources/registry.properties @@ -0,0 +1,22 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# This file is only to override the production configurations in standalone server. + +registry.plugin.dir=./dolphinscheduler-dist/target/dolphinscheduler-dist-1.3.6-SNAPSHOT/lib/plugin/registry/zookeeper +registry.plugin.name=zookeeper +registry.servers=127.0.0.1:2181 diff --git a/dolphinscheduler-ui/src/js/conf/home/pages/projects/pages/definition/pages/list/_source/start.vue b/dolphinscheduler-ui/src/js/conf/home/pages/projects/pages/definition/pages/list/_source/start.vue index 2b478be062..982a15664b 100644 --- a/dolphinscheduler-ui/src/js/conf/home/pages/projects/pages/definition/pages/list/_source/start.vue +++ b/dolphinscheduler-ui/src/js/conf/home/pages/projects/pages/definition/pages/list/_source/start.vue @@ -116,12 +116,31 @@ {{$t('Mode of execution')}}
- + {{$t('Serial execution')}} {{$t('Parallel execution')}}
+
+
+ + {{$t('Parallelism')}} +
+
+ {{$t('Custom Parallelism')}} + + + +
+
{{$t('Schedule date')}} @@ -164,6 +183,7 @@