Merge branch 'dev' of https://github.com/apache/incubator-dolphinscheduler into dev_wuxiaofei_server

docker/build/README.md

@@ -13,84 +13,86 @@ Official Website: https://dolphinscheduler.apache.org
 
 ## How to use this docker image
 
-#### You can start a dolphinscheduler instance
+#### You can start a dolphinscheduler by docker-compose (recommended)
+
 ```
-$ docker run -dit --name dolphinscheduler \
--e DATABASE_USERNAME=test -e DATABASE_PASSWORD=test -e DATABASE_DATABASE=dolphinscheduler \
--p 12345:12345 \
-dolphinscheduler all
+$ docker-compose -f ./docker/docker-swarm/docker-compose.yml up -d
 ```
 
-The default postgres user `root`, postgres password `root` and database `dolphinscheduler` are created in the `startup.sh`.
+The default **postgres** user `root`, postgres password `root` and database `dolphinscheduler` are created in the `docker-compose.yml`.
 
-The default zookeeper is created in the `startup.sh`.
+The default **zookeeper** is created in the `docker-compose.yml`.
+
+Access the Web UI：http://192.168.xx.xx:12345/dolphinscheduler
 
 #### Or via Environment Variables **`DATABASE_HOST`** **`DATABASE_PORT`** **`DATABASE_DATABASE`** **`ZOOKEEPER_QUORUM`**
 
-You can specify **existing postgres service**. Example:
+You can specify **existing postgres and zookeeper service**. Example:
 
 ```
-$ docker run -dit --name dolphinscheduler \
+$ docker run -d --name dolphinscheduler \
+-e ZOOKEEPER_QUORUM="192.168.x.x:2181" \
 -e DATABASE_HOST="192.168.x.x" -e DATABASE_PORT="5432" -e DATABASE_DATABASE="dolphinscheduler" \
 -e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" \
 -p 12345:12345 \
-dolphinscheduler all
+apache/dolphinscheduler:latest all
 ```
 
-You can specify **existing zookeeper service**. Example:
-
-```
-$ docker run -dit --name dolphinscheduler \
--e ZOOKEEPER_QUORUM="l92.168.x.x:2181"
--e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" -e DATABASE_DATABASE="dolphinscheduler" \
--p 12345:12345 \
-dolphinscheduler all
-```
+Access the Web UI：http://192.168.xx.xx:12345/dolphinscheduler
 
 #### Or start a standalone dolphinscheduler server
 
 You can start a standalone dolphinscheduler server.
 
+* Create a **local volume** for resource storage, For example:
+
+```
+docker volume create dolphinscheduler-resource-local
+```
+
 * Start a **master server**, For example:
 
 ```
-$ docker run -dit --name dolphinscheduler \
--e ZOOKEEPER_QUORUM="l92.168.x.x:2181"
+$ docker run -d --name dolphinscheduler-master \
+-e ZOOKEEPER_QUORUM="192.168.x.x:2181" \
 -e DATABASE_HOST="192.168.x.x" -e DATABASE_PORT="5432" -e DATABASE_DATABASE="dolphinscheduler" \
 -e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" \
-dolphinscheduler master-server
+apache/dolphinscheduler:latest master-server
 ```
 
 * Start a **worker server**, For example:
 
 ```
-$ docker run -dit --name dolphinscheduler \
--e ZOOKEEPER_QUORUM="l92.168.x.x:2181"
+$ docker run -d --name dolphinscheduler-worker \
+-e ZOOKEEPER_QUORUM="192.168.x.x:2181" \
 -e DATABASE_HOST="192.168.x.x" -e DATABASE_PORT="5432" -e DATABASE_DATABASE="dolphinscheduler" \
 -e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" \
-dolphinscheduler worker-server
+-e ALERT_LISTEN_HOST="dolphinscheduler-alert" \
+-v dolphinscheduler-resource-local:/dolphinscheduler \
+apache/dolphinscheduler:latest worker-server
 ```
 
 * Start a **api server**, For example:
 
 ```
-$ docker run -dit --name dolphinscheduler \
+$ docker run -d --name dolphinscheduler-api \
+-e ZOOKEEPER_QUORUM="192.168.x.x:2181" \
 -e DATABASE_HOST="192.168.x.x" -e DATABASE_PORT="5432" -e DATABASE_DATABASE="dolphinscheduler" \
 -e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" \
+-v dolphinscheduler-resource-local:/dolphinscheduler \
 -p 12345:12345 \
-dolphinscheduler api-server
+apache/dolphinscheduler:latest api-server
 ```
 
 * Start a **alert server**, For example:
 
 ```
-$ docker run -dit --name dolphinscheduler \
+$ docker run -d --name dolphinscheduler-alert \
 -e DATABASE_HOST="192.168.x.x" -e DATABASE_PORT="5432" -e DATABASE_DATABASE="dolphinscheduler" \
 -e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" \
-dolphinscheduler alert-server
+apache/dolphinscheduler:latest alert-server
 ```
 
-
 **Note**: You must be specify `DATABASE_HOST` `DATABASE_PORT` `DATABASE_DATABASE` `DATABASE_USERNAME` `DATABASE_PASSWORD` `ZOOKEEPER_QUORUM` when start a standalone dolphinscheduler server.
 
 ## How to build a docker image
@@ -107,7 +109,7 @@ $ sh ./docker/build/hooks/build
 In Windows, Example:
 
 ```bat
-c:\incubator-dolphinscheduler>.\docker\build\hooks\build.bat
+C:\incubator-dolphinscheduler>.\docker\build\hooks\build.bat
 ```
 
 Please read `./docker/build/hooks/build` `./docker/build/hooks/build.bat` script files if you don't understand
@@ -172,12 +174,44 @@ This environment variable sets the runtime environment for task. The default val
 
 User data directory path, self configuration, please make sure the directory exists and have read write permissions. The default value is `/tmp/dolphinscheduler`
 
+**`DOLPHINSCHEDULER_OPTS`**
+
+This environment variable sets java options. The default value is empty.
+
+**`RESOURCE_STORAGE_TYPE`**
+
+This environment variable sets resource storage type for dolphinscheduler like `HDFS`, `S3`, `NONE`. The default value is `HDFS`.
+
+**`RESOURCE_UPLOAD_PATH`**
+
+This environment variable sets resource store path on HDFS/S3 for resource storage. The default value is `/dolphinscheduler`.
+
+**`FS_DEFAULT_FS`**
+
+This environment variable sets fs.defaultFS for resource storage like `file:///`, `hdfs://mycluster:8020` or `s3a://dolphinscheduler`. The default value is `file:///`.
+
+**`FS_S3A_ENDPOINT`**
+
+This environment variable sets s3 endpoint for resource storage. The default value is `s3.xxx.amazonaws.com`.
+
+**`FS_S3A_ACCESS_KEY`**
+
+This environment variable sets s3 access key for resource storage. The default value is `xxxxxxx`.
+
+**`FS_S3A_SECRET_KEY`**
+
+This environment variable sets s3 secret key for resource storage. The default value is `xxxxxxx`.
+
 **`ZOOKEEPER_QUORUM`**
 
 This environment variable sets zookeeper quorum for `master-server` and `worker-serverr`. The default value is `127.0.0.1:2181`.
 
 **Note**: You must be specify it when start a standalone dolphinscheduler server. Like `master-server`, `worker-server`.
 
+**`ZOOKEEPER_ROOT`**
+
+This environment variable sets zookeeper root directory for dolphinscheduler. The default value is `/dolphinscheduler`.
+
 **`MASTER_EXEC_THREADS`**
 
 This environment variable sets exec thread num for `master-server`. The default value is `100`.
@@ -218,10 +252,6 @@ This environment variable sets exec thread num for `worker-server`. The default
 
 This environment variable sets heartbeat interval for `worker-server`. The default value is `10`.
 
-**`WORKER_FETCH_TASK_NUM`**
-
-This environment variable sets fetch task num for `worker-server`. The default value is `3`.
-
 **`WORKER_MAX_CPULOAD_AVG`**
 
 This environment variable sets max cpu load avg for `worker-server`. The default value is `100`.
@@ -230,10 +260,6 @@ This environment variable sets max cpu load avg for `worker-server`. The default
 
 This environment variable sets reserved memory for `worker-server`. The default value is `0.1`.
 
-**`WORKER_WEIGHT`**
-
-This environment variable sets port for `worker-server`. The default value is `100`.
-
 **`WORKER_LISTEN_PORT`**
 
 This environment variable sets port for `worker-server`. The default value is `1234`.
@@ -242,61 +268,17 @@ This environment variable sets port for `worker-server`. The default value is `1
 
 This environment variable sets group for `worker-server`. The default value is `default`.
 
-**`XLS_FILE_PATH`**
-
-This environment variable sets xls file path for `alert-server`. The default value is `/tmp/xls`.
-
-**`MAIL_SERVER_HOST`**
-
-This environment variable sets mail server host for `alert-server`. The default value is empty.
-
-**`MAIL_SERVER_PORT`**
-
-This environment variable sets mail server port for `alert-server`. The default value is empty.
-
-**`MAIL_SENDER`**
-
-This environment variable sets mail sender for `alert-server`. The default value is empty.
-
-**`MAIL_USER=`**
-
-This environment variable sets mail user for `alert-server`. The default value is empty.
-
-**`MAIL_PASSWD`**
-
-This environment variable sets mail password for `alert-server`. The default value is empty.
-
-**`MAIL_SMTP_STARTTLS_ENABLE`**
-
-This environment variable sets SMTP tls for `alert-server`. The default value is `true`.
-
-**`MAIL_SMTP_SSL_ENABLE`**
-
-This environment variable sets SMTP ssl for `alert-server`. The default value is `false`.
-
-**`MAIL_SMTP_SSL_TRUST`**
-
-This environment variable sets SMTP ssl truest for `alert-server`. The default value is empty.
-
-**`ENTERPRISE_WECHAT_ENABLE`**
-
-This environment variable sets enterprise wechat enable for `alert-server`. The default value is `false`.
-
-**`ENTERPRISE_WECHAT_CORP_ID`**
-
-This environment variable sets enterprise wechat corp id for `alert-server`. The default value is empty.
-
-**`ENTERPRISE_WECHAT_SECRET`**
+**`WORKER_WEIGHT`**
 
-This environment variable sets enterprise wechat secret for `alert-server`. The default value is empty.
+This environment variable sets weight for `worker-server`. The default value is `100`.
 
-**`ENTERPRISE_WECHAT_AGENT_ID`**
+**`ALERT_LISTEN_HOST`**
 
-This environment variable sets enterprise wechat agent id for `alert-server`. The default value is empty.
+This environment variable sets the host of `alert-server` for `worker-server`. The default value is `127.0.0.1`.
 
-**`ENTERPRISE_WECHAT_USERS`**
+**`ALERT_PLUGIN_DIR`**
 
-This environment variable sets enterprise wechat users for `alert-server`. The default value is empty.
+This environment variable sets the alert plugin directory for `alert-server`. The default value is `lib/plugin/alert`.
 
 ## Initialization scripts
 

docker/build/README_zh_CN.md

@@ -13,81 +13,83 @@ Official Website: https://dolphinscheduler.apache.org
 
 ## 如何使用docker镜像
 
-#### 你可以运行一个dolphinscheduler实例
+#### 以 docker-compose 的方式启动dolphinscheduler(推荐)
 ```
-$ docker run -dit --name dolphinscheduler \
--e DATABASE_USERNAME=test -e DATABASE_PASSWORD=test -e DATABASE_DATABASE=dolphinscheduler \
--p 12345:12345 \
-dolphinscheduler all
+$ docker-compose -f ./docker/docker-swarm/docker-compose.yml up -d
 ```
 
-在`startup.sh`脚本中，默认的创建`Postgres`的用户、密码和数据库，默认值分别为：`root`、`root`、`dolphinscheduler`。
+在`docker-compose.yml`文件中，默认的创建`Postgres`的用户、密码和数据库，默认值分别为：`root`、`root`、`dolphinscheduler`。
+
+同时，默认的`Zookeeper`也会在`docker-compose.yml`文件中被创建。
 
-同时，默认的`Zookeeper`也会在`startup.sh`脚本中被创建。
+访问前端界面：http://192.168.xx.xx:12345/dolphinscheduler
 
 #### 或者通过环境变量 **`DATABASE_HOST`** **`DATABASE_PORT`** **`ZOOKEEPER_QUORUM`** 使用已存在的服务
 
-你可以指定一个已经存在的 **`Postgres`** 服务. 如下:
+你可以指定已经存在的 **`Postgres`** 和 **`Zookeeper`** 服务. 如下:
 
 ```
-$ docker run -dit --name dolphinscheduler \
+$ docker run -d --name dolphinscheduler \
+-e ZOOKEEPER_QUORUM="192.168.x.x:2181" \
 -e DATABASE_HOST="192.168.x.x" -e DATABASE_PORT="5432" -e DATABASE_DATABASE="dolphinscheduler" \
 -e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" \
 -p 12345:12345 \
-dolphinscheduler all
+apache/dolphinscheduler:latest all
 ```
 
-你也可以指定一个已经存在的 **Zookeeper** 服务. 如下:
-
-```
-$ docker run -dit --name dolphinscheduler \
--e ZOOKEEPER_QUORUM="l92.168.x.x:2181"
--e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" -e DATABASE_DATABASE="dolphinscheduler" \
--p 12345:12345 \
-dolphinscheduler all
-```
+访问前端界面：http://192.168.xx.xx:12345/dolphinscheduler
 
 #### 或者运行dolphinscheduler中的部分服务
 
 你能够运行dolphinscheduler中的部分服务。
 
+* 创建一个 **本地卷** 用于资源存储，如下:
+
+```
+docker volume create dolphinscheduler-resource-local
+```
+
 * 启动一个 **master server**, 如下:
 
 ```
-$ docker run -dit --name dolphinscheduler \
--e ZOOKEEPER_QUORUM="l92.168.x.x:2181"
+$ docker run -d --name dolphinscheduler-master \
+-e ZOOKEEPER_QUORUM="192.168.x.x:2181" \
 -e DATABASE_HOST="192.168.x.x" -e DATABASE_PORT="5432" -e DATABASE_DATABASE="dolphinscheduler" \
 -e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" \
-dolphinscheduler master-server
+apache/dolphinscheduler:latest master-server
 ```
 
 * 启动一个 **worker server**, 如下:
 
 ```
-$ docker run -dit --name dolphinscheduler \
--e ZOOKEEPER_QUORUM="l92.168.x.x:2181"
+$ docker run -d --name dolphinscheduler-worker \
+-e ZOOKEEPER_QUORUM="192.168.x.x:2181" \
 -e DATABASE_HOST="192.168.x.x" -e DATABASE_PORT="5432" -e DATABASE_DATABASE="dolphinscheduler" \
 -e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" \
-dolphinscheduler worker-server
+-e ALERT_LISTEN_HOST="dolphinscheduler-alert" \
+-v dolphinscheduler-resource-local:/dolphinscheduler \
+apache/dolphinscheduler:latest worker-server
 ```
 
 * 启动一个 **api server**, 如下:
 
 ```
-$ docker run -dit --name dolphinscheduler \
+$ docker run -d --name dolphinscheduler-api \
+-e ZOOKEEPER_QUORUM="192.168.x.x:2181" \
 -e DATABASE_HOST="192.168.x.x" -e DATABASE_PORT="5432" -e DATABASE_DATABASE="dolphinscheduler" \
 -e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" \
+-v dolphinscheduler-resource-local:/dolphinscheduler \
 -p 12345:12345 \
-dolphinscheduler api-server
+apache/dolphinscheduler:latest api-server
 ```
 
 * 启动一个 **alert server**, 如下:
 
 ```
-$ docker run -dit --name dolphinscheduler \
+$ docker run -d --name dolphinscheduler-alert \
 -e DATABASE_HOST="192.168.x.x" -e DATABASE_PORT="5432" -e DATABASE_DATABASE="dolphinscheduler" \
 -e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" \
-dolphinscheduler alert-server
+apache/dolphinscheduler:latest alert-server
 ```
 
 **注意**: 当你运行dolphinscheduler中的部分服务时，你必须指定这些环境变量 `DATABASE_HOST` `DATABASE_PORT` `DATABASE_DATABASE` `DATABASE_USERNAME` `DATABASE_PASSWORD` `ZOOKEEPER_QUORUM`。
@@ -106,7 +108,7 @@ $ sh ./docker/build/hooks/build
 Windows系统, 如下:
 
 ```bat
-c:\incubator-dolphinscheduler>.\docker\build\hooks\build.bat
+C:\incubator-dolphinscheduler>.\docker\build\hooks\build.bat
 ```
 
 如果你不理解这些脚本 `./docker/build/hooks/build` `./docker/build/hooks/build.bat`，请阅读里面的内容。
@@ -171,12 +173,44 @@ Dolphin Scheduler映像使用了几个容易遗漏的环境变量。虽然这些
 
 用户数据目录, 用户自己配置, 请确保这个目录存在并且用户读写权限， 默认值 `/tmp/dolphinscheduler`。
 
+**`DOLPHINSCHEDULER_OPTS`**
+
+配置`dolphinscheduler`的`java options`，默认值 `""`、
+
+**`RESOURCE_STORAGE_TYPE`**
+
+配置`dolphinscheduler`的资源存储类型，可选项为 `HDFS`、`S3`、`NONE`，默认值 `HDFS`。
+
+**`RESOURCE_UPLOAD_PATH`**
+
+配置`HDFS/S3`上的资源存储路径，默认值 `/dolphinscheduler`。
+
+**`FS_DEFAULT_FS`**
+
+配置资源存储的文件系统协议，如 `file:///`, `hdfs://mycluster:8020` or `s3a://dolphinscheduler`，默认值 `file:///`。
+
+**`FS_S3A_ENDPOINT`**
+
+当`RESOURCE_STORAGE_TYPE=S3`时，需要配置`S3`的访问路径，默认值 `s3.xxx.amazonaws.com`。
+
+**`FS_S3A_ACCESS_KEY`**
+
+当`RESOURCE_STORAGE_TYPE=S3`时，需要配置`S3`的`s3 access key`，默认值 `xxxxxxx`。
+
+**`FS_S3A_SECRET_KEY`**
+
+当`RESOURCE_STORAGE_TYPE=S3`时，需要配置`S3`的`s3 secret key`，默认值 `xxxxxxx`。
+
 **`ZOOKEEPER_QUORUM`**
 
 配置`master-server`和`worker-serverr`的`Zookeeper`地址, 默认值 `127.0.0.1:2181`。
 
 **注意**: 当运行`dolphinscheduler`中`master-server`、`worker-server`这些服务时，必须指定这个环境变量，以便于你更好的搭建分布式服务。
 
+**`ZOOKEEPER_ROOT`**
+
+配置`dolphinscheduler`在`zookeeper`中数据存储的根目录，默认值 `/dolphinscheduler`。
+
 **`MASTER_EXEC_THREADS`**
 
 配置`master-server`中的执行线程数量，默认值 `100`。
@@ -217,10 +251,6 @@ Dolphin Scheduler映像使用了几个容易遗漏的环境变量。虽然这些
 
 配置`worker-server`中的心跳交互时间，默认值 `10`。
 
-**`WORKER_FETCH_TASK_NUM`**
-
-配置`worker-server`中的获取任务的数量，默认值 `3`。
-
 **`WORKER_MAX_CPULOAD_AVG`**
 
 配置`worker-server`中的CPU中的最大`load average`值，默认值 `100`。
@@ -229,10 +259,6 @@ Dolphin Scheduler映像使用了几个容易遗漏的环境变量。虽然这些
 
 配置`worker-server`的保留内存，默认值 `0.1`。
 
-**`WORKER_WEIGHT`**
-
-配置`worker-server`的权重，默认之`100`。
-
 **`WORKER_LISTEN_PORT`**
 
 配置`worker-server`的端口，默认值 `1234`。
@@ -241,61 +267,17 @@ Dolphin Scheduler映像使用了几个容易遗漏的环境变量。虽然这些
 
 配置`worker-server`的分组，默认值 `default`。
 
-**`XLS_FILE_PATH`**
-
-配置`alert-server`的`XLS`文件的存储路径，默认值 `/tmp/xls`。
-
-**`MAIL_SERVER_HOST`**
-
-配置`alert-server`的邮件服务地址，默认值 `空`。
-
-**`MAIL_SERVER_PORT`**
-
-配置`alert-server`的邮件服务端口，默认值 `空`。
-
-**`MAIL_SENDER`**
-
-配置`alert-server`的邮件发送人，默认值 `空`。
-
-**`MAIL_USER=`**
-
-配置`alert-server`的邮件服务用户名，默认值 `空`。
-
-**`MAIL_PASSWD`**
-
-配置`alert-server`的邮件服务用户密码，默认值 `空`。
-
-**`MAIL_SMTP_STARTTLS_ENABLE`**
-
-配置`alert-server`的邮件服务是否启用TLS，默认值 `true`。
-
-**`MAIL_SMTP_SSL_ENABLE`**
-
-配置`alert-server`的邮件服务是否启用SSL，默认值 `false`。
-
-**`MAIL_SMTP_SSL_TRUST`**
-
-配置`alert-server`的邮件服务SSL的信任地址，默认值 `空`。
-
-**`ENTERPRISE_WECHAT_ENABLE`**
-
-配置`alert-server`的邮件服务是否启用企业微信，默认值 `false`。
-
-**`ENTERPRISE_WECHAT_CORP_ID`**
-
-配置`alert-server`的邮件服务企业微信`ID`，默认值 `空`。
-
-**`ENTERPRISE_WECHAT_SECRET`**
+**`WORKER_WEIGHT`**
 
-配置`alert-server`的邮件服务企业微信`SECRET`，默认值 `空`。
+配置`worker-server`的权重，默认之`100`。
 
-**`ENTERPRISE_WECHAT_AGENT_ID`**
+**`ALERT_LISTEN_HOST`**
 
-配置`alert-server`的邮件服务企业微信`AGENT_ID`，默认值 `空`。
+配置`worker-server`的告警主机，即`alert-server`的主机名，默认值 `127.0.0.1`。
 
-**`ENTERPRISE_WECHAT_USERS`**
+**`ALERT_PLUGIN_DIR`**
 
-配置`alert-server`的邮件服务企业微信`USERS`，默认值 `空`。
+配置`alert-server`的告警插件目录，默认值 `lib/plugin/alert`。
 
 ## 初始化脚本
 

docker/kubernetes/dolphinscheduler/templates/_helpers.tpl

@@ -129,13 +129,4 @@ We truncate at 63 chars because some Kubernetes name fields are limited to this
 {{- define "dolphinscheduler.zookeeper.quorum" -}}
 {{- $port := default "2181" (.Values.zookeeper.service.port | toString) -}}
 {{- printf "%s:%s" (include "dolphinscheduler.zookeeper.fullname" .) $port | trunc 63 | trimSuffix "-" -}}
-{{- end -}}
-
-{{/*
-Create a default dolphinscheduler worker base dir.
-We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
-*/}}
-{{- define "dolphinscheduler.data.basedir.path" -}}
-{{- $name := default "/tmp/dolphinscheduler" .Values.common.configmap.DOLPHINSCHEDULER_DATA_BASEDIR_PATH -}}
-{{- printf "%s" $name | trunc 63 | trimSuffix "/" -}}
 {{- end -}}

docker/kubernetes/dolphinscheduler/templates/configmap-dolphinscheduler-common.yaml

@@ -28,7 +28,7 @@ data:
   {{- range .Values.common.configmap.DOLPHINSCHEDULER_ENV }}
     {{ . }}
   {{- end }}
-  DOLPHINSCHEDULER_DATA_BASEDIR_PATH: {{ include "dolphinscheduler.data.basedir.path" . | quote }}
+  DOLPHINSCHEDULER_DATA_BASEDIR_PATH: {{ .Values.common.configmap.DOLPHINSCHEDULER_DATA_BASEDIR_PATH | quote }}
   RESOURCE_STORAGE_TYPE: {{ .Values.common.configmap.RESOURCE_STORAGE_TYPE | quote }}
   RESOURCE_UPLOAD_PATH: {{ .Values.common.configmap.RESOURCE_UPLOAD_PATH | quote }}
   FS_DEFAULT_FS: {{ .Values.common.configmap.FS_DEFAULT_FS | quote }}

docker/kubernetes/dolphinscheduler/templates/deployment-dolphinscheduler-api.yaml

@@ -206,6 +206,10 @@ spec:
           volumeMounts:
             - mountPath: "/opt/dolphinscheduler/logs"
               name: {{ include "dolphinscheduler.fullname" . }}-api
+            {{- if .Values.common.fsFileResourcePersistence.enabled }}
+            - mountPath: {{ .Values.common.configmap.RESOURCE_UPLOAD_PATH | quote }}
+              name: {{ include "dolphinscheduler.fullname" . }}-fs-file
+            {{- end }}
       volumes:
         - name: {{ include "dolphinscheduler.fullname" . }}-api
           {{- if .Values.api.persistentVolumeClaim.enabled }}
@@ -213,4 +217,9 @@ spec:
             claimName: {{ include "dolphinscheduler.fullname" . }}-api
           {{- else }}
           emptyDir: {}
-          {{- end }}
+          {{- end }}
+        {{- if .Values.common.fsFileResourcePersistence.enabled }}
+        - name: {{ include "dolphinscheduler.fullname" . }}-fs-file
+          persistentVolumeClaim:
+            claimName: {{ include "dolphinscheduler.fullname" . }}-fs-file
+        {{- end }}

docker/kubernetes/dolphinscheduler/templates/pvc-dolphinscheduler-fs-file.yaml

@@ -0,0 +1,37 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+{{- if .Values.common.fsFileResourcePersistence.enabled }}
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: {{ include "dolphinscheduler.fullname" . }}-fs-file
+  labels:
+    app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-fs-file
+    app.kubernetes.io/instance: {{ .Release.Name }}
+    app.kubernetes.io/managed-by: {{ .Release.Service }}
+  annotations:
+    "helm.sh/resource-policy": keep
+spec:
+  accessModes:
+  {{- range .Values.common.fsFileResourcePersistence.accessModes }}
+    - {{ . | quote }}
+  {{- end }}
+  storageClassName: {{ .Values.common.fsFileResourcePersistence.storageClassName | quote }}
+  resources:
+    requests:
+      storage: {{ .Values.common.fsFileResourcePersistence.storage | quote }}
+{{- end }}

docker/kubernetes/dolphinscheduler/templates/statefulset-dolphinscheduler-master.yaml

@@ -247,4 +247,4 @@ spec:
         resources:
           requests:
             storage: {{ .Values.master.persistentVolumeClaim.storage | quote }}
-  {{- end }}
+  {{- end }}

docker/kubernetes/dolphinscheduler/templates/statefulset-dolphinscheduler-worker.yaml

@@ -324,13 +324,17 @@ spec:
             failureThreshold: {{ .Values.worker.readinessProbe.failureThreshold }}
           {{- end }}
           volumeMounts:
-            - mountPath: {{ include "dolphinscheduler.data.basedir.path" . | quote }}
+            - mountPath: {{ .Values.common.configmap.DOLPHINSCHEDULER_DATA_BASEDIR_PATH | quote }}
               name: {{ include "dolphinscheduler.fullname" . }}-worker-data
             - mountPath: "/opt/dolphinscheduler/logs"
               name: {{ include "dolphinscheduler.fullname" . }}-worker-logs
             - mountPath: "/opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh"
               subPath: "dolphinscheduler_env.sh"
               name: {{ include "dolphinscheduler.fullname" . }}-common-env
+            {{- if .Values.common.fsFileResourcePersistence.enabled }}
+            - mountPath: {{ .Values.common.configmap.RESOURCE_UPLOAD_PATH | quote }}
+              name: {{ include "dolphinscheduler.fullname" . }}-fs-file
+            {{- end }}
       volumes:
         - name: {{ include "dolphinscheduler.fullname" . }}-worker-data
           {{- if .Values.worker.persistentVolumeClaim.dataPersistentVolume.enabled }}
@@ -353,6 +357,11 @@ spec:
             items:
               - key: DOLPHINSCHEDULER_ENV
                 path: dolphinscheduler_env.sh
+        {{- if .Values.common.fsFileResourcePersistence.enabled }}
+        - name: {{ include "dolphinscheduler.fullname" . }}-fs-file
+          persistentVolumeClaim:
+            claimName: {{ include "dolphinscheduler.fullname" . }}-fs-file
+        {{- end }}
   {{- if .Values.worker.persistentVolumeClaim.enabled }}
   volumeClaimTemplates:
     {{- if .Values.worker.persistentVolumeClaim.dataPersistentVolume.enabled }}
@@ -389,4 +398,4 @@ spec:
           requests:
             storage: {{ .Values.worker.persistentVolumeClaim.logsPersistentVolume.storage | quote }}
     {{- end }}
-  {{- end }}
+  {{- end }}

docker/kubernetes/dolphinscheduler/values.yaml

@@ -94,6 +94,14 @@ common:
     FS_S3A_ENDPOINT: "s3.xxx.amazonaws.com"
     FS_S3A_ACCESS_KEY: "xxxxxxx"
     FS_S3A_SECRET_KEY: "xxxxxxx"
+  ## If RESOURCE_STORAGE_TYPE is HDFS and FS_DEFAULT_FS is file:///, fsFileResourcePersistence should be enabled for resource storage
+  fsFileResourcePersistence:
+    enabled: false
+    accessModes:
+    - "ReadWriteMany"
+    ## storageClassName must support the access mode: ReadWriteMany
+    storageClassName: "-"
+    storage: "20Gi"
 
 master:
   ## PodManagementPolicy controls how pods are created during initial scale up, when replacing pods on nodes, or when scaling down.