Merge pull request #5002 from chengshiwen/improve-docker-compatibility

[Improvement-4837][Docker] Support docker compose file 3.1+ version and Optimize k8s template

docker/build/README.md

@@ -11,6 +11,11 @@ Official Website: https://dolphinscheduler.apache.org
 [![EN doc](https://img.shields.io/badge/document-English-blue.svg)](README.md)
 [![CN doc](https://img.shields.io/badge/文档-中文版-blue.svg)](README_zh_CN.md)
 
+## Prerequisites
+
+- [Docker](https://docs.docker.com/engine/) 1.13.1+
+- [Docker Compose](https://docs.docker.com/compose/) 1.11.0+
+
 ## How to use this docker image
 
 #### You can start a dolphinscheduler by docker-compose (recommended)
@@ -27,6 +32,8 @@ Access the Web UI: http://192.168.xx.xx:12345/dolphinscheduler
 
 The default username is `admin` and the default password is `dolphinscheduler123`
 
+> **Tip**: For quick start in docker, you can create a tenant named `ds` and associate the user `admin` with the tenant `ds`
+
 #### Or via Environment Variables **`DATABASE_HOST`** **`DATABASE_PORT`** **`DATABASE_DATABASE`** **`ZOOKEEPER_QUORUM`**
 
 You can specify **existing postgres and zookeeper service**. Example:

docker/build/README_zh_CN.md

@@ -11,6 +11,11 @@ Official Website: https://dolphinscheduler.apache.org
 [![EN doc](https://img.shields.io/badge/document-English-blue.svg)](README.md)
 [![CN doc](https://img.shields.io/badge/文档-中文版-blue.svg)](README_zh_CN.md)
 
+## 先决条件
+
+- [Docker](https://docs.docker.com/engine/) 1.13.1+
+- [Docker Compose](https://docs.docker.com/compose/) 1.11.0+
+
 ## 如何使用docker镜像
 
 #### 以 docker-compose 的方式启动dolphinscheduler(推荐)
@@ -27,6 +32,8 @@ $ docker-compose -f ./docker/docker-swarm/docker-compose.yml up -d
 
 默认的用户是`admin`，默认的密码是`dolphinscheduler123`
 
+> **提示**: 为了在docker中快速开始，你可以创建一个名为`ds`的租户，并将这个租户`ds`关联到用户`admin`
+
 #### 或者通过环境变量 **`DATABASE_HOST`** **`DATABASE_PORT`** **`ZOOKEEPER_QUORUM`** 使用已存在的服务
 
 你可以指定已经存在的 **`Postgres`** 和 **`Zookeeper`** 服务. 如下:

docker/docker-swarm/docker-compose.yml

@@ -14,7 +14,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-version: "3.4"
+version: "3.1"
 
 services:
 
@@ -76,7 +76,6 @@ services:
       interval: 30s
       timeout: 5s
       retries: 3
-      start_period: 30s
     depends_on:
     - dolphinscheduler-postgresql
     - dolphinscheduler-zookeeper
@@ -110,7 +109,6 @@ services:
       interval: 30s
       timeout: 5s
       retries: 3
-      start_period: 30s
     depends_on:
     - dolphinscheduler-postgresql
     volumes:
@@ -150,7 +148,6 @@ services:
       interval: 30s
       timeout: 5s
       retries: 3
-      start_period: 30s
     depends_on:
     - dolphinscheduler-postgresql
     - dolphinscheduler-zookeeper
@@ -204,7 +201,6 @@ services:
       interval: 30s
       timeout: 5s
       retries: 3
-      start_period: 30s
     depends_on:
     - dolphinscheduler-postgresql
     - dolphinscheduler-zookeeper

docker/docker-swarm/docker-stack.yml

@@ -14,7 +14,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-version: "3.4"
+version: "3.1"
 
 services:
 
@@ -76,7 +76,6 @@ services:
       interval: 30s
       timeout: 5s
       retries: 3
-      start_period: 30s
     volumes:
     - dolphinscheduler-logs:/opt/dolphinscheduler/logs
     networks:
@@ -107,7 +106,6 @@ services:
       interval: 30s
       timeout: 5s
       retries: 3
-      start_period: 30s
     volumes:
     - dolphinscheduler-logs:/opt/dolphinscheduler/logs
     networks:
@@ -146,7 +144,6 @@ services:
       interval: 30s
       timeout: 5s
       retries: 3
-      start_period: 30s
     volumes:
     - dolphinscheduler-logs:/opt/dolphinscheduler/logs
     networks:
@@ -198,7 +195,6 @@ services:
       interval: 30s
       timeout: 5s
       retries: 3
-      start_period: 30s
     volumes:
     - dolphinscheduler-worker-data:/tmp/dolphinscheduler
     - dolphinscheduler-logs:/opt/dolphinscheduler/logs

docker/kubernetes/dolphinscheduler/README.md

@@ -54,6 +54,8 @@ And then access the web: http://192.168.xx.xx:12345/dolphinscheduler
 
 The default username is `admin` and the default password is `dolphinscheduler123`
 
+> **Tip**: For quick start in docker, you can create a tenant named `ds` and associate the user `admin` with the tenant `ds`
+
 ## Uninstalling the Chart
 
 To uninstall/delete the `dolphinscheduler` deployment:
@@ -78,14 +80,12 @@ The Configuration file is `values.yaml`, and the following tables lists the conf
 
 | Parameter                                                                         | Description                                                                                                                    | Default                                               |
 | --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------- |
-| `nameOverride`                                                                    | String to partially override common.names.fullname                                                                             | `nil`                                                 |
-| `fullnameOverride`                                                                | String to fully override common.names.fullname                                                                                 | `nil`                                                 |
 | `timezone`                                                                        | World time and date for cities in all time zones                                                                               | `Asia/Shanghai`                                       |
-| `image.registry`                                                                  | Docker image registry for the DolphinScheduler                                                                                 | `docker.io`                                           |
+|                                                                                   |                                                                                                                                |                                                       |
 | `image.repository`                                                                | Docker image repository for the DolphinScheduler                                                                               | `apache/dolphinscheduler`                             |
 | `image.tag`                                                                       | Docker image version for the DolphinScheduler                                                                                  | `latest`                                              |
 | `image.pullPolicy`                                                                | Image pull policy. One of Always, Never, IfNotPresent                                                                          | `IfNotPresent`                                        |
-| `image.pullSecrets`                                                               | Image pull secrets. An optional list of references to secrets in the same namespace to use for pulling any of the images       | `[]`                                                  |
+| `image.pullSecret`                                                                | Image pull secret. An optional reference to secret in the same namespace to use for pulling any of the images                  | `nil`                                                 |
 |                                                                                   |                                                                                                                                |                                                       |
 | `postgresql.enabled`                                                              | If not exists external PostgreSQL, by default, the DolphinScheduler will use a internal PostgreSQL                             | `true`                                                |
 | `postgresql.postgresqlUsername`                                                   | The username for internal PostgreSQL                                                                                           | `root`                                                |
@@ -283,7 +283,7 @@ docker build -t apache/dolphinscheduler:mysql .
 
 4. Push the docker image `apache/dolphinscheduler:mysql` to a docker registry
 
-5. Modify image `registry` and `repository`, and update `tag` to `mysql` in `values.yaml`
+5. Modify image `repository` and update `tag` to `mysql` in `values.yaml`
 
 6. Modify postgresql `enabled` to `false`
 
@@ -326,7 +326,7 @@ docker build -t apache/dolphinscheduler:mysql-driver .
 
 4. Push the docker image `apache/dolphinscheduler:mysql-driver` to a docker registry
 
-5. Modify image `registry` and `repository`, and update `tag` to `mysql-driver` in `values.yaml`
+5. Modify image `repository` and update `tag` to `mysql-driver` in `values.yaml`
 
 6. Run a DolphinScheduler release in Kubernetes (See **Installing the Chart**)
 
@@ -355,7 +355,7 @@ docker build -t apache/dolphinscheduler:oracle-driver .
 
 4. Push the docker image `apache/dolphinscheduler:oracle-driver` to a docker registry
 
-5. Modify image `registry` and `repository`, and update `tag` to `oracle-driver` in `values.yaml`
+5. Modify image `repository` and update `tag` to `oracle-driver` in `values.yaml`
 
 6. Run a DolphinScheduler release in Kubernetes (See **Installing the Chart**)
 

docker/kubernetes/dolphinscheduler/templates/_helpers.tpl

@@ -16,12 +16,6 @@
 #
 
 {{/* vim: set filetype=mustache: */}}
-{{/*
-Expand the name of the chart.
-*/}}
-{{- define "dolphinscheduler.name" -}}
-{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
-{{- end -}}
 
 {{/*
 Create a default fully qualified app name.
@@ -29,79 +23,14 @@ We truncate at 63 chars because some Kubernetes name fields are limited to this
 If release name contains chart name it will be used as a full name.
 */}}
 {{- define "dolphinscheduler.fullname" -}}
-{{- if .Values.fullnameOverride -}}
-{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
-{{- else -}}
-{{- $name := default .Chart.Name .Values.nameOverride -}}
-{{- if contains $name .Release.Name -}}
 {{- .Release.Name | trunc 63 | trimSuffix "-" -}}
-{{- else -}}
-{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
-{{- end -}}
-{{- end -}}
-{{- end -}}
-
-{{/*
-Create chart name and version as used by the chart label.
-*/}}
-{{- define "dolphinscheduler.chart" -}}
-{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
-{{- end -}}
-
-{{/*
-Common labels
-*/}}
-{{- define "dolphinscheduler.labels" -}}
-helm.sh/chart: {{ include "dolphinscheduler.chart" . }}
-{{ include "dolphinscheduler.selectorLabels" . }}
-{{- if .Chart.AppVersion }}
-app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
-{{- end }}
-app.kubernetes.io/managed-by: {{ .Release.Service }}
 {{- end -}}
 
 {{/*
-Selector labels
+Create a default docker image fullname.
 */}}
-{{- define "dolphinscheduler.selectorLabels" -}}
-app.kubernetes.io/name: {{ include "dolphinscheduler.name" . }}
-app.kubernetes.io/instance: {{ .Release.Name }}
-{{- end -}}
-
-{{/*
-Create the name of the service account to use
-*/}}
-{{- define "dolphinscheduler.serviceAccountName" -}}
-{{- if .Values.serviceAccount.create -}}
-    {{ default (include "dolphinscheduler.fullname" .) .Values.serviceAccount.name }}
-{{- else -}}
-    {{ default "default" .Values.serviceAccount.name }}
-{{- end -}}
-{{- end -}}
-
-{{/*
-Create a default docker image registry.
-We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
-*/}}
-{{- define "dolphinscheduler.image.registry" -}}
-{{- $registry := default "docker.io" .Values.image.registry -}}
-{{- printf "%s" $registry | trunc 63 | trimSuffix "/" -}}
-{{- end -}}
-
-{{/*
-Create a default docker image repository.
-We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
-*/}}
-{{- define "dolphinscheduler.image.repository" -}}
-{{- printf "%s/%s:%s" (include "dolphinscheduler.image.registry" .) .Values.image.repository .Values.image.tag -}}
-{{- end -}}
-
-{{/*
-Create a default image pull secrects.
-We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
-*/}}
-{{- define "dolphinscheduler.image.pullSecrets" -}}
-{{- default nil .Values.image.pullSecrets -}}
+{{- define "dolphinscheduler.image.fullname" -}}
+{{- printf "%s:%s" .Values.image.repository .Values.image.tag -}}
 {{- end -}}
 
 {{/*
@@ -124,9 +53,8 @@ We truncate at 63 chars because some Kubernetes name fields are limited to this
 
 {{/*
 Create a default fully qualified zookkeeper quorum.
-We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
 */}}
 {{- define "dolphinscheduler.zookeeper.quorum" -}}
 {{- $port := default "2181" (.Values.zookeeper.service.port | toString) -}}
-{{- printf "%s:%s" (include "dolphinscheduler.zookeeper.fullname" .) $port | trunc 63 | trimSuffix "-" -}}
+{{- printf "%s:%s" (include "dolphinscheduler.zookeeper.fullname" .) $port -}}
 {{- end -}}

docker/kubernetes/dolphinscheduler/templates/deployment-dolphinscheduler-alert.yaml

@@ -57,13 +57,13 @@ spec:
       {{- if .Values.alert.tolerations }}
       tolerations: {{- toYaml . | nindent 8 }}
       {{- end }}
-      {{- if .Values.image.pullSecrets }}
+      {{- if .Values.image.pullSecret }}
       imagePullSecrets:
-      - name: {{ include "dolphinscheduler.image.pullSecrets" . }}
+      - name: {{ .Values.image.pullSecret }}
       {{- end }}
       containers:
         - name: {{ include "dolphinscheduler.fullname" . }}-alert
-          image: {{ include "dolphinscheduler.image.repository" . | quote }}
+          image: {{ include "dolphinscheduler.image.fullname" . }}
           imagePullPolicy: {{ .Values.image.pullPolicy }}
           args:
             - "alert-server"
@@ -120,8 +120,8 @@ spec:
                   name: {{ template "dolphinscheduler.postgresql.fullname" . }}
                   key: postgresql-password
                   {{- else }}
-                  name: {{ printf "%s-%s" .Release.Name "externaldb"  }}
-                  key: db-password
+                  name: {{ include "dolphinscheduler.fullname" . }}-externaldb
+                  key: database-password
                   {{- end }}
             - name: DATABASE_DATABASE
               {{- if .Values.postgresql.enabled }}

docker/kubernetes/dolphinscheduler/templates/deployment-dolphinscheduler-api.yaml

@@ -57,13 +57,13 @@ spec:
       {{- if .Values.api.tolerations }}
       tolerations: {{- toYaml . | nindent 8 }}
       {{- end }}
-      {{- if .Values.image.pullSecrets }}
+      {{- if .Values.image.pullSecret }}
       imagePullSecrets:
-      - name: {{ include "dolphinscheduler.image.pullSecrets" . }}
+      - name: {{ .Values.image.pullSecret }}
       {{- end }}
       containers:
         - name: {{ include "dolphinscheduler.fullname" . }}-api
-          image: {{ include "dolphinscheduler.image.repository" . | quote }}
+          image: {{ include "dolphinscheduler.image.fullname" . }}
           imagePullPolicy: {{ .Values.image.pullPolicy }}
           args:
             - "api-server"
@@ -115,8 +115,8 @@ spec:
                   name: {{ template "dolphinscheduler.postgresql.fullname" . }}
                   key: postgresql-password
                   {{- else }}
-                  name: {{ printf "%s-%s" .Release.Name "externaldb"  }}
-                  key: db-password
+                  name: {{ include "dolphinscheduler.fullname" . }}-externaldb
+                  key: database-password
                   {{- end }}
             - name: DATABASE_DATABASE
               {{- if .Values.postgresql.enabled }}
@@ -172,7 +172,7 @@ spec:
               valueFrom:
                 secretKeyRef:
                   key: fs-s3a-secret-key
-                  name: {{ printf "%s-%s" .Release.Name "fs-s3a"  }}
+                  name: {{ include "dolphinscheduler.fullname" . }}-fs-s3a
             {{- end }}
           {{- if .Values.api.resources }}
           resources:

docker/kubernetes/dolphinscheduler/templates/ingress.yaml

@@ -26,7 +26,7 @@ kind: Ingress
 metadata:
   name: {{ include "dolphinscheduler.fullname" . }}
   labels:
-    app.kubernetes.io/name: {{ include "dolphinscheduler.name" . }}
+    app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}
     app.kubernetes.io/instance: {{ .Release.Name }}
     app.kubernetes.io/managed-by: {{ .Release.Service }}
 spec:

docker/kubernetes/dolphinscheduler/templates/secret-external-database.yaml

@@ -18,12 +18,12 @@
 apiVersion: v1
 kind: Secret
 metadata:
-  name: {{ printf "%s-%s" .Release.Name "externaldb"  }}
+  name: {{ include "dolphinscheduler.fullname" . }}-externaldb
   labels:
-    app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-postgresql
+    app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-externaldb
     app.kubernetes.io/instance: {{ .Release.Name }}
     app.kubernetes.io/managed-by: {{ .Release.Service }}
 type: Opaque
 data:
-  db-password: {{ .Values.externalDatabase.password | b64enc | quote }}
+  database-password: {{ .Values.externalDatabase.password | b64enc | quote }}
 {{- end }}

docker/kubernetes/dolphinscheduler/templates/secret-external-fs-s3a.yaml

@@ -18,7 +18,7 @@
 apiVersion: v1
 kind: Secret
 metadata:
-  name: {{ printf "%s-%s" .Release.Name "fs-s3a"  }}
+  name: {{ include "dolphinscheduler.fullname" . }}-fs-s3a
   labels:
     app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-fs-s3a
     app.kubernetes.io/instance: {{ .Release.Name }}

docker/kubernetes/dolphinscheduler/templates/statefulset-dolphinscheduler-master.yaml

@@ -54,13 +54,13 @@ spec:
       {{- if .Values.master.tolerations }}
       tolerations: {{- toYaml . | nindent 8 }}
       {{- end }}
-      {{- if .Values.image.pullSecrets }}
+      {{- if .Values.image.pullSecret }}
       imagePullSecrets:
-      - name: {{ include "dolphinscheduler.image.pullSecrets" . }}
+      - name: {{ .Values.image.pullSecret }}
       {{- end }}
       containers:
         - name: {{ include "dolphinscheduler.fullname" . }}-master
-          image: {{ include "dolphinscheduler.image.repository" . | quote }}
+          image: {{ include "dolphinscheduler.image.fullname" . }}
           imagePullPolicy: {{ .Values.image.pullPolicy }}
           args:
             - "master-server"
@@ -157,8 +157,8 @@ spec:
                   name: {{ template "dolphinscheduler.postgresql.fullname" . }}
                   key: postgresql-password
                   {{- else }}
-                  name: {{ printf "%s-%s" .Release.Name "externaldb"  }}
-                  key: db-password
+                  name: {{ include "dolphinscheduler.fullname" . }}-externaldb
+                  key: database-password
                   {{- end }}
             - name: DATABASE_DATABASE
               {{- if .Values.postgresql.enabled }}

docker/kubernetes/dolphinscheduler/templates/statefulset-dolphinscheduler-worker.yaml

@@ -54,13 +54,13 @@ spec:
       {{- if .Values.worker.tolerations }}
       tolerations: {{- toYaml . | nindent 8 }}
       {{- end }}
-      {{- if .Values.image.pullSecrets }}
+      {{- if .Values.image.pullSecret }}
       imagePullSecrets:
-      - name: {{ include "dolphinscheduler.image.pullSecrets" . }}
+      - name: {{ .Values.image.pullSecret }}
       {{- end }}
       containers:
         - name: {{ include "dolphinscheduler.fullname" . }}-worker
-          image: {{ include "dolphinscheduler.image.repository" . | quote }}
+          image: {{ include "dolphinscheduler.image.fullname" . }}
           imagePullPolicy: {{ .Values.image.pullPolicy }}
           args:
             - "worker-server"
@@ -156,8 +156,8 @@ spec:
                   name: {{ template "dolphinscheduler.postgresql.fullname" . }}
                   key: postgresql-password
                   {{- else }}
-                  name: {{ printf "%s-%s" .Release.Name "externaldb"  }}
-                  key: db-password
+                  name: {{ include "dolphinscheduler.fullname" . }}-externaldb
+                  key: database-password
                   {{- end }}
             - name: DATABASE_DATABASE
               {{- if .Values.postgresql.enabled }}
@@ -213,7 +213,7 @@ spec:
               valueFrom:
                 secretKeyRef:
                   key: fs-s3a-secret-key
-                  name: {{ printf "%s-%s" .Release.Name "fs-s3a"  }}
+                  name: {{ include "dolphinscheduler.fullname" . }}-fs-s3a
             {{- end }}
           {{- if .Values.worker.resources }}
           resources:

docker/kubernetes/dolphinscheduler/templates/svc-dolphinscheduler-master-headless.yaml

@@ -20,7 +20,7 @@ metadata:
   name: {{ include "dolphinscheduler.fullname" . }}-master-headless
   labels:
     app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-master-headless
-    app.kubernetes.io/instance: {{ .Release.Name }}-master-headless
+    app.kubernetes.io/instance: {{ .Release.Name }}
     app.kubernetes.io/managed-by: {{ .Release.Service }}
 spec:
   clusterIP: "None"

docker/kubernetes/dolphinscheduler/templates/svc-dolphinscheduler-worker-headless.yaml

@@ -20,7 +20,7 @@ metadata:
   name: {{ include "dolphinscheduler.fullname" . }}-worker-headless
   labels:
     app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-worker-headless
-    app.kubernetes.io/instance: {{ .Release.Name }}-worker-headless
+    app.kubernetes.io/instance: {{ .Release.Name }}
     app.kubernetes.io/managed-by: {{ .Release.Service }}
 spec:
   clusterIP: "None"

docker/kubernetes/dolphinscheduler/values.yaml

@@ -19,17 +19,13 @@
 # This is a YAML-formatted file.
 # Declare variables to be passed into your templates.
 
-nameOverride: ""
-fullnameOverride: ""
-
 timezone: "Asia/Shanghai"
 
 image:
-  registry: "docker.io"
   repository: "apache/dolphinscheduler"
   tag: "latest"
   pullPolicy: "IfNotPresent"
-  pullSecrets: []
+  pullSecret: ""
 
 ## If not exists external database, by default, Dolphinscheduler's database will use it.
 postgresql:
@@ -52,7 +48,6 @@ externalDatabase:
   username: "root"
   password: "root"
   database: "dolphinscheduler"
-  ## multi params should join with & char
   params: "characterEncoding=utf8"
 
 ## If not exists external zookeeper, by default, Dolphinscheduler's zookeeper will use it.