Browse Source

Merge pull request #5002 from chengshiwen/improve-docker-compatibility

[Improvement-4837][Docker] Support docker compose file 3.1+ version and Optimize k8s template
pull/3/MERGE
xingchun-chen 4 years ago committed by GitHub
parent
commit
0ac0f31548
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
  1. 7
      docker/build/README.md
  2. 7
      docker/build/README_zh_CN.md
  3. 6
      docker/docker-swarm/docker-compose.yml
  4. 6
      docker/docker-swarm/docker-stack.yml
  5. 14
      docker/kubernetes/dolphinscheduler/README.md
  6. 80
      docker/kubernetes/dolphinscheduler/templates/_helpers.tpl
  7. 10
      docker/kubernetes/dolphinscheduler/templates/deployment-dolphinscheduler-alert.yaml
  8. 12
      docker/kubernetes/dolphinscheduler/templates/deployment-dolphinscheduler-api.yaml
  9. 2
      docker/kubernetes/dolphinscheduler/templates/ingress.yaml
  10. 6
      docker/kubernetes/dolphinscheduler/templates/secret-external-database.yaml
  11. 2
      docker/kubernetes/dolphinscheduler/templates/secret-external-fs-s3a.yaml
  12. 10
      docker/kubernetes/dolphinscheduler/templates/statefulset-dolphinscheduler-master.yaml
  13. 12
      docker/kubernetes/dolphinscheduler/templates/statefulset-dolphinscheduler-worker.yaml
  14. 2
      docker/kubernetes/dolphinscheduler/templates/svc-dolphinscheduler-master-headless.yaml
  15. 2
      docker/kubernetes/dolphinscheduler/templates/svc-dolphinscheduler-worker-headless.yaml
  16. 7
      docker/kubernetes/dolphinscheduler/values.yaml

7
docker/build/README.md

@ -11,6 +11,11 @@ Official Website: https://dolphinscheduler.apache.org
[![EN doc](https://img.shields.io/badge/document-English-blue.svg)](README.md)
[![CN doc](https://img.shields.io/badge/文档-中文版-blue.svg)](README_zh_CN.md)
## Prerequisites
- [Docker](https://docs.docker.com/engine/) 1.13.1+
- [Docker Compose](https://docs.docker.com/compose/) 1.11.0+
## How to use this docker image
#### You can start a dolphinscheduler by docker-compose (recommended)
@ -27,6 +32,8 @@ Access the Web UI: http://192.168.xx.xx:12345/dolphinscheduler
The default username is `admin` and the default password is `dolphinscheduler123`
> **Tip**: For quick start in docker, you can create a tenant named `ds` and associate the user `admin` with the tenant `ds`
#### Or via Environment Variables **`DATABASE_HOST`** **`DATABASE_PORT`** **`DATABASE_DATABASE`** **`ZOOKEEPER_QUORUM`**
You can specify **existing postgres and zookeeper service**. Example:

7
docker/build/README_zh_CN.md

@ -11,6 +11,11 @@ Official Website: https://dolphinscheduler.apache.org
[![EN doc](https://img.shields.io/badge/document-English-blue.svg)](README.md)
[![CN doc](https://img.shields.io/badge/文档-中文版-blue.svg)](README_zh_CN.md)
## 先决条件
- [Docker](https://docs.docker.com/engine/) 1.13.1+
- [Docker Compose](https://docs.docker.com/compose/) 1.11.0+
## 如何使用docker镜像
#### 以 docker-compose 的方式启动dolphinscheduler(推荐)
@ -27,6 +32,8 @@ $ docker-compose -f ./docker/docker-swarm/docker-compose.yml up -d
默认的用户是`admin`,默认的密码是`dolphinscheduler123`
> **提示**: 为了在docker中快速开始,你可以创建一个名为`ds`的租户,并将这个租户`ds`关联到用户`admin`
#### 或者通过环境变量 **`DATABASE_HOST`** **`DATABASE_PORT`** **`ZOOKEEPER_QUORUM`** 使用已存在的服务
你可以指定已经存在的 **`Postgres`** 和 **`Zookeeper`** 服务. 如下:

6
docker/docker-swarm/docker-compose.yml

@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
version: "3.4"
version: "3.1"
services:
@ -76,7 +76,6 @@ services:
interval: 30s
timeout: 5s
retries: 3
start_period: 30s
depends_on:
- dolphinscheduler-postgresql
- dolphinscheduler-zookeeper
@ -110,7 +109,6 @@ services:
interval: 30s
timeout: 5s
retries: 3
start_period: 30s
depends_on:
- dolphinscheduler-postgresql
volumes:
@ -150,7 +148,6 @@ services:
interval: 30s
timeout: 5s
retries: 3
start_period: 30s
depends_on:
- dolphinscheduler-postgresql
- dolphinscheduler-zookeeper
@ -204,7 +201,6 @@ services:
interval: 30s
timeout: 5s
retries: 3
start_period: 30s
depends_on:
- dolphinscheduler-postgresql
- dolphinscheduler-zookeeper

6
docker/docker-swarm/docker-stack.yml

@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
version: "3.4"
version: "3.1"
services:
@ -76,7 +76,6 @@ services:
interval: 30s
timeout: 5s
retries: 3
start_period: 30s
volumes:
- dolphinscheduler-logs:/opt/dolphinscheduler/logs
networks:
@ -107,7 +106,6 @@ services:
interval: 30s
timeout: 5s
retries: 3
start_period: 30s
volumes:
- dolphinscheduler-logs:/opt/dolphinscheduler/logs
networks:
@ -146,7 +144,6 @@ services:
interval: 30s
timeout: 5s
retries: 3
start_period: 30s
volumes:
- dolphinscheduler-logs:/opt/dolphinscheduler/logs
networks:
@ -198,7 +195,6 @@ services:
interval: 30s
timeout: 5s
retries: 3
start_period: 30s
volumes:
- dolphinscheduler-worker-data:/tmp/dolphinscheduler
- dolphinscheduler-logs:/opt/dolphinscheduler/logs

14
docker/kubernetes/dolphinscheduler/README.md

@ -54,6 +54,8 @@ And then access the web: http://192.168.xx.xx:12345/dolphinscheduler
The default username is `admin` and the default password is `dolphinscheduler123`
> **Tip**: For quick start in docker, you can create a tenant named `ds` and associate the user `admin` with the tenant `ds`
## Uninstalling the Chart
To uninstall/delete the `dolphinscheduler` deployment:
@ -78,14 +80,12 @@ The Configuration file is `values.yaml`, and the following tables lists the conf
| Parameter | Description | Default |
| --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------- |
| `nameOverride` | String to partially override common.names.fullname | `nil` |
| `fullnameOverride` | String to fully override common.names.fullname | `nil` |
| `timezone` | World time and date for cities in all time zones | `Asia/Shanghai` |
| `image.registry` | Docker image registry for the DolphinScheduler | `docker.io` |
| | | |
| `image.repository` | Docker image repository for the DolphinScheduler | `apache/dolphinscheduler` |
| `image.tag` | Docker image version for the DolphinScheduler | `latest` |
| `image.pullPolicy` | Image pull policy. One of Always, Never, IfNotPresent | `IfNotPresent` |
| `image.pullSecrets` | Image pull secrets. An optional list of references to secrets in the same namespace to use for pulling any of the images | `[]` |
| `image.pullSecret` | Image pull secret. An optional reference to secret in the same namespace to use for pulling any of the images | `nil` |
| | | |
| `postgresql.enabled` | If not exists external PostgreSQL, by default, the DolphinScheduler will use a internal PostgreSQL | `true` |
| `postgresql.postgresqlUsername` | The username for internal PostgreSQL | `root` |
@ -283,7 +283,7 @@ docker build -t apache/dolphinscheduler:mysql .
4. Push the docker image `apache/dolphinscheduler:mysql` to a docker registry
5. Modify image `registry` and `repository`, and update `tag` to `mysql` in `values.yaml`
5. Modify image `repository` and update `tag` to `mysql` in `values.yaml`
6. Modify postgresql `enabled` to `false`
@ -326,7 +326,7 @@ docker build -t apache/dolphinscheduler:mysql-driver .
4. Push the docker image `apache/dolphinscheduler:mysql-driver` to a docker registry
5. Modify image `registry` and `repository`, and update `tag` to `mysql-driver` in `values.yaml`
5. Modify image `repository` and update `tag` to `mysql-driver` in `values.yaml`
6. Run a DolphinScheduler release in Kubernetes (See **Installing the Chart**)
@ -355,7 +355,7 @@ docker build -t apache/dolphinscheduler:oracle-driver .
4. Push the docker image `apache/dolphinscheduler:oracle-driver` to a docker registry
5. Modify image `registry` and `repository`, and update `tag` to `oracle-driver` in `values.yaml`
5. Modify image `repository` and update `tag` to `oracle-driver` in `values.yaml`
6. Run a DolphinScheduler release in Kubernetes (See **Installing the Chart**)

80
docker/kubernetes/dolphinscheduler/templates/_helpers.tpl

@ -16,12 +16,6 @@
#
{{/* vim: set filetype=mustache: */}}
{{/*
Expand the name of the chart.
*/}}
{{- define "dolphinscheduler.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create a default fully qualified app name.
@ -29,79 +23,14 @@ We truncate at 63 chars because some Kubernetes name fields are limited to this
If release name contains chart name it will be used as a full name.
*/}}
{{- define "dolphinscheduler.fullname" -}}
{{- if .Values.fullnameOverride -}}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- if contains $name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "dolphinscheduler.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Common labels
*/}}
{{- define "dolphinscheduler.labels" -}}
helm.sh/chart: {{ include "dolphinscheduler.chart" . }}
{{ include "dolphinscheduler.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end -}}
{{/*
Selector labels
Create a default docker image fullname.
*/}}
{{- define "dolphinscheduler.selectorLabels" -}}
app.kubernetes.io/name: {{ include "dolphinscheduler.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end -}}
{{/*
Create the name of the service account to use
*/}}
{{- define "dolphinscheduler.serviceAccountName" -}}
{{- if .Values.serviceAccount.create -}}
{{ default (include "dolphinscheduler.fullname" .) .Values.serviceAccount.name }}
{{- else -}}
{{ default "default" .Values.serviceAccount.name }}
{{- end -}}
{{- end -}}
{{/*
Create a default docker image registry.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "dolphinscheduler.image.registry" -}}
{{- $registry := default "docker.io" .Values.image.registry -}}
{{- printf "%s" $registry | trunc 63 | trimSuffix "/" -}}
{{- end -}}
{{/*
Create a default docker image repository.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "dolphinscheduler.image.repository" -}}
{{- printf "%s/%s:%s" (include "dolphinscheduler.image.registry" .) .Values.image.repository .Values.image.tag -}}
{{- end -}}
{{/*
Create a default image pull secrects.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "dolphinscheduler.image.pullSecrets" -}}
{{- default nil .Values.image.pullSecrets -}}
{{- define "dolphinscheduler.image.fullname" -}}
{{- printf "%s:%s" .Values.image.repository .Values.image.tag -}}
{{- end -}}
{{/*
@ -124,9 +53,8 @@ We truncate at 63 chars because some Kubernetes name fields are limited to this
{{/*
Create a default fully qualified zookkeeper quorum.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "dolphinscheduler.zookeeper.quorum" -}}
{{- $port := default "2181" (.Values.zookeeper.service.port | toString) -}}
{{- printf "%s:%s" (include "dolphinscheduler.zookeeper.fullname" .) $port | trunc 63 | trimSuffix "-" -}}
{{- printf "%s:%s" (include "dolphinscheduler.zookeeper.fullname" .) $port -}}
{{- end -}}

10
docker/kubernetes/dolphinscheduler/templates/deployment-dolphinscheduler-alert.yaml

@ -57,13 +57,13 @@ spec:
{{- if .Values.alert.tolerations }}
tolerations: {{- toYaml . | nindent 8 }}
{{- end }}
{{- if .Values.image.pullSecrets }}
{{- if .Values.image.pullSecret }}
imagePullSecrets:
- name: {{ include "dolphinscheduler.image.pullSecrets" . }}
- name: {{ .Values.image.pullSecret }}
{{- end }}
containers:
- name: {{ include "dolphinscheduler.fullname" . }}-alert
image: {{ include "dolphinscheduler.image.repository" . | quote }}
image: {{ include "dolphinscheduler.image.fullname" . }}
imagePullPolicy: {{ .Values.image.pullPolicy }}
args:
- "alert-server"
@ -120,8 +120,8 @@ spec:
name: {{ template "dolphinscheduler.postgresql.fullname" . }}
key: postgresql-password
{{- else }}
name: {{ printf "%s-%s" .Release.Name "externaldb" }}
key: db-password
name: {{ include "dolphinscheduler.fullname" . }}-externaldb
key: database-password
{{- end }}
- name: DATABASE_DATABASE
{{- if .Values.postgresql.enabled }}

12
docker/kubernetes/dolphinscheduler/templates/deployment-dolphinscheduler-api.yaml

@ -57,13 +57,13 @@ spec:
{{- if .Values.api.tolerations }}
tolerations: {{- toYaml . | nindent 8 }}
{{- end }}
{{- if .Values.image.pullSecrets }}
{{- if .Values.image.pullSecret }}
imagePullSecrets:
- name: {{ include "dolphinscheduler.image.pullSecrets" . }}
- name: {{ .Values.image.pullSecret }}
{{- end }}
containers:
- name: {{ include "dolphinscheduler.fullname" . }}-api
image: {{ include "dolphinscheduler.image.repository" . | quote }}
image: {{ include "dolphinscheduler.image.fullname" . }}
imagePullPolicy: {{ .Values.image.pullPolicy }}
args:
- "api-server"
@ -115,8 +115,8 @@ spec:
name: {{ template "dolphinscheduler.postgresql.fullname" . }}
key: postgresql-password
{{- else }}
name: {{ printf "%s-%s" .Release.Name "externaldb" }}
key: db-password
name: {{ include "dolphinscheduler.fullname" . }}-externaldb
key: database-password
{{- end }}
- name: DATABASE_DATABASE
{{- if .Values.postgresql.enabled }}
@ -172,7 +172,7 @@ spec:
valueFrom:
secretKeyRef:
key: fs-s3a-secret-key
name: {{ printf "%s-%s" .Release.Name "fs-s3a" }}
name: {{ include "dolphinscheduler.fullname" . }}-fs-s3a
{{- end }}
{{- if .Values.api.resources }}
resources:

2
docker/kubernetes/dolphinscheduler/templates/ingress.yaml

@ -26,7 +26,7 @@ kind: Ingress
metadata:
name: {{ include "dolphinscheduler.fullname" . }}
labels:
app.kubernetes.io/name: {{ include "dolphinscheduler.name" . }}
app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
spec:

6
docker/kubernetes/dolphinscheduler/templates/secret-external-postgresql.yaml → docker/kubernetes/dolphinscheduler/templates/secret-external-database.yaml

@ -18,12 +18,12 @@
apiVersion: v1
kind: Secret
metadata:
name: {{ printf "%s-%s" .Release.Name "externaldb" }}
name: {{ include "dolphinscheduler.fullname" . }}-externaldb
labels:
app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-postgresql
app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-externaldb
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
type: Opaque
data:
db-password: {{ .Values.externalDatabase.password | b64enc | quote }}
database-password: {{ .Values.externalDatabase.password | b64enc | quote }}
{{- end }}

2
docker/kubernetes/dolphinscheduler/templates/secret-external-fs-s3a.yaml

@ -18,7 +18,7 @@
apiVersion: v1
kind: Secret
metadata:
name: {{ printf "%s-%s" .Release.Name "fs-s3a" }}
name: {{ include "dolphinscheduler.fullname" . }}-fs-s3a
labels:
app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-fs-s3a
app.kubernetes.io/instance: {{ .Release.Name }}

10
docker/kubernetes/dolphinscheduler/templates/statefulset-dolphinscheduler-master.yaml

@ -54,13 +54,13 @@ spec:
{{- if .Values.master.tolerations }}
tolerations: {{- toYaml . | nindent 8 }}
{{- end }}
{{- if .Values.image.pullSecrets }}
{{- if .Values.image.pullSecret }}
imagePullSecrets:
- name: {{ include "dolphinscheduler.image.pullSecrets" . }}
- name: {{ .Values.image.pullSecret }}
{{- end }}
containers:
- name: {{ include "dolphinscheduler.fullname" . }}-master
image: {{ include "dolphinscheduler.image.repository" . | quote }}
image: {{ include "dolphinscheduler.image.fullname" . }}
imagePullPolicy: {{ .Values.image.pullPolicy }}
args:
- "master-server"
@ -157,8 +157,8 @@ spec:
name: {{ template "dolphinscheduler.postgresql.fullname" . }}
key: postgresql-password
{{- else }}
name: {{ printf "%s-%s" .Release.Name "externaldb" }}
key: db-password
name: {{ include "dolphinscheduler.fullname" . }}-externaldb
key: database-password
{{- end }}
- name: DATABASE_DATABASE
{{- if .Values.postgresql.enabled }}

12
docker/kubernetes/dolphinscheduler/templates/statefulset-dolphinscheduler-worker.yaml

@ -54,13 +54,13 @@ spec:
{{- if .Values.worker.tolerations }}
tolerations: {{- toYaml . | nindent 8 }}
{{- end }}
{{- if .Values.image.pullSecrets }}
{{- if .Values.image.pullSecret }}
imagePullSecrets:
- name: {{ include "dolphinscheduler.image.pullSecrets" . }}
- name: {{ .Values.image.pullSecret }}
{{- end }}
containers:
- name: {{ include "dolphinscheduler.fullname" . }}-worker
image: {{ include "dolphinscheduler.image.repository" . | quote }}
image: {{ include "dolphinscheduler.image.fullname" . }}
imagePullPolicy: {{ .Values.image.pullPolicy }}
args:
- "worker-server"
@ -156,8 +156,8 @@ spec:
name: {{ template "dolphinscheduler.postgresql.fullname" . }}
key: postgresql-password
{{- else }}
name: {{ printf "%s-%s" .Release.Name "externaldb" }}
key: db-password
name: {{ include "dolphinscheduler.fullname" . }}-externaldb
key: database-password
{{- end }}
- name: DATABASE_DATABASE
{{- if .Values.postgresql.enabled }}
@ -213,7 +213,7 @@ spec:
valueFrom:
secretKeyRef:
key: fs-s3a-secret-key
name: {{ printf "%s-%s" .Release.Name "fs-s3a" }}
name: {{ include "dolphinscheduler.fullname" . }}-fs-s3a
{{- end }}
{{- if .Values.worker.resources }}
resources:

2
docker/kubernetes/dolphinscheduler/templates/svc-dolphinscheduler-master-headless.yaml

@ -20,7 +20,7 @@ metadata:
name: {{ include "dolphinscheduler.fullname" . }}-master-headless
labels:
app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-master-headless
app.kubernetes.io/instance: {{ .Release.Name }}-master-headless
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
spec:
clusterIP: "None"

2
docker/kubernetes/dolphinscheduler/templates/svc-dolphinscheduler-worker-headless.yaml

@ -20,7 +20,7 @@ metadata:
name: {{ include "dolphinscheduler.fullname" . }}-worker-headless
labels:
app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-worker-headless
app.kubernetes.io/instance: {{ .Release.Name }}-worker-headless
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
spec:
clusterIP: "None"

7
docker/kubernetes/dolphinscheduler/values.yaml

@ -19,17 +19,13 @@
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
nameOverride: ""
fullnameOverride: ""
timezone: "Asia/Shanghai"
image:
registry: "docker.io"
repository: "apache/dolphinscheduler"
tag: "latest"
pullPolicy: "IfNotPresent"
pullSecrets: []
pullSecret: ""
## If not exists external database, by default, Dolphinscheduler's database will use it.
postgresql:
@ -52,7 +48,6 @@ externalDatabase:
username: "root"
password: "root"
database: "dolphinscheduler"
## multi params should join with & char
params: "characterEncoding=utf8"
## If not exists external zookeeper, by default, Dolphinscheduler's zookeeper will use it.

Loading…
Cancel
Save