github
/
DolphinScheduler
mirror of https://gitee.com/dolphinscheduler/DolphinScheduler.git
1
0
Fork 0
Code Issues Releases Wiki Activity
Browse Source

Remove sensitive configs from dolphinscheduler_env.sh (#11917)

3.2.0-release
kezhenxu94 2 years ago committed by GitHub
parent
511149f644
commit
b73a9511a7
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
10 changed files with 30 additions and 71 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Split View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 18
      docs/docs/en/architecture/configuration.md
  2. 7
      docs/docs/en/guide/howto/datasource-setting.md
  3. 4
      docs/docs/en/guide/upgrade/upgrade.md
  4. 20
      docs/docs/zh/architecture/configuration.md
  5. 7
      docs/docs/zh/guide/howto/datasource-setting.md
  6. 2
      docs/docs/zh/guide/howto/general-setting.md
  7. 2
      docs/docs/zh/guide/installation/pseudo-cluster.md
  8. 2
      docs/docs/zh/guide/start/docker.md
  9. 4
      docs/docs/zh/guide/upgrade/upgrade.md
  10. 17
      script/env/dolphinscheduler_env.sh

18
docs/docs/en/architecture/configuration.md
Unescape View File

@ -339,28 +339,12 @@ The default configuration is as follows:
### dolphinscheduler_env.sh [load environment variables configs]
When using shell to commit tasks, DolphinScheduler will export environment variables from `bin/env/dolphinscheduler_env.sh`. The
mainly configuration including `JAVA_HOME`, mata database, registry center, and task configuration.
mainly configuration including `JAVA_HOME` and other environment paths.
```bash
# JAVA_HOME, will use it to start DolphinScheduler server
export JAVA_HOME=${JAVA_HOME:-/opt/soft/java}
# Database related configuration, set database type, username and password
export DATABASE=${DATABASE:-postgresql}
export SPRING_PROFILES_ACTIVE=${DATABASE}
export SPRING_DATASOURCE_URL
export SPRING_DATASOURCE_USERNAME
export SPRING_DATASOURCE_PASSWORD
# DolphinScheduler server related configuration
export SPRING_CACHE_TYPE=${SPRING_CACHE_TYPE:-none}
export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-UTC}
export MASTER_FETCH_COMMAND_NUM=${MASTER_FETCH_COMMAND_NUM:-10}
# Registry center configuration, determines the type and link of the registry center
export REGISTRY_TYPE=${REGISTRY_TYPE:-zookeeper}
export REGISTRY_ZOOKEEPER_CONNECT_STRING=${REGISTRY_ZOOKEEPER_CONNECT_STRING:-localhost:2181}
# Tasks related configurations, need to change the configuration if you use the related tasks.
export HADOOP_HOME=${HADOOP_HOME:-/opt/soft/hadoop}
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/opt/soft/hadoop/etc/hadoop}

7
docs/docs/en/guide/howto/datasource-setting.md
Unescape View File

@ -8,11 +8,12 @@ We here use MySQL as an example to illustrate how to configure an external datab
which is `api-server/libs` and `alert-server/libs` and `master-server/libs` and `worker-server/libs`.
* First of all, follow the instructions in [datasource-setting](datasource-setting.md) `Pseudo-Cluster/Cluster Initialize the Database` section to create and initialize database
* Set the following environment variables in your terminal or modify the `bin/env/dolphinscheduler_env.sh` with your database username and password for `{user}` and `{password}`:
* Set the following environment variables in your terminal with your database address, username and password for `{address}`, `{user}` and `{password}`:
```shell
export DATABASE=mysql
export SPRING_PROFILES_ACTIVE=${DATABASE}
export SPRING_DATASOURCE_URL="jdbc:mysql://{address}/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&useSSL=false"
export SPRING_DATASOURCE_USERNAME={user}
export SPRING_DATASOURCE_PASSWORD={password}
```
@ -72,9 +73,10 @@ echo "host dolphinscheduler {user} {ip} md5" >> $PGDATA/pg_hba.conf
pg_ctl reload
```
Then, modify `./bin/env/dolphinscheduler_env.sh`, change {user} and {password} to what you set in the previous step.
Then, set the database configurations by exporting the following environment variables, change {user} and {password} to what you set in the previous step.
For MySQL:
```shell
# for mysql
export DATABASE=${DATABASE:-mysql}
@ -85,6 +87,7 @@ export SPRING_DATASOURCE_PASSWORD={password}
```
For PostgreSQL:
```shell
# for postgresql
export DATABASE=${DATABASE:-postgresql}

4
docs/docs/en/guide/upgrade/upgrade.md
Unescape View File

@ -23,10 +23,10 @@ Stop all services of dolphinscheduler according to your deployment method. If yo
### Upgrade Database
Change configuration in `./bin/env/dolphinscheduler_env.sh` ({user} and {password} are changed to your database username and password), and then run the upgrade script.
Set the following environment variables ({user} and {password} are changed to your database username and password), and then run the upgrade script.
Using MySQL as an example, change the value if you use other databases. Please manually download the [mysql-connector-java driver jar](https://downloads.MySQL.com/archives/c-j/)
jar package and add it to the `./tools/libs` directory, then change `./bin/ env/dolphinscheduler_env.sh` file
jar package and add it to the `./tools/libs` directory, then export the following environment variables
```shell
export DATABASE=${DATABASE:-mysql}

20
docs/docs/zh/architecture/configuration.md
Unescape View File

@ -147,7 +147,7 @@ export DOLPHINSCHEDULER_OPTS="
|spring.datasource.hikari.leak-detection-threshold|0|连接泄露检测阈值|
|spring.datasource.hikari.initialization-fail-timeout|1|连接池初始化失败timeout|
DolphinScheduler同样可以通过`bin/env/dolphinscheduler_env.sh`进行数据库连接相关的配置。
DolphinScheduler同样可以通过设置环境变量进行数据库连接相关的配置, 将以上小写字母转成大写并把`.`换成`_`作为环境变量名, 设置值即可。
## Zookeeper相关配置
@ -322,28 +322,12 @@ common.properties配置文件目前主要是配置hadoop/s3/yarn相关的配置
## dolphinscheduler_env.sh [环境变量配置]
通过类似shell方式提交任务的的时候，会加载该配置文件中的环境变量到主机中。涉及到的 `JAVA_HOME`、元数据库、注册中心和任务类型配置，其中任务类型主要有: Shell任务、Python任务、Spark任务、Flink任务、Datax任务等等。
通过类似shell方式提交任务的的时候，会加载该配置文件中的环境变量到主机中。涉及到的 `JAVA_HOME` 任务类型的环境配置，其中任务类型主要有: Shell任务、Python任务、Spark任务、Flink任务、Datax任务等等。
```bash
# JAVA_HOME, will use it to start DolphinScheduler server
export JAVA_HOME=${JAVA_HOME:-/opt/soft/java}
# Database related configuration, set database type, username and password
export DATABASE=${DATABASE:-postgresql}
export SPRING_PROFILES_ACTIVE=${DATABASE}
export SPRING_DATASOURCE_URL
export SPRING_DATASOURCE_USERNAME
export SPRING_DATASOURCE_PASSWORD
# DolphinScheduler server related configuration
export SPRING_CACHE_TYPE=${SPRING_CACHE_TYPE:-none}
export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-UTC}
export MASTER_FETCH_COMMAND_NUM=${MASTER_FETCH_COMMAND_NUM:-10}
# Registry center configuration, determines the type and link of the registry center
export REGISTRY_TYPE=${REGISTRY_TYPE:-zookeeper}
export REGISTRY_ZOOKEEPER_CONNECT_STRING=${REGISTRY_ZOOKEEPER_CONNECT_STRING:-localhost:2181}
# Tasks related configurations, need to change the configuration if you use the related tasks.
export HADOOP_HOME=${HADOOP_HOME:-/opt/soft/hadoop}
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/opt/soft/hadoop/etc/hadoop}

7
docs/docs/zh/guide/howto/datasource-setting.md
Unescape View File

@ -7,11 +7,12 @@
> 如果使用 MySQL 需要手动下载 [mysql-connector-java 驱动][mysql] (8.0.16) 并移动到 DolphinScheduler 的每个模块的 libs 目录下，其中包括 `api-server/libs` 和 `alert-server/libs` 和 `master-server/libs` 和 `worker-server/libs`。
* 首先，参照 [数据源配置](datasource-setting.md) `伪分布式/分布式安装初始化数据库` 创建并初始化数据库
* 在你的命令行或者修改 bin/env/dolphinscheduler_env.sh 设定下列环境变量，将 `{user}` 和 `{password}` 改为你数据库的用户名和密码
* 在你的命令行设定下列环境变量，将 `{address}`, `{user}` 和 `{password}` 改为你数据库的地址, 用户名和密码
```shell
export DATABASE=mysql
export SPRING_PROFILES_ACTIVE=${DATABASE}
export SPRING_DATASOURCE_URL="jdbc:mysql://{address}/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&useSSL=false"
export SPRING_DATASOURCE_USERNAME={user}
export SPRING_DATASOURCE_PASSWORD={password}
```
@ -72,9 +73,10 @@ echo "host dolphinscheduler {user} {ip} md5" >> $PGDATA/pg_hba.conf
pg_ctl reload
```
然后修改`./bin/env/dolphinscheduler_env.sh`，将username和password改成你在上一步中设置的用户名{user}和密码{password}
然后设置以下环境变量，将username和password改成你在上一步中设置的用户名{user}和密码{password}
对于 MySQL：
```shell
# for mysql
export DATABASE=${DATABASE:-mysql}
@ -85,6 +87,7 @@ export SPRING_DATASOURCE_PASSWORD={password}
```
对于 PostgreSQL：
```shell
# for postgresql
export DATABASE=${DATABASE:-postgresql}

2
docs/docs/zh/guide/howto/general-setting.md
Unescape View File

@ -16,7 +16,7 @@ DolphinScheduler 支持时区设置。
服务时区
使用脚本 `bin/dolphinshceduler_daemon.sh`启动服务， 服务的默认时区为UTC， 可以在`bin/env/dolphinscheduler_env.sh`中进行修改， 如`export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-Asia/Shanghai}`。<br>
使用脚本 `bin/dolphinshceduler_daemon.sh`启动服务， 服务的默认时区为UTC， 可以在 `application.yaml` 文件中进行修改，或通过环境变量修改, 如`export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-Asia/Shanghai}`。<br>
IDEA 启动服务默认时区为本地时区，可以加jvm参数如`-Duser.timezone=UTC`来修改时区。 时区选择详见[List of tz database time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones)
用户时区

2
docs/docs/zh/guide/installation/pseudo-cluster.md
Unescape View File

@ -100,8 +100,6 @@ deployUser="dolphinscheduler"
* DolphinScheduler 的数据库配置，详细配置方法见[初始化数据库](#初始化数据库)
* 一些任务类型外部依赖路径或库文件，如 `JAVA_HOME` 和 `SPARK_HOME`都是在这里定义的
* 注册中心`zookeeper`
* 服务端相关配置，比如缓存，时区设置等
如果您不使用某些任务类型，您可以忽略任务外部依赖项，但您必须根据您的环境更改 `JAVA_HOME`、注册中心和数据库相关配置。

2
docs/docs/zh/guide/start/docker.md
Unescape View File

@ -126,4 +126,4 @@ $ docker run -d --name dolphinscheduler-alert-server \
## 环境变量
可以通过环境变量来修改 Docker 运行的配置，我们在沿用已有的 PostgreSQL 和 ZooKeeper 服务中就通过环境变量修改了 Docker 的数据库配置和
注册中心配置，关于全部的配置环境可以查看[全部的配置文件](https://github.com/apache/dolphinscheduler/blob/<version>/script/env/dolphinscheduler_env.sh) 了解 <!-- markdown-link-check-disable-line -->
注册中心配置，关于全部的配置环境可以查看对应组件的 application.yaml 文件了解 <!-- markdown-link-check-disable-line -->

4
docs/docs/zh/guide/upgrade/upgrade.md
Unescape View File

@ -22,10 +22,10 @@
### 数据库升级
修改 `./bin/env/dolphinscheduler_env.sh` 中的如下配置（{user}和{password}改成你数据库的用户名和密码），然后运行升级脚本。
设置相关环境变量（{user}和{password}改成你数据库的用户名和密码），然后运行升级脚本。
下面以 MySQL 为例，别的数据库仅需要修改成对应的配置即可。请先手动下载 [mysql-connector-java 驱动 jar](https://downloads.MySQL.com/archives/c-j/)
jar 包 并添加到 `./tools/libs` 目录下，修改 `./bin/env/dolphinscheduler_env.sh` 文件
jar 包 并添加到 `./tools/libs` 目录下，设置以下环境变量
```shell
export DATABASE=${DATABASE:-mysql}

17
script/env/dolphinscheduler_env.sh vendored
Unescape View File

@ -18,21 +18,8 @@
# JAVA_HOME, will use it to start DolphinScheduler server
export JAVA_HOME=${JAVA_HOME:-/opt/soft/java}
# Database related configuration, set database type, username and password
export DATABASE=${DATABASE:-postgresql}
export SPRING_PROFILES_ACTIVE=${DATABASE}
export SPRING_DATASOURCE_URL
export SPRING_DATASOURCE_USERNAME
export SPRING_DATASOURCE_PASSWORD
# DolphinScheduler server related configuration
export SPRING_CACHE_TYPE=${SPRING_CACHE_TYPE:-none}
export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-UTC}
export MASTER_FETCH_COMMAND_NUM=${MASTER_FETCH_COMMAND_NUM:-10}
# Registry center configuration, determines the type and link of the registry center
export REGISTRY_TYPE=${REGISTRY_TYPE:-zookeeper}
export REGISTRY_ZOOKEEPER_CONNECT_STRING=${REGISTRY_ZOOKEEPER_CONNECT_STRING:-localhost:2181}
# Never put sensitive config such as database password here in your production environment,
# this file will be sourced everytime a new task is executed.
# Tasks related configurations, need to change the configuration if you use the related tasks.
export HADOOP_HOME=${HADOOP_HOME:-/opt/soft/hadoop}

Write Preview
Loading…
Cancel
Save