xiangzihao
2 years ago
committed by
GitHub
14 changed files with 202 additions and 175 deletions
@ -1,23 +0,0 @@ |
|||||||
# DataSource |
|
||||||
|
|
||||||
DataSource supports MySQL, PostgreSQL, Hive/Impala, Spark, ClickHouse, Oracle, SQL Server and other DataSources. |
|
||||||
|
|
||||||
- Click bottom `Data Source Center -> Create Data Source` to create a new datasource. |
|
||||||
- Click `Test Connection` to test whether the DataSource can connect successfully (datasource can be saved only if it passes the connection test). |
|
||||||
|
|
||||||
## Using datasource incompatible to Apache LICENSE V2 LICENSE |
|
||||||
|
|
||||||
Some of datasource are native supported to DolphinScheduler while others need users download JDBC driver package manually, |
|
||||||
because those JDBC driver incompatible to Apache LICENSE V2 LICENSE. For this reason we have to release DolphinScheduler's |
|
||||||
distribute package without those packages, even if this will make more complicated for users. Datasource such as MySQL, |
|
||||||
Oracle, SQL Server as the examples, but we have the solution to solve this |
|
||||||
|
|
||||||
### Example |
|
||||||
|
|
||||||
For example, if you want to use MySQL datasource, you need to download the correct JDBC driver from [mysql maven repository](https://repo1.maven.org/maven2/mysql/mysql-connector-java), |
|
||||||
and move it into directory `api-server/libs` and `worker-server/libs`. After that, you could activate MySQL datasource by |
|
||||||
restarting `api-server` and `worker-server`. Mount to container volume in the same path and restart it if you use container |
|
||||||
like Docker. |
|
||||||
|
|
||||||
> Note: If you only want to use MySQL in the datasource center, there is no requirement for the version of MySQL JDBC driver. |
|
||||||
> But if you want to use MySQL as the metabase of DolphinScheduler, it only supports [8.0.16 and above](https:/ /repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.16/mysql-connector-java-8.0.16.jar) version. |
|
||||||
@ -0,0 +1,94 @@ |
|||||||
|
# Datasource Setting |
||||||
|
|
||||||
|
## Standalone Switching Metadata Database Configuration |
||||||
|
|
||||||
|
We here use MySQL as an example to illustrate how to configure an external database: |
||||||
|
|
||||||
|
* First of all, follow the instructions in [datasource-setting](datasource-setting.md) `Pseudo-Cluster/Cluster Initialize the Database` section to create and initialize database |
||||||
|
* Set the following environment variables in your terminal or modify the `bin/env/dolphinscheduler_env.sh` with your database username and password for `{user}` and `{password}`: |
||||||
|
|
||||||
|
```shell |
||||||
|
export DATABASE=mysql |
||||||
|
export SPRING_PROFILES_ACTIVE=${DATABASE} |
||||||
|
export SPRING_DATASOURCE_USERNAME={user} |
||||||
|
export SPRING_DATASOURCE_PASSWORD={password} |
||||||
|
``` |
||||||
|
|
||||||
|
* Add mysql-connector-java driver to `./standalone-server/libs/standalone-server/`, see [general-setting](general-setting.md) `Pseudo-Cluster/Cluster Initialize the Database` section about where to download |
||||||
|
* Start standalone-server, now you are using mysql as database and it will not clear up your data when you stop or restart standalone-server. |
||||||
|
|
||||||
|
## Pseudo-Cluster/Cluster Initialize the Database |
||||||
|
|
||||||
|
DolphinScheduler metadata is stored in the relational database. Currently, supports PostgreSQL and MySQL. If you use MySQL, you need to manually download [mysql-connector-java driver][mysql] (8.0.16) and move it to the libs directory of DolphinScheduler |
||||||
|
which is `api-server/libs/` and `alert-server/libs` and `master-server/libs` and `worker-server/libs`. Let's take MySQL as an example for how to initialize the database: |
||||||
|
|
||||||
|
For mysql 5.6 / 5.7 |
||||||
|
|
||||||
|
```shell |
||||||
|
mysql -uroot -p |
||||||
|
|
||||||
|
mysql> CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; |
||||||
|
|
||||||
|
# Replace {user} and {password} with your username and password |
||||||
|
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'%' IDENTIFIED BY '{password}'; |
||||||
|
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'localhost' IDENTIFIED BY '{password}'; |
||||||
|
|
||||||
|
mysql> flush privileges; |
||||||
|
``` |
||||||
|
|
||||||
|
For mysql 8: |
||||||
|
|
||||||
|
```shell |
||||||
|
mysql -uroot -p |
||||||
|
|
||||||
|
mysql> CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; |
||||||
|
|
||||||
|
# Replace {user} and {password} with your username and password |
||||||
|
mysql> CREATE USER '{user}'@'%' IDENTIFIED BY '{password}'; |
||||||
|
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'%'; |
||||||
|
mysql> CREATE USER '{user}'@'localhost' IDENTIFIED BY '{password}'; |
||||||
|
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'localhost'; |
||||||
|
mysql> FLUSH PRIVILEGES; |
||||||
|
``` |
||||||
|
|
||||||
|
Then, modify `./bin/env/dolphinscheduler_env.sh` to use mysql, change {user} and {password} to what you set in the previous step. |
||||||
|
|
||||||
|
```shell |
||||||
|
export DATABASE=${DATABASE:-mysql} |
||||||
|
export SPRING_PROFILES_ACTIVE=${DATABASE} |
||||||
|
export SPRING_DATASOURCE_URL="jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&useSSL=false" |
||||||
|
export SPRING_DATASOURCE_USERNAME={user} |
||||||
|
export SPRING_DATASOURCE_PASSWORD={password} |
||||||
|
``` |
||||||
|
|
||||||
|
After the above steps done you would create a new database for DolphinScheduler, then run the Shell script to init database: |
||||||
|
|
||||||
|
```shell |
||||||
|
sh tools/bin/upgrade-schema.sh |
||||||
|
``` |
||||||
|
|
||||||
|
## DataSource Center |
||||||
|
|
||||||
|
DataSource supports MySQL, PostgreSQL, Hive/Impala, Spark, ClickHouse, Oracle, SQL Server and other DataSources. |
||||||
|
|
||||||
|
- Click bottom `Data Source Center -> Create Data Source` to create a new datasource. |
||||||
|
- Click `Test Connection` to test whether the DataSource can connect successfully (datasource can be saved only if it passes the connection test). |
||||||
|
|
||||||
|
### Using datasource incompatible to Apache LICENSE V2 LICENSE |
||||||
|
|
||||||
|
Some of datasource are native supported to DolphinScheduler while others need users download JDBC driver package manually, |
||||||
|
because those JDBC driver incompatible to Apache LICENSE V2 LICENSE. For this reason we have to release DolphinScheduler's |
||||||
|
distribute package without those packages, even if this will make more complicated for users. Datasource such as MySQL, |
||||||
|
Oracle, SQL Server as the examples, but we have the solution to solve this |
||||||
|
|
||||||
|
### Example |
||||||
|
|
||||||
|
For example, if you want to use MySQL datasource, you need to download the correct JDBC driver from [mysql maven repository](https://repo1.maven.org/maven2/mysql/mysql-connector-java), |
||||||
|
and move it into directory `api-server/libs` and `worker-server/libs`. After that, you could activate MySQL datasource by |
||||||
|
restarting `api-server` and `worker-server`. Mount to container volume in the same path and restart it if you use container |
||||||
|
like Docker. |
||||||
|
|
||||||
|
> Note: If you only want to use MySQL in the datasource center, there is no requirement for the version of MySQL JDBC driver. |
||||||
|
> But if you want to use MySQL as the metabase of DolphinScheduler, it only supports [8.0.16 and above](https:/ /repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.16/mysql-connector-java-8.0.16.jar) version. |
||||||
|
|
||||||
|
[mysql]: https://downloads.MySQL.com/archives/c-j/ |
||||||
@ -1,21 +0,0 @@ |
|||||||
# 数据源 |
|
||||||
|
|
||||||
数据源中心支持MySQL、POSTGRESQL、HIVE/IMPALA、SPARK、CLICKHOUSE、ORACLE、SQLSERVER等数据源。 |
|
||||||
|
|
||||||
- 点击"数据源中心->创建数据源"，根据需求创建不同类型的数据源 |
|
||||||
- 点击"测试连接"，测试数据源是否可以连接成功（只有当数据源通过连接性测试后才能保存数据源）。 |
|
||||||
|
|
||||||
## 使用不兼容 Apache LICENSE V2 许可的数据库 |
|
||||||
|
|
||||||
数据源中心里，DolphinScheduler 对部分数据源有原生的支持，但是部分数据源需要用户下载对应的 JDBC 驱动包并放置到正确的位置才能正常使用。 |
|
||||||
这对用户会增加用户的使用成本，但是我们不得不这么做，因为这部分数据源的 JDBC 驱动和 Apache LICENSE V2 不兼容，所以我们无法在 |
|
||||||
DolphinScheduler 分发的二进制包中包含他们。这部分数据源主要包括 MySQL，Oracle，SQL Server 等，幸运的是我们为这部分数据源的支持给出了解决方案。 |
|
||||||
|
|
||||||
### 样例 |
|
||||||
|
|
||||||
我们以 MySQL 为例，如果你想要使用 MySQL 数据源，你需要先在 [mysql maven 仓库](https://repo1.maven.org/maven2/mysql/mysql-connector-java) |
|
||||||
中下载对应版本的 JDBC 驱动，将其移入 `api-server/libs` 以及 `worker-server/libs` 文件夹中，最后重启 `api-server` 和 `worker-server` |
|
||||||
服务，即可使用 MySQL 数据源。如果你使用容器启动 DolphinScheduler，同样也是将 JDBC 驱动挂载放到以上两个服务的对应路径下后，重启驱动即可。 |
|
||||||
|
|
||||||
> 注意：如果你只是想要在数据源中心使用 MySQL，则对 MySQL JDBC 驱动的版本没有要求，如果你想要将 MySQL 作为 DolphinScheduler 的元数据库， |
|
||||||
> 则仅支持 [8.0.16 及以上](https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.16/mysql-connector-java-8.0.16.jar)的版本。 |
|
||||||
@ -0,0 +1,92 @@ |
|||||||
|
# 数据源配置 |
||||||
|
|
||||||
|
## Standalone 切换元数据库 |
||||||
|
|
||||||
|
我们这里以 MySQL 为例来说明如何配置外部数据库： |
||||||
|
|
||||||
|
* 首先，参照 [数据源配置](datasource-setting.md) `伪分布式/分布式安装初始化数据库` 创建并初始化数据库 |
||||||
|
* 在你的命令行或者修改 bin/env/dolphinscheduler_env.sh 设定下列环境变量，将 `{user}` 和 `{password}` 改为你数据库的用户名和密码 |
||||||
|
|
||||||
|
```shell |
||||||
|
export DATABASE=mysql |
||||||
|
export SPRING_PROFILES_ACTIVE=${DATABASE} |
||||||
|
export SPRING_DATASOURCE_USERNAME={user} |
||||||
|
export SPRING_DATASOURCE_PASSWORD={password} |
||||||
|
``` |
||||||
|
|
||||||
|
* 将mysql-connector-java驱动加到`./standalone-server/libs/standalone-server/`目录下, 下载方法见 [数据源配置](datasource-setting.md) `伪分布式/分布式安装初始化数据库` 一栏 |
||||||
|
* 启动standalone-server，此时你已经连接上mysql，重启或者停止standalone-server并不会清空您数据库里的数据 |
||||||
|
|
||||||
|
## 伪分布式/分布式安装初始化数据库 |
||||||
|
|
||||||
|
DolphinScheduler 元数据存储在关系型数据库中，目前支持 PostgreSQL 和 MySQL，如果使用 MySQL 则需要手动下载 [mysql-connector-java 驱动][mysql] (8.0.16) 并移动到 DolphinScheduler 的每个模块的 libs 目录下 |
||||||
|
其中包括 `api-server/libs/` 和 `alert-server/libs` 和 `master-server/libs` 和 `worker-server/libs`。下面以 MySQL 为例，说明如何初始化数据库 |
||||||
|
|
||||||
|
对于mysql 5.6 / 5.7： |
||||||
|
|
||||||
|
```shell |
||||||
|
mysql -uroot -p |
||||||
|
|
||||||
|
mysql> CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; |
||||||
|
|
||||||
|
# 修改 {user} 和 {password} 为你希望的用户名和密码 |
||||||
|
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'%' IDENTIFIED BY '{password}'; |
||||||
|
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'localhost' IDENTIFIED BY '{password}'; |
||||||
|
|
||||||
|
mysql> flush privileges; |
||||||
|
``` |
||||||
|
|
||||||
|
对于mysql 8： |
||||||
|
|
||||||
|
```shell |
||||||
|
mysql -uroot -p |
||||||
|
|
||||||
|
mysql> CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; |
||||||
|
|
||||||
|
# 修改 {user} 和 {password} 为你希望的用户名和密码 |
||||||
|
mysql> CREATE USER '{user}'@'%' IDENTIFIED BY '{password}'; |
||||||
|
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'%'; |
||||||
|
mysql> CREATE USER '{user}'@'localhost' IDENTIFIED BY '{password}'; |
||||||
|
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'localhost'; |
||||||
|
mysql> FLUSH PRIVILEGES; |
||||||
|
``` |
||||||
|
|
||||||
|
然后修改`./bin/env/dolphinscheduler_env.sh`，将username和password改成你在上一步中设置的用户名{user}和密码{password} |
||||||
|
|
||||||
|
```shell |
||||||
|
export DATABASE=${DATABASE:-mysql} |
||||||
|
export SPRING_PROFILES_ACTIVE=${DATABASE} |
||||||
|
export SPRING_DATASOURCE_URL="jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&useSSL=false" |
||||||
|
export SPRING_DATASOURCE_USERNAME={user} |
||||||
|
export SPRING_DATASOURCE_PASSWORD={password} |
||||||
|
``` |
||||||
|
|
||||||
|
完成上述步骤后，您已经为 DolphinScheduler 创建一个新数据库，现在你可以通过快速的 Shell 脚本来初始化数据库 |
||||||
|
|
||||||
|
```shell |
||||||
|
sh tools/bin/upgrade-schema.sh |
||||||
|
``` |
||||||
|
|
||||||
|
## 数据源中心 |
||||||
|
|
||||||
|
数据源中心支持MySQL、POSTGRESQL、HIVE/IMPALA、SPARK、CLICKHOUSE、ORACLE、SQLSERVER等数据源。 |
||||||
|
|
||||||
|
- 点击"数据源中心->创建数据源"，根据需求创建不同类型的数据源 |
||||||
|
- 点击"测试连接"，测试数据源是否可以连接成功（只有当数据源通过连接性测试后才能保存数据源）。 |
||||||
|
|
||||||
|
### 使用不兼容 Apache LICENSE V2 许可的数据库 |
||||||
|
|
||||||
|
数据源中心里，DolphinScheduler 对部分数据源有原生的支持，但是部分数据源需要用户下载对应的 JDBC 驱动包并放置到正确的位置才能正常使用。 |
||||||
|
这对用户会增加用户的使用成本，但是我们不得不这么做，因为这部分数据源的 JDBC 驱动和 Apache LICENSE V2 不兼容，所以我们无法在 |
||||||
|
DolphinScheduler 分发的二进制包中包含他们。这部分数据源主要包括 MySQL，Oracle，SQL Server 等，幸运的是我们为这部分数据源的支持给出了解决方案。 |
||||||
|
|
||||||
|
#### 样例 |
||||||
|
|
||||||
|
我们以 MySQL 为例，如果你想要使用 MySQL 数据源，你需要先在 [mysql maven 仓库](https://repo1.maven.org/maven2/mysql/mysql-connector-java) |
||||||
|
中下载对应版本的 JDBC 驱动，将其移入 `api-server/libs` 以及 `worker-server/libs` 文件夹中，最后重启 `api-server` 和 `worker-server` |
||||||
|
服务，即可使用 MySQL 数据源。如果你使用容器启动 DolphinScheduler，同样也是将 JDBC 驱动挂载放到以上两个服务的对应路径下后，重启驱动即可。 |
||||||
|
|
||||||
|
> 注意：如果你只是想要在数据源中心使用 MySQL，则对 MySQL JDBC 驱动的版本没有要求，如果你想要将 MySQL 作为 DolphinScheduler 的元数据库， |
||||||
|
> 则仅支持 [8.0.16 及以上](https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.16/mysql-connector-java-8.0.16.jar)的版本。 |
||||||
|
|
||||||
|
[mysql]: https://downloads.MySQL.com/archives/c-j/ |
||||||
Loading…
Reference in new issue