xiangzihao
2 years ago
committed by
caishunfeng
14 changed files with 202 additions and 173 deletions
@ -1,23 +0,0 @@
|
||||
# DataSource |
||||
|
||||
DataSource supports MySQL, PostgreSQL, Hive/Impala, Spark, ClickHouse, Oracle, SQL Server and other DataSources. |
||||
|
||||
- Click bottom `Data Source Center -> Create Data Source` to create a new datasource. |
||||
- Click `Test Connection` to test whether the DataSource can connect successfully (datasource can be saved only if it passes the connection test). |
||||
|
||||
## Using datasource incompatible to Apache LICENSE V2 LICENSE |
||||
|
||||
Some of datasource are native supported to DolphinScheduler while others need users download JDBC driver package manually, |
||||
because those JDBC driver incompatible to Apache LICENSE V2 LICENSE. For this reason we have to release DolphinScheduler's |
||||
distribute package without those packages, even if this will make more complicated for users. Datasource such as MySQL, |
||||
Oracle, SQL Server as the examples, but we have the solution to solve this |
||||
|
||||
### Example |
||||
|
||||
For example, if you want to use MySQL datasource, you need to download the correct JDBC driver from [mysql maven repository](https://repo1.maven.org/maven2/mysql/mysql-connector-java), |
||||
and move it into directory `api-server/libs` and `worker-server/libs`. After that, you could activate MySQL datasource by |
||||
restarting `api-server` and `worker-server`. Mount to container volume in the same path and restart it if you use container |
||||
like Docker. |
||||
|
||||
> Note: If you only want to use MySQL in the datasource center, there is no requirement for the version of MySQL JDBC driver. |
||||
> But if you want to use MySQL as the metabase of DolphinScheduler, it only supports [8.0.16 and above](https:/ /repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.16/mysql-connector-java-8.0.16.jar) version. |
||||
@ -0,0 +1,94 @@
|
||||
# Datasource Setting |
||||
|
||||
## Standalone Switching Metadata Database Configuration |
||||
|
||||
We here use MySQL as an example to illustrate how to configure an external database: |
||||
|
||||
* First of all, follow the instructions in [datasource-setting](datasource-setting.md) `Pseudo-Cluster/Cluster Initialize the Database` section to create and initialize database |
||||
* Set the following environment variables in your terminal or modify the `bin/env/dolphinscheduler_env.sh` with your database username and password for `{user}` and `{password}`: |
||||
|
||||
```shell |
||||
export DATABASE=mysql |
||||
export SPRING_PROFILES_ACTIVE=${DATABASE} |
||||
export SPRING_DATASOURCE_USERNAME={user} |
||||
export SPRING_DATASOURCE_PASSWORD={password} |
||||
``` |
||||
|
||||
* Add mysql-connector-java driver to `./standalone-server/libs/standalone-server/`, see [general-setting](general-setting.md) `Pseudo-Cluster/Cluster Initialize the Database` section about where to download |
||||
* Start standalone-server, now you are using mysql as database and it will not clear up your data when you stop or restart standalone-server. |
||||
|
||||
## Pseudo-Cluster/Cluster Initialize the Database |
||||
|
||||
DolphinScheduler metadata is stored in the relational database. Currently, supports PostgreSQL and MySQL. If you use MySQL, you need to manually download [mysql-connector-java driver][mysql] (8.0.16) and move it to the libs directory of DolphinScheduler |
||||
which is `api-server/libs/` and `alert-server/libs` and `master-server/libs` and `worker-server/libs`. Let's take MySQL as an example for how to initialize the database: |
||||
|
||||
For mysql 5.6 / 5.7 |
||||
|
||||
```shell |
||||
mysql -uroot -p |
||||
|
||||
mysql> CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; |
||||
|
||||
# Replace {user} and {password} with your username and password |
||||
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'%' IDENTIFIED BY '{password}'; |
||||
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'localhost' IDENTIFIED BY '{password}'; |
||||
|
||||
mysql> flush privileges; |
||||
``` |
||||
|
||||
For mysql 8: |
||||
|
||||
```shell |
||||
mysql -uroot -p |
||||
|
||||
mysql> CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; |
||||
|
||||
# Replace {user} and {password} with your username and password |
||||
mysql> CREATE USER '{user}'@'%' IDENTIFIED BY '{password}'; |
||||
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'%'; |
||||
mysql> CREATE USER '{user}'@'localhost' IDENTIFIED BY '{password}'; |
||||
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'localhost'; |
||||
mysql> FLUSH PRIVILEGES; |
||||
``` |
||||
|
||||
Then, modify `./bin/env/dolphinscheduler_env.sh` to use mysql, change {user} and {password} to what you set in the previous step. |
||||
|
||||
```shell |
||||
export DATABASE=${DATABASE:-mysql} |
||||
export SPRING_PROFILES_ACTIVE=${DATABASE} |
||||
export SPRING_DATASOURCE_URL="jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&useSSL=false" |
||||
export SPRING_DATASOURCE_USERNAME={user} |
||||
export SPRING_DATASOURCE_PASSWORD={password} |
||||
``` |
||||
|
||||
After the above steps done you would create a new database for DolphinScheduler, then run the Shell script to init database: |
||||
|
||||
```shell |
||||
sh tools/bin/upgrade-schema.sh |
||||
``` |
||||
|
||||
## DataSource Center |
||||
|
||||
DataSource supports MySQL, PostgreSQL, Hive/Impala, Spark, ClickHouse, Oracle, SQL Server and other DataSources. |
||||
|
||||
- Click bottom `Data Source Center -> Create Data Source` to create a new datasource. |
||||
- Click `Test Connection` to test whether the DataSource can connect successfully (datasource can be saved only if it passes the connection test). |
||||
|
||||
### Using datasource incompatible to Apache LICENSE V2 LICENSE |
||||
|
||||
Some of datasource are native supported to DolphinScheduler while others need users download JDBC driver package manually, |
||||
because those JDBC driver incompatible to Apache LICENSE V2 LICENSE. For this reason we have to release DolphinScheduler's |
||||
distribute package without those packages, even if this will make more complicated for users. Datasource such as MySQL, |
||||
Oracle, SQL Server as the examples, but we have the solution to solve this |
||||
|
||||
### Example |
||||
|
||||
For example, if you want to use MySQL datasource, you need to download the correct JDBC driver from [mysql maven repository](https://repo1.maven.org/maven2/mysql/mysql-connector-java), |
||||
and move it into directory `api-server/libs` and `worker-server/libs`. After that, you could activate MySQL datasource by |
||||
restarting `api-server` and `worker-server`. Mount to container volume in the same path and restart it if you use container |
||||
like Docker. |
||||
|
||||
> Note: If you only want to use MySQL in the datasource center, there is no requirement for the version of MySQL JDBC driver. |
||||
> But if you want to use MySQL as the metabase of DolphinScheduler, it only supports [8.0.16 and above](https:/ /repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.16/mysql-connector-java-8.0.16.jar) version. |
||||
|
||||
[mysql]: https://downloads.MySQL.com/archives/c-j/ |
||||
@ -1,21 +0,0 @@
|
||||
# 数据源 |
||||
|
||||
数据源中心支持MySQL、POSTGRESQL、HIVE/IMPALA、SPARK、CLICKHOUSE、ORACLE、SQLSERVER等数据源。 |
||||
|
||||
- 点击"数据源中心->创建数据源"，根据需求创建不同类型的数据源 |
||||
- 点击"测试连接"，测试数据源是否可以连接成功（只有当数据源通过连接性测试后才能保存数据源）。 |
||||
|
||||
## 使用不兼容 Apache LICENSE V2 许可的数据库 |
||||
|
||||
数据源中心里，DolphinScheduler 对部分数据源有原生的支持，但是部分数据源需要用户下载对应的 JDBC 驱动包并放置到正确的位置才能正常使用。 |
||||
这对用户会增加用户的使用成本，但是我们不得不这么做，因为这部分数据源的 JDBC 驱动和 Apache LICENSE V2 不兼容，所以我们无法在 |
||||
DolphinScheduler 分发的二进制包中包含他们。这部分数据源主要包括 MySQL，Oracle，SQL Server 等，幸运的是我们为这部分数据源的支持给出了解决方案。 |
||||
|
||||
### 样例 |
||||
|
||||
我们以 MySQL 为例，如果你想要使用 MySQL 数据源，你需要先在 [mysql maven 仓库](https://repo1.maven.org/maven2/mysql/mysql-connector-java) |
||||
中下载对应版本的 JDBC 驱动，将其移入 `api-server/libs` 以及 `worker-server/libs` 文件夹中，最后重启 `api-server` 和 `worker-server` |
||||
服务，即可使用 MySQL 数据源。如果你使用容器启动 DolphinScheduler，同样也是将 JDBC 驱动挂载放到以上两个服务的对应路径下后，重启驱动即可。 |
||||
|
||||
> 注意：如果你只是想要在数据源中心使用 MySQL，则对 MySQL JDBC 驱动的版本没有要求，如果你想要将 MySQL 作为 DolphinScheduler 的元数据库， |
||||
> 则仅支持 [8.0.16 及以上](https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.16/mysql-connector-java-8.0.16.jar)的版本。 |
||||
@ -0,0 +1,92 @@
|
||||
# 数据源配置 |
||||
|
||||
## Standalone 切换元数据库 |
||||
|
||||
我们这里以 MySQL 为例来说明如何配置外部数据库： |
||||
|
||||
* 首先，参照 [数据源配置](datasource-setting.md) `伪分布式/分布式安装初始化数据库` 创建并初始化数据库 |
||||
* 在你的命令行或者修改 bin/env/dolphinscheduler_env.sh 设定下列环境变量，将 `{user}` 和 `{password}` 改为你数据库的用户名和密码 |
||||
|
||||
```shell |
||||
export DATABASE=mysql |
||||
export SPRING_PROFILES_ACTIVE=${DATABASE} |
||||
export SPRING_DATASOURCE_USERNAME={user} |
||||
export SPRING_DATASOURCE_PASSWORD={password} |
||||
``` |
||||
|
||||
* 将mysql-connector-java驱动加到`./standalone-server/libs/standalone-server/`目录下, 下载方法见 [数据源配置](datasource-setting.md) `伪分布式/分布式安装初始化数据库` 一栏 |
||||
* 启动standalone-server，此时你已经连接上mysql，重启或者停止standalone-server并不会清空您数据库里的数据 |
||||
|
||||
## 伪分布式/分布式安装初始化数据库 |
||||
|
||||
DolphinScheduler 元数据存储在关系型数据库中，目前支持 PostgreSQL 和 MySQL，如果使用 MySQL 则需要手动下载 [mysql-connector-java 驱动][mysql] (8.0.16) 并移动到 DolphinScheduler 的每个模块的 libs 目录下 |
||||
其中包括 `api-server/libs/` 和 `alert-server/libs` 和 `master-server/libs` 和 `worker-server/libs`。下面以 MySQL 为例，说明如何初始化数据库 |
||||
|
||||
对于mysql 5.6 / 5.7： |
||||
|
||||
```shell |
||||
mysql -uroot -p |
||||
|
||||
mysql> CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; |
||||
|
||||
# 修改 {user} 和 {password} 为你希望的用户名和密码 |
||||
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'%' IDENTIFIED BY '{password}'; |
||||
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'localhost' IDENTIFIED BY '{password}'; |
||||
|
||||
mysql> flush privileges; |
||||
``` |
||||
|
||||
对于mysql 8： |
||||
|
||||
```shell |
||||
mysql -uroot -p |
||||
|
||||
mysql> CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; |
||||
|
||||
# 修改 {user} 和 {password} 为你希望的用户名和密码 |
||||
mysql> CREATE USER '{user}'@'%' IDENTIFIED BY '{password}'; |
||||
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'%'; |
||||
mysql> CREATE USER '{user}'@'localhost' IDENTIFIED BY '{password}'; |
||||
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'localhost'; |
||||
mysql> FLUSH PRIVILEGES; |
||||
``` |
||||
|
||||
然后修改`./bin/env/dolphinscheduler_env.sh`，将username和password改成你在上一步中设置的用户名{user}和密码{password} |
||||
|
||||
```shell |
||||
export DATABASE=${DATABASE:-mysql} |
||||
export SPRING_PROFILES_ACTIVE=${DATABASE} |
||||
export SPRING_DATASOURCE_URL="jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&useSSL=false" |
||||
export SPRING_DATASOURCE_USERNAME={user} |
||||
export SPRING_DATASOURCE_PASSWORD={password} |
||||
``` |
||||
|
||||
完成上述步骤后，您已经为 DolphinScheduler 创建一个新数据库，现在你可以通过快速的 Shell 脚本来初始化数据库 |
||||
|
||||
```shell |
||||
sh tools/bin/upgrade-schema.sh |
||||
``` |
||||
|
||||
## 数据源中心 |
||||
|
||||
数据源中心支持MySQL、POSTGRESQL、HIVE/IMPALA、SPARK、CLICKHOUSE、ORACLE、SQLSERVER等数据源。 |
||||
|
||||
- 点击"数据源中心->创建数据源"，根据需求创建不同类型的数据源 |
||||
- 点击"测试连接"，测试数据源是否可以连接成功（只有当数据源通过连接性测试后才能保存数据源）。 |
||||
|
||||
### 使用不兼容 Apache LICENSE V2 许可的数据库 |
||||
|
||||
数据源中心里，DolphinScheduler 对部分数据源有原生的支持，但是部分数据源需要用户下载对应的 JDBC 驱动包并放置到正确的位置才能正常使用。 |
||||
这对用户会增加用户的使用成本，但是我们不得不这么做，因为这部分数据源的 JDBC 驱动和 Apache LICENSE V2 不兼容，所以我们无法在 |
||||
DolphinScheduler 分发的二进制包中包含他们。这部分数据源主要包括 MySQL，Oracle，SQL Server 等，幸运的是我们为这部分数据源的支持给出了解决方案。 |
||||
|
||||
#### 样例 |
||||
|
||||
我们以 MySQL 为例，如果你想要使用 MySQL 数据源，你需要先在 [mysql maven 仓库](https://repo1.maven.org/maven2/mysql/mysql-connector-java) |
||||
中下载对应版本的 JDBC 驱动，将其移入 `api-server/libs` 以及 `worker-server/libs` 文件夹中，最后重启 `api-server` 和 `worker-server` |
||||
服务，即可使用 MySQL 数据源。如果你使用容器启动 DolphinScheduler，同样也是将 JDBC 驱动挂载放到以上两个服务的对应路径下后，重启驱动即可。 |
||||
|
||||
> 注意：如果你只是想要在数据源中心使用 MySQL，则对 MySQL JDBC 驱动的版本没有要求，如果你想要将 MySQL 作为 DolphinScheduler 的元数据库， |
||||
> 则仅支持 [8.0.16 及以上](https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.16/mysql-connector-java-8.0.16.jar)的版本。 |
||||
|
||||
[mysql]: https://downloads.MySQL.com/archives/c-j/ |
||||
Loading…
Reference in new issue