From 5c1edd2912b63cfa57a2dc914670ffe5e0a70e09 Mon Sep 17 00:00:00 2001 From: JieguangZhou Date: Fri, 21 Apr 2023 17:00:14 +0800 Subject: [PATCH] [Improvement][DOC] Update resource S3 configuration docs (#13985) * update resoure s3 docs --- docs/docs/en/guide/resource/configuration.md | 28 +++++++++++++++++++- docs/docs/zh/guide/resource/configuration.md | 26 ++++++++++++++++++ 2 files changed, 53 insertions(+), 1 deletion(-) diff --git a/docs/docs/en/guide/resource/configuration.md b/docs/docs/en/guide/resource/configuration.md index bd12598f1d..9b756b7736 100644 --- a/docs/docs/en/guide/resource/configuration.md +++ b/docs/docs/en/guide/resource/configuration.md @@ -26,9 +26,35 @@ The configuration you may need to change: > and `resource.hdfs.fs.defaultFS=file:///`, The configuration of `resource.storage.type=LOCAL` is for user-friendly, and enables > the local resource center to be enabled by default +## connect AWS S3 + +if you want to upload resources to `Resource Center` connected to `S3`, you need to configure `api-server/conf/common.properties` and `worker-server/conf/common.properties`. You can refer to the following: + +config the following fields + +```properties +...... + +resource.storage.type=S3 + +...... + +resource.aws.access.key.id=aws_access_key_id +# The AWS secret access key. if resource.storage.type=S3 or use EMR-Task, This configuration is required +resource.aws.secret.access.key=aws_secret_access_key +# The AWS Region to use. if resource.storage.type=S3 or use EMR-Task, This configuration is required +resource.aws.region=us-west-2 +# The name of the bucket. You need to create them by yourself. Otherwise, the system cannot start. All buckets in Amazon S3 share a single namespace; ensure the bucket is given a unique name. +resource.aws.s3.bucket.name=dolphinscheduler +# You need to set this parameter when private cloud s4. If S3 uses public cloud, you only need to set resource.aws.region or set to the endpoint of a public cloud such as S3.cn-north-1.amazonaws.com.cn +resource.aws.s3.endpoint= + +...... +``` + ## Use HDFS or Remote Object Storage -After version 3.0.0-alpha, if you want to upload resources to `Resource Center` connected to `HDFS` or `S3`, you need to configure `api-server/conf/common.properties` and `worker-server/conf/common.properties`. +After version 3.0.0-alpha, if you want to upload resources to `Resource Center` connected to `HDFS`, you need to configure `api-server/conf/common.properties` and `worker-server/conf/common.properties`. ```properties # diff --git a/docs/docs/zh/guide/resource/configuration.md b/docs/docs/zh/guide/resource/configuration.md index 054b593b29..7bd152c2f7 100644 --- a/docs/docs/zh/guide/resource/configuration.md +++ b/docs/docs/zh/guide/resource/configuration.md @@ -24,6 +24,32 @@ Dolphinscheduler 资源中心使用本地系统默认是开启的，不需要用 > 3. 当配置 `resource.storage.type=LOCAL`，其实您配置了两个配置项，分别是 `resource.storage.type=HDFS` 和 `resource.hdfs.fs.defaultFS=file:///` ，我们单独配置 `resource.storage.type=LOCAL` 这个值是为了 > 方便用户，并且能使得本地资源中心默认开启 +## 对接AWS S3 + +如果需要使用到资源中心的 S3 上传资源，我们需要对以下路径的进行配置：`api-server/conf/common.properties` 和 `worker-server/conf/common.properties`。可参考如下： + +配置以下字段 + +```properties +...... + +resource.storage.type=S3 + +...... + +resource.aws.access.key.id=aws_access_key_id +# The AWS secret access key. if resource.storage.type=S3 or use EMR-Task, This configuration is required +resource.aws.secret.access.key=aws_secret_access_key +# The AWS Region to use. if resource.storage.type=S3 or use EMR-Task, This configuration is required +resource.aws.region=us-west-2 +# The name of the bucket. You need to create them by yourself. Otherwise, the system cannot start. All buckets in Amazon S3 share a single namespace; ensure the bucket is given a unique name. +resource.aws.s3.bucket.name=dolphinscheduler +# You need to set this parameter when private cloud s4. If S3 uses public cloud, you only need to set resource.aws.region or set to the endpoint of a public cloud such as S3.cn-north-1.amazonaws.com.cn +resource.aws.s3.endpoint= + +...... +``` + ## 对接分布式或远端对象存储 当需要使用资源中心进行相关文件的创建或者上传操作时，所有的文件和资源都会被存储在分布式文件系统`HDFS`或者远端的对象存储，如`S3`上。所以需要进行以下配置：