Browse Source

[doc] refactor release (#13135)

* [doc] refactor release

* [doc] refactor release
3.2.0-release
Jay Chung 2 years ago committed by GitHub
parent
commit
316496211a
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
  1. 34
      docs/configs/docsdev.js
  2. 292
      docs/docs/en/contribute/release.md
  3. 26
      docs/docs/en/contribute/release/release-post.md
  4. 31
      docs/docs/en/contribute/release/release-prepare.md
  5. 284
      docs/docs/zh/contribute/release.md
  6. 24
      docs/docs/zh/contribute/release/release-post.md
  7. 32
      docs/docs/zh/contribute/release/release-prepare.md

34
docs/configs/docsdev.js

@ -622,21 +622,8 @@ export default {
],
},
{
title: 'Release Guide',
children: [
{
title: 'Release Preparation',
link: '/en-us/docs/dev/user_doc/contribute/release/release-prepare.html',
},
{
title: 'Release Guide',
link: '/en-us/docs/dev/user_doc/contribute/release/release.html',
},
{
title: 'Release Post',
link: '/en-us/docs/dev/user_doc/contribute/release/release-post.html',
},
],
title: 'Releasing',
link: '/en-us/docs/dev/user_doc/contribute/release.html',
},
{
title: 'Questions & Communications',
@ -1262,21 +1249,8 @@ export default {
],
},
{
title: '发版指南',
children: [
{
title: '发版准备',
link: '/zh-cn/docs/dev/user_doc/contribute/release/release-prepare.html',
},
{
title: '发版指南',
link: '/zh-cn/docs/dev/user_doc/contribute/release/release.html',
},
{
title: '发版后续',
link: '/zh-cn/docs/dev/user_doc/contribute/release/release-post.html',
},
],
title: '发版',
link: '/zh-cn/docs/dev/user_doc/contribute/release.html',
},
{
title: '问题与交流',

292
docs/docs/en/contribute/release/release.md → docs/docs/en/contribute/release.md

@ -1,6 +1,11 @@
# Release Guide
## Check Your Environment
## Preparation
This section is pre-release work, most of it is one-off, meaning **it is only needed for your first release**. If you have
released before, then skip this section to the next.
### Check Your Environment
To make sure you could successfully complete the release for DolphinScheduler, you should check your environment and make sure
all conditions are met, if any or them are missing, you should install them and make sure them work.
@ -12,9 +17,9 @@ java -version
mvn -version
```
## GPG Settings
### GPG Settings
### Install GPG
#### Install GPG
Download installation package on [official GnuPG website](https://www.gnupg.org/download/index.html).
The command of GnuPG 1.x version can differ a little from that of 2.x version.
@ -26,7 +31,7 @@ After the installation, execute the following command to check the version numbe
gpg --version
```
### Create Key
#### Create Key
After the installation, execute the following command to create key.
@ -80,7 +85,7 @@ Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
You need a Passphrase to protect your secret key. # Input your Apache mail passwords
```
### Check Generated Key
#### Check Generated Key
```shell
gpg --list-keys
@ -96,7 +101,7 @@ sub 4096R/A63BC462 2019-11-15
Among them, 85E11560 is public key ID.
### Upload the Public Key to Key Server
#### Upload the Public Key to Key Server
The command is as follow:
@ -108,9 +113,9 @@ gpg --keyserver hkp://pool.sks-keyservers.net --send-key 85E11560
Each server will automatically synchronize with one another, so it would be okay to choose any one, a backup keys servers
is `gpg --keyserver hkp://keyserver.ubuntu.com --send-key <YOUR_KEY_ID>`
## Apache Maven Central Repository Release
### Configure Apache Maven Central Repository
### Set `settings-security.xml` and `settings.xml`
#### Set `settings-security.xml` and `settings.xml`
In this section, we add Apache server maven configuration to prepare the release, we have to add `settings-security.xml` according
to [here](http://maven.apache.org/guides/mini/guide-encryption.html) firstly and then change your `~/.m2/settings.xml` like below
@ -132,6 +137,16 @@ to [here](http://maven.apache.org/guides/mini/guide-encryption.html) firstly and
</settings>
```
## Releasing
### Check release-docs
Compared with the last release, the `release-docs` of the current release needs to be updated to the latest, if there are dependencies and versions changes
- `dolphinscheduler-dist/release-docs/LICENSE`
- `dolphinscheduler-dist/release-docs/NOTICE`
- `dolphinscheduler-dist/release-docs/licenses`
### Set Release in Environment
We will use the release version, your github name and your Apache username below several times, so it is better to store
@ -139,8 +154,13 @@ it to bash variable for easier use.
```shell
VERSION=<THE-VERSION-YOU-RELEASE>
SOURCE_CODE_DIR=<YOUR-SOURCE-CODE-ROOT-DIR> # the directory of your source code hold, the location of parent pom.xml instead of binary package
GH_USERNAME=<YOUR-GITHUB-USERNAME>
GH_REMOTE=<GITHUB-REMOTE> # we use `upstream` or `origin` mostly base on your release environment
A_USERNAME=<YOUR-APACHE-USERNAME>
SVN_DIR=<PATH-TO-SVN-ROOT> # to keep binary package checkout from SVN, the sub path end with `/dolphinscheduler/dev` and `/dolphinscheduler/release` will be create
```
> Note: We can use the variable directly in you bash after we set environment, without changing anything. For example, we
@ -149,19 +169,43 @@ A_USERNAME=<YOUR-APACHE-USERNAME>
> some of not bash step like [vote mail](#vote-procedure), we are using `<VERSION>` instead of `"${VERSION}"` to notice release
> manager they have to change by hand.
### Create Release Branch
### Update Documentation or Code Version
We need to update some documentation before the Maven release. For example, to release version `VERSION`, the following updates are required:
In this section, we dwonload source code from github and create new branch to release
- Version in the code:
- `sql`:
- `dolphinscheduler_mysql.sql`: `t_ds_version` needs to be updated to `VERSION`
- `dolphinscheduler_postgre.sql`: `t_ds_version` needs to be updated to `VERSION`
- `dolphinscheduler_h2.sql`: `t_ds_version` needs to be updated to `VERSION`
- `upgrade`: whether to add `VERSION_schema` if there are some upgrade DDL or DML, can skip this step if no any DDL or DML added.
- `soft_version`: need to be updated to `VERSION`
- `deploy/docker/.env`: `HUB` change to `apache`,`TAG` change to `VERSION`
- `deploy/kubernetes/dolphinscheduler`:
- `Chart.yaml`: `appVersion` and `version` needs to be updated to x.y.z
- `values.yaml`: `image.tag` needs to be updated to x.y.z
- Version in the docs:
- Change the placeholder `<version>`(except `pom`) to the `x.y.z` in directory `docs`
- Add new history version
- `docs/docs/en/history-versions.md` and `docs/docs/zh/history-versions.md`: Add the new version and link for `x.y.z`
- `docs/configs/docsdev.js`: change `/dev/` to `/x.y.z/`, **DO NOT** change this filename, is will be auto change by website tools.
> Note: `VERSION` is a place hold string, is same as the version we set in `VERSION=<THE-VERSION-YOU-RELEASE>`.
### Maven Release
#### Maven Release Check
Create release branch base on prepare branch.
```shell
git clone -b "${VERSION}"-prepare https://github.com/apache/dolphinscheduler.git
cd ~/dolphinscheduler/
git pull
git checkout -b "${VERSION}"-release
git push origin "${VERSION}"-release
cd "${SOURCE_CODE_DIR}"
git checkout -b "${VERSION}"-release "${VERSION}"-prepare
git push "${GH_REMOTE}" "${VERSION}"-release
```
### Pre-Release Check
> Note: If you release in remote host without source code, you should run `git clone -b "${VERSION}"-prepare https://github.com/apache/dolphinscheduler.git`
> first to clone the source code. And then make sure you set `GH_REMOTE="origin"` to make all command work fine.
```shell
mvn release:prepare -Prelease -Darguments="-Dmaven.test.skip=true -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true -Dspotless.check.skip=true" -DautoVersionSubmodules=true -DdryRun=true -Dusername="${GH_USERNAME}"
@ -171,7 +215,7 @@ mvn release:prepare -Prelease -Darguments="-Dmaven.test.skip=true -Dcheckstyle.s
* `-DautoVersionSubmodules=true`: it can make the version number is inputted only once and not for each sub-module.
* `-DdryRun=true`: dry run which means not to generate or submit new version number and new tag.
### Prepare for the Release
#### Maven Release Check Prepare
First, clean local pre-release check information.
@ -196,8 +240,8 @@ It is basically the same as the previous rehearsal command, but deleting `-DdryR
After making sure there is no mistake in local files, submit them to GitHub.
```shell
git push -u origin "${VERSION}"-release
git push origin --tags
git push -u "${GH_REMOTE}" "${VERSION}"-release
git push "${GH_REMOTE}" --tags
```
<!-- markdown-link-check-disable -->
@ -211,7 +255,7 @@ git push origin --tags
<!-- markdown-link-check-enable -->
### Deploy the Release
#### Maven Release Deploy
```shell
mvn release:perform -Prelease -Darguments="-Dmaven.test.skip=true -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true -Dspotless.check.skip=true" -DautoVersionSubmodules=true -Dusername="${GH_USERNAME}"
@ -222,112 +266,69 @@ Go to [apache staging repositories](https://repository.apache.org/#stagingReposi
Click `Close` to tell Nexus that the construction is finished, because only in this way, this version can be usable.
If there is any problem in gpg signature, `Close` will fail, but you can see the failure information through `Activity`.
## Apache SVN Repository Release
### SVN
### Checkout dolphinscheduler Release Directory
#### Checkout Dolphinscheduler Release Directory
If there is no local work directory, create one at first.
We need too checkout Dolphinscheduler dev release directory to local, and
```shell
mkdir -p ~/ds_svn/dev/
cd ~/ds_svn/dev/
```
SVN_DIR_DEV="${SVN_DIR}/dolphinscheduler/dev"
SVN_DIR_RELEASE="${SVN_DIR}/dolphinscheduler/release"
# Optional, only if the SVN root path not exists.
mkdir -p "${SVN_DIR_DEV}"
After the creation, checkout dolphinscheduler release directory from Apache SVN.
```shell
svn --username="${A_USERNAME}" co https://dist.apache.org/repos/dist/dev/dolphinscheduler
cd ~/ds_svn/dev/dolphinscheduler
# When you first time checkout from this path
svn --username="${A_USERNAME}" co https://dist.apache.org/repos/dist/dev/dolphinscheduler "${SVN_DIR_DEV}"
# Or update when the svn directory exists, and you already checkout
svn --username="${A_USERNAME}" update "${SVN_DIR_DEV}"
```
### Export your new gpg KEYS to release(optional)
> NOTE: it may take a few minutes to sync to the mirror in your first time checkout, because it will download all the files
#### Export New GPG Key to KEYS(Optional)
Only if the first time you release with this gpg KEY, including it is you first release, or you change your KEY. You should
change working directory to another one because this step need checkout and change KEYS in release directory.
```shell
mkdir -p ~/ds_svn/release/
cd ~/ds_svn/release/
# Optional, only if the SVN root path not exists.
mkdir -p "${SVN_DIR_RELEASE}"
cd "${SVN_DIR_RELEASE}"
svn --username="${A_USERNAME}" co https://dist.apache.org/repos/dist/release/dolphinscheduler
# Change the placeholder <YOUR-GPG-KEY-ID> to your id
gpg -a --export <YOUR-GPG-KEY-ID> >> KEYS
svn add *
svn --username="${A_USERNAME}" commit -m "new key <YOUR-GPG-KEY-ID> add"
```
> NOTE: it may take a few minutes to sync to the mirror in your first time checkout, because it will download all the files
### Add the Release Content to SVN Directory
#### Add Release Content to SVN
Create folder by version number.
Create folder by version number, and move the source code packages, binary packages and executable binary packages to SVN working directory.
```shell
mkdir -p ~/ds_svn/dev/dolphinscheduler/"${VERSION}"
cd ~/ds_svn/dev/dolphinscheduler/"${VERSION}"
```
mkdir -p "${SVN_DIR_DEV}/${VERSION}"
Add source code packages, binary packages and executable binary packages to SVN working directory.
# Add to SVN
cp -f "${SOURCE_CODE_DIR}"/dolphinscheduler-dist/target/*.tar.gz "${SVN_DIR_DEV}/${VERSION}"
cp -f "${SOURCE_CODE_DIR}"/dolphinscheduler-dist/target/*.tar.gz.asc "${SVN_DIR_DEV}/${VERSION}"
```shell
# Source and binary tarball for main code
cp -f ~/dolphinscheduler/dolphinscheduler-dist/target/*.tar.gz ~/ds_svn/dev/dolphinscheduler/"${VERSION}"
cp -f ~/dolphinscheduler/dolphinscheduler-dist/target/*.tar.gz.asc ~/ds_svn/dev/dolphinscheduler/"${VERSION}"
```
### Generate sign files
```shell
# Create sign
cd "${SVN_DIR_DEV}"
shasum -a 512 apache-dolphinscheduler-"${VERSION}"-src.tar.gz >> apache-dolphinscheduler-"${VERSION}"-src.tar.gz.sha512
shasum -b -a 512 apache-dolphinscheduler-"${VERSION}"-bin.tar.gz >> apache-dolphinscheduler-"${VERSION}"-bin.tar.gz.sha512
```
### Commit to Apache SVN
```shell
cd ~/ds_svn/dev/dolphinscheduler
svn add *
svn --username="${A_USERNAME}" commit -m "release ${VERSION}"
```
## Check Release
### Check sha512 hash
```shell
# Check sha512
shasum -c apache-dolphinscheduler-"${VERSION}"-src.tar.gz.sha512
shasum -c apache-dolphinscheduler-"${VERSION}"-bin.tar.gz.sha512
```
### Check gpg Signature
First, import releaser's public key.
Import KEYS from SVN repository to local. (The releaser does not need to import again; the checking assistant needs to import it, with the user name filled as the releaser's. )
```shell
curl https://dist.apache.org/repos/dist/release/dolphinscheduler/KEYS >> KEYS
gpg --import KEYS
gpg --edit-key "${A_USERNAME}"
> trust
Please decide how far you trust this user to correctly verify other users' keys
(by looking at passports, checking fingerprints from different sources, etc.)
1 = I don't know or won't say
2 = I do NOT trust
3 = I trust marginally
4 = I trust fully
5 = I trust ultimately
m = back to the main menu
Your decision? 5
> save
```
Then, check the gpg signature.
```shell
# Check gpg signature
gpg --verify apache-dolphinscheduler-"${VERSION}"-src.tar.gz.asc
gpg --verify apache-dolphinscheduler-"${VERSION}"-bin.tar.gz.asc
# Commit to Apache SVN
svn add *
svn --username="${A_USERNAME}" commit -m "release ${VERSION}"
```
> Note: You have to create gpg signature manually when you can not find your `asc` file, the command
@ -361,7 +362,7 @@ Decompress `apache-dolphinscheduler-<VERSION>-src.tar.gz` to check the following
- All the third party dependency licenses are under `licenses` folder
- If it depends on Apache license and has a `NOTICE` file, that `NOTICE` file need to be added to `NOTICE` file of the release
## Call for a Vote
## VOTE
### Update Release Notes
@ -374,15 +375,15 @@ It should be done before vote mail because we need the release note in the mail.
### Vote procedure
1. DolphinScheduler community vote: send the vote e-mail to `dev@dolphinscheduler.apache.org`.
PMC needs to check the rightness of the version according to the document before they vote.
After at least 72 hours and with at least 3 `+1 and no -1 PMC member` votes, it can come to the next stage of the vote.
DolphinScheduler community vote: send the vote e-mail to `dev@dolphinscheduler.apache.org`. PMC needs to check the
rightness of the version according to the document before they vote. After at least 72 hours and with at least 3
`+1 and no -1 PMC member` votes, it can come to the next stage of the vote.
2. Announce the vote result: send the result vote e-mail to `dev@dolphinscheduler.apache.org`
Announce the vote result: send the result vote e-mail to `dev@dolphinscheduler.apache.org`
### Vote Templates
### Templates
#### DolphinScheduler Community Vote Template
#### Vote Template
Title:
@ -429,7 +430,13 @@ Checklist for reference:
[ ] No compiled archives bundled in source archive.
```
2. Announce the vote result:
#### Result Template
Title:
```txt
[RESULT][VOTE] Release Apache DolphinScheduler <VERSION>
```
Body:
@ -449,14 +456,24 @@ xxx
Thanks everyone for taking time to check this release and help us.
```
## Finish the Release
## Announce
### Move source packages, binary packages from the `dev` directory to `release` directory
### Handle Release Tarball and Remove Release Branch
```shell
# move to release directory
svn mv -m "release ${VERSION}" https://dist.apache.org/repos/dist/dev/dolphinscheduler/"${VERSION}" https://dist.apache.org/repos/dist/release/dolphinscheduler/
# remove old release directory
svn delete -m "remove old release" https://dist.apache.org/repos/dist/release/dolphinscheduler/<PREVIOUS-RELEASE-VERSION>
# Remove prepare branch
cd "${SOURCE_CODE_DIR}"
git push --delete "${GH_REMOTE}" "${VERSION}-prepare"
```
and then find DolphinScheduler in [apache staging repositories](https://repository.apache.org/#stagingRepositories) and click `Release`
### Update Document
Website should be present before you send the announce mail this section will tell you how to change the website. For example,
@ -472,10 +489,56 @@ the release version is `<VERSION>`, the following updates are required(note it w
- `docs/configs/index.md.jsx`: Add `<VERSION>: docsxyzConfig` and add new `import` for the new `docsxyzConfig`
- `docs/docs/en/history-versions.md` and `docs/docs/zh/history-versions.md`: Add new `<VERSION>` release docs.
- `.github/ISSUE_TEMPLATE/bug-report.yml`: DolphinScheduler's GitHub [bug-report](https://github.com/apache/dolphinscheduler/blob/dev/.github/ISSUE_TEMPLATE/bug-report.yml)
issue template have **Version** selection bottom. So after released we should and the new `<VERSION>` to
issue template have **Version** selection bottom. So after released we should add the new `<VERSION>` to
bug-report.yml
### Find DolphinScheduler in [apache staging repositories](https://repository.apache.org/#stagingRepositories) and click `Release`
### Publish Docker Image
we already have the exists CI to publish the latest Docker image to GitHub container register with [config](https://github.com/apache/dolphinscheduler/blob/d80cf21456265c9d84e642bdb4db4067c7577fc6/.github/workflows/publish-docker.yaml#L55-L63).
We could reuse the main command the CI run and publish our Docker images to Docker Hub by single command.
It is highly recommended to build and test docker images locally first before push to docker hub
```shell
# Checkout and create to target tag
git checkout -b "${VERSION}" "${VERSION}"
# Build docker images locally
./mvnw -B clean package \
-Dmaven.test.skip \
-Dmaven.javadoc.skip \
-Dmaven.checkstyle.skip \
-Dmaven.deploy.skip \
-Ddocker.tag="${VERSION}" \
-Pdocker,release
# You should test whether the standalone-server images work or not
docker run --name dolphinscheduler-standalone-server -p 12345:12345 -p 25333:25333 -d apache/dolphinscheduler-standalone-server:"${DOLPHINSCHEDULER_VERSION}"
# If success, push to dockerhub
docker push apache/dolphinscheduler-tools:"${VERSION}"
docker push apache/dolphinscheduler-standalone-server:"${VERSION}"
docker push apache/dolphinscheduler-master:"${VERSION}"
docker push apache/dolphinscheduler-worker:"${VERSION}"
docker push apache/dolphinscheduler-api:"${VERSION}"
docker push apache/dolphinscheduler-alert-server:"${VERSION}"
```
> Note: To push to dockerhub, you must have Apache organization permission of dockerhub. If you don’t you need to require
> from Apache infra Jira. You can refer to here to submit an application from [here](https://issues.apache.org/jira/projects/INFRA/issues/INFRA-23314)
>
> You can also build and push docker by single command if you make sure the images work fine
>
> ```shell
> ./mvnw -B clean deploy \
> -Dmaven.test.skip \
> -Dmaven.javadoc.skip \
> -Dmaven.checkstyle.skip \
> -Dmaven.deploy.skip \
> -Ddocker.tag="${VERSION}" \
> -Ddocker.hub=apache \
> -Pdocker,release
> ```
### Send Announcement E-mail Community
@ -513,3 +576,16 @@ DolphinScheduler Resources:
- Documents: https://dolphinscheduler.apache.org/zh-cn/docs/<VERSION>/user_doc/about/introduction.html
```
## News
After all set, an article should be written and publish it to the community, it should include:
* What is the major purpose of the version, feature add, bugfix, or both
* Major new features and how to use it, better with picture or gif
* Major bugfix and make what different from previous version, better with picture or gif
* All contributors since previous release
### Get All Contributors
You might need all contributors in current release when you want to publish the release news or announcement, you could
use command `python release.py contributor` in directory `tools/release` to auto generate contributor GitHub id.

26
docs/docs/en/contribute/release/release-post.md

@ -1,26 +0,0 @@
# Release Post
We still have some publish task to do after we send the announcement mail, currently we have to publish Docker images to
Docker Hub.
## Publish Docker Image
we already have the exists CI to publish the latest Docker image to GitHub container register with [config](https://github.com/apache/dolphinscheduler/blob/d80cf21456265c9d84e642bdb4db4067c7577fc6/.github/workflows/publish-docker.yaml#L55-L63).
We could reuse the main command the CI run and publish our Docker images to Docker Hub by single command.
```bash
# Please change the <VERSION> place hold to the version you release
./mvnw -B clean deploy \
-Dmaven.test.skip \
-Dmaven.javadoc.skip \
-Dmaven.checkstyle.skip \
-Dmaven.deploy.skip \
-Ddocker.tag=<VERSION> \
-Ddocker.hub=apache \
-Pdocker,release
```
## Get All Contributors
You might need all contributors in current release when you want to publish the release news or announcement, you could
use command `python release.py contributor` in directory `tools/release` to auto generate contributor GitHub id.

31
docs/docs/en/contribute/release/release-prepare.md

@ -1,31 +0,0 @@
# Release Preparation
## Check release-docs
Compared with the last release, the `release-docs` of the current release needs to be updated to the latest, if there are dependencies and versions changes
- `dolphinscheduler-dist/release-docs/LICENSE`
- `dolphinscheduler-dist/release-docs/NOTICE`
- `dolphinscheduler-dist/release-docs/licenses`
## Update Version
For example, to release `x.y.z`, the following updates are required:
- Version in the code:
- `sql`:
- `dolphinscheduler_mysql.sql`: `t_ds_version` needs to be updated to x.y.z
- `dolphinscheduler_postgre.sql`: `t_ds_version` needs to be updated to x.y.z
- `dolphinscheduler_h2.sql`: `t_ds_version` needs to be updated to x.y.z
- `upgrade`: whether to add`x.y.z_schema`
- `soft_version`: need to be updated to x.y.z
- `deploy/docker/.env`: `HUB` change to `apache`,`TAG` change to `x.y.z`
- `deploy/kubernetes/dolphinscheduler`:
- `Chart.yaml`: `appVersion` and `version` needs to be updated to x.y.z
- `values.yaml`: `image.tag` needs to be updated to x.y.z
- Version in the docs:
- Change the placeholder `<version>`(except `pom`) to the `x.y.z` in directory `docs`
- Add new history version
- `docs/docs/en/history-versions.md` and `docs/docs/zh/history-versions.md`: Add the new version and link for `x.y.z`
- `docs/configs/docsdev.js`: change `/dev/` to `/x.y.z/`, **DO NOT** change this filename, is will be auto change by website tools.

284
docs/docs/zh/contribute/release/release.md → docs/docs/zh/contribute/release.md

@ -1,6 +1,10 @@
# 发版指南
## 检查环境
## 准备
这部分是预发布工作,大部分是一次性的,意思是**它只在你的第一次发布时需要**。 如果你有之前发布过,请跳过本节到下一个部分。
### 检查环境
为确保您可以成功完成 DolphinScheduler 的发布,您应该检查您的环境并确保满足所有条件,如果缺少任何条件,您应该安装它们并确保它们正常工作。
@ -11,9 +15,9 @@ java -version
mvn -version
```
## GPG设置
### GPG设置
### 安装GPG
#### 安装GPG
在[GnuPG官网](https://www.gnupg.org/download/index.html)下载安装包。
GnuPG的1.x版本和2.x版本的命令有细微差别,下列说明以`GnuPG-2.1.23`版本为例。
@ -24,7 +28,7 @@ GnuPG的1.x版本和2.x版本的命令有细微差别,下列说明以`GnuPG-2.
gpg --version
```
### 创建key
#### 创建key
安装完成后,执行以下命令创建key。
@ -87,7 +91,7 @@ gpg: Key generation canceled.
需要使用自己的用户登录服务器,而不是root切到自己的账户
### 查看生成的key
#### 查看生成的key
```shell
gpg --list-keys
@ -103,7 +107,7 @@ sub 4096R/A63BC462 2019-11-15
其中85E11560为公钥ID。
### 将公钥同步到服务器
#### 将公钥同步到服务器
命令如下:
@ -117,9 +121,9 @@ gpg --keyserver hkp://pool.sks-keyservers.net --send-key 85E11560
http://keyserver.ubuntu.com:11371/pks/lookup?search=${用户名}&fingerprint=on&op=index
备用公钥服务器 gpg --keyserver hkp://keyserver.ubuntu.com --send-key ${公钥ID}
## 发布Apache Maven中央仓库
### 配置 Apache Maven Central Repository
### 设置 `settings-security.xml``settings.xml` 文件
#### 设置 `settings-security.xml``settings.xml` 文件
在本节中,我们添加 Apache 服务器 maven 配置以准备发布,请参考[这里](http://maven.apache.org/guides/mini/guide-encryption.html) 添加
`settings-security.xml` 文件,并且像下面这样更改你的 `~/.m2/settings.xml`
@ -141,33 +145,73 @@ http://keyserver.ubuntu.com:11371/pks/lookup?search=${用户名}&fingerprint=on&
</settings>
```
## Releasing
## 检查 release-docs
和上一个版本比较,如果有依赖及版本发生了变化,当前版本的 `release-docs` 需要被更新到最新
- `dolphinscheduler-dist/release-docs/LICENSE`
- `dolphinscheduler-dist/release-docs/NOTICE`
- `dolphinscheduler-dist/release-docs/licenses`
### 配置环境变量
我们将多次使用发布版本 `VERSION`,github名称 `GH_USERNAME`,以及 Apache 用户名 `<YOUR-APACHE-USERNAME>`,因此最好将其存储到bash变量中以便于使用。
```shell
VERSION=<THE-VERSION-YOU-RELEASE>
SOURCE_CODE_DIR=<YOUR-SOURCE-CODE-ROOT-DIR> # the directory of your source code hold, the location of parent pom.xml instead of binary package
GH_USERNAME=<YOUR-GITHUB-USERNAME>
GH_REMOTE=<GITHUB-REMOTE> # we use `upstream` or `origin` mostly base on your release environment
A_USERNAME=<YOUR-APACHE-USERNAME>
SVN_DIR=<PATH-TO-SVN-ROOT> # to keep binary package checkout from SVN, the sub path end with `/dolphinscheduler/dev` and `/dolphinscheduler/release` will be create
```
> 注意:设置环境变量后,我们可以直接在你的 bash 中使用该变量,而无需更改任何内容。例如,我们可以直接使用命令 `git clone -b "${VERSION}"-prepare https://github.com/apache/dolphinscheduler.git`
> 来克隆发布分支,他会自动将其中的 `"${VERSION}"` 转化成你设置的值 `<THE-VERSION-YOU-RELEASE>`。 但是您必须在一些非 bash 步骤中手动更改
> `<VERSION>` 为对应的版本号,例如发起投票中的内容。我们使用 `<VERSION>` 而不是 `"${VERSION}"` 来提示 release manager 他们必须手动更改这部分内容
### 创建发布分支
### 更新文档和代码的版本
我们需要在 Maven 发布之前更新一些文档。 例如,要发布版本 `VERSION`,需要进行以下更新:
在本节中,我们从 github 下载源代码并创建新分支以发布
- 修改代码中的版本号:
- `sql`:
- `dolphinscheduler_mysql.sql`: `t_ds_version` 版本更新为 x.y.z
- `dolphinscheduler_postgre.sql`: `t_ds_version` 版本更新为 x.y.z
- `dolphinscheduler_h2.sql`: `t_ds_version` 版本更新为 x.y.z
- `upgrade`: 是否新增 `x.y.z_schema` 文件夹,如果有一些升级的DDL或DML,如果没有添加任何DDL或DML可以跳过这一步。
- `soft_version`: 版本更新为 x.y.z
- `deploy/docker/.env`: `HUB` 改为 `apache`,`TAG` 改为 `x.y.z`
- `deploy/kubernetes/dolphinscheduler`:
- `Chart.yaml`: `appVersion``version` 版本更新为 x.y.z
- `values.yaml`: `image.tag` 版本更新为 x.y.z
- 修改文档(docs模块)中的版本号:
- 将 `docs` 文件夹下文件的占位符 `<version>` (除了 pom.xml 相关的) 修改成 `x.y.z`
- 新增历史版本
- `docs/docs/en/history-versions.md``docs/docs/zh/history-versions.md`: 增加新的历史版本为 `x.y.z`
- 修改文档 sidebar
- `docs/configs/docsdev.js`: 将里面的 `/dev/` 修改成 `/x.y.z/`,**不要**修改文件名称,website 仓库的 shell 脚本会对他进行修改
> 注意:`VERSION` 是一个占位字符串,与我们在 `VERSION=<THE-VERSION-YOU-RELEASE>` 中设置的版本相同。
### Maven 发布
#### Maven 发布检查
在准备分支的基础上创建发布分支。
```shell
git clone -b "${VERSION}"-prepare https://github.com/apache/dolphinscheduler.git
cd ~/dolphinscheduler/
git pull
git checkout -b "${VERSION}"-release
git push origin "${VERSION}"-release
cd "${SOURCE_CODE_DIR}"
git checkout -b "${VERSION}"-release "${VERSION}"-prepare
git push "${GH_REMOTE}" "${VERSION}"-release
```
### 发布预校验
> 注意:如果你在没有源代码的远程主机上发布,你应该先运行 `git clone -b "${VERSION}"-prepare https://github.com/apache/dolphinscheduler.git`
> 首先要克隆源代码。 然后确保设置`GH_REMOTE="origin"` 以使所有命令正常工作。
```shell
# 运行发版校验
@ -178,7 +222,7 @@ mvn release:prepare -Prelease -Darguments="-Dmaven.test.skip=true -Dcheckstyle.s
* `-DautoVersionSubmodules=true`: 作用是发布过程中版本号只需要输入一次,不必为每个子模块都输入一次。
* `-DdryRun=true`: 演练,即不产生版本号提交,不生成新的tag。
### 准备发布
#### 准备发布
首先清理发布预校验本地信息。
@ -202,8 +246,8 @@ mvn release:prepare -Prelease -Darguments="-Dmaven.test.skip=true -Dcheckstyle.s
将本地文件检查无误后,提交至github。
```shell
git push -u origin "${VERSION}"-release
git push origin --tags
git push -u "${GH_REMOTE}" "${VERSION}"-release
git push "${GH_REMOTE}" --tags
```
<!-- markdown-link-check-disable -->
@ -215,7 +259,7 @@ git push origin --tags
<!-- markdown-link-check-enable -->
### 部署发布
#### 部署发布
```shell
mvn release:perform -Prelease -Darguments="-Dmaven.test.skip=true -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true -Dspotless.check.skip=true" -DautoVersionSubmodules=true -Dusername="${GH_USERNAME}"
@ -225,33 +269,38 @@ mvn release:perform -Prelease -Darguments="-Dmaven.test.skip=true -Dcheckstyle.s
, 然后使用Apache的LDAP账户登录后,就会看到上传的版本,`Repository` 列的内容即为 `${STAGING.REPOSITORY}`
点击 `Close` 来告诉Nexus这个构建已经完成,只有这样该版本才是可用的。如果电子签名等出现问题,`Close` 会失败,可以通过 `Activity` 查看失败信息。
## 发布Apache SVN仓库
### SVN
### 检出dolphinscheduler发布目录
#### 检出dolphinscheduler发布目录
如无本地工作目录,则先创建本地工作目录。
我们还需要将 Dolphinscheduler 开发版本目录检出到本地,并且
```shell
mkdir -p ~/ds_svn/dev/
cd ~/ds_svn/dev/
```
SVN_DIR_DEV="${SVN_DIR}/dolphinscheduler/dev"
SVN_DIR_RELEASE="${SVN_DIR}/dolphinscheduler/release"
# 可选,只有当路径不存在时
mkdir -p "${SVN_DIR_DEV}"
创建完毕后,从Apache SVN检出dolphinscheduler发布目录。
```shell
svn --username="${A_USERNAME}" co https://dist.apache.org/repos/dist/dev/dolphinscheduler
cd ~/ds_svn/dev/dolphinscheduler
# When you first time checkout from this path
svn --username="${A_USERNAME}" co https://dist.apache.org/repos/dist/dev/dolphinscheduler "${SVN_DIR_DEV}"
# Or update when the svn directory exists, and you already checkout
svn --username="${A_USERNAME}" update "${SVN_DIR_DEV}"
```
### 将 gpg KEYS 文件拷贝至发布目录(可选)
> 注意:第一次结帐时可能需要几分钟才能同步到镜像,因为它会下载所有文件
#### 将 gpg KEYS 文件拷贝至发布目录(可选)
只有你第一次使用该 KEY 发版时才需要,如果之前已经发过版且 KEY 没有变化则不需要。你需要切换到一个新的目录,因为这步骤需要 checkout 并修改 release
库中的 KEYS 文件
```shell
mkdir -p ~/ds_svn/release/
cd ~/ds_svn/release/
# Optional, only if the SVN root path not exists.
mkdir -p "${SVN_DIR_RELEASE}"
cd "${SVN_DIR_RELEASE}"
svn --username="${A_USERNAME}" co https://dist.apache.org/repos/dist/release/dolphinscheduler
# Change the placeholder <YOUR-GPG-KEY-ID> to your id
gpg -a --export <YOUR-GPG-KEY-ID> >> KEYS
svn add *
svn --username="${A_USERNAME}" commit -m "new key <YOUR-GPG-KEY-ID> add"
@ -259,83 +308,40 @@ svn --username="${A_USERNAME}" commit -m "new key <YOUR-GPG-KEY-ID> add"
> 注意:这个步骤需要一定的时间去 checkout 特别是在你第一次 checkout 的时候,因为这个库比较大,且这个操作需要 checkout 出全部文件
### 将待发布的内容添加至SVN目录
#### 将待发布的内容添加至SVN目录
创建版本号目录。
按版本号创建文件夹,将源码包、二进制包、可执行二进制包移动到SVN工作目录。
```shell
mkdir -p ~/ds_svn/dev/dolphinscheduler/"${VERSION}"
cd ~/ds_svn/dev/dolphinscheduler/"${VERSION}"
```
将源码包和二进制包添加至SVN工作目录。
mkdir -p "${SVN_DIR_DEV}/${VERSION}"
```shell
# 主程序源码包和二进制包
cp -f ~/dolphinscheduler/dolphinscheduler-dist/target/*.tar.gz ~/ds_svn/dev/dolphinscheduler/"${VERSION}"
cp -f ~/dolphinscheduler/dolphinscheduler-dist/target/*.tar.gz.asc ~/ds_svn/dev/dolphinscheduler/"${VERSION}"
```
# Add to SVN
cp -f "${SOURCE_CODE_DIR}"/dolphinscheduler-dist/target/*.tar.gz "${SVN_DIR_DEV}/${VERSION}"
cp -f "${SOURCE_CODE_DIR}"/dolphinscheduler-dist/target/*.tar.gz.asc "${SVN_DIR_DEV}/${VERSION}"
### 生成文件签名
```shell
# Create sign
cd "${SVN_DIR_DEV}"
shasum -a 512 apache-dolphinscheduler-"${VERSION}"-src.tar.gz >> apache-dolphinscheduler-"${VERSION}"-src.tar.gz.sha512
shasum -b -a 512 apache-dolphinscheduler-"${VERSION}"-bin.tar.gz >> apache-dolphinscheduler-"${VERSION}"-bin.tar.gz.sha512
```
### 提交Apache SVN
```shell
cd ~/ds_svn/dev/dolphinscheduler
svn add *
svn --username="${A_USERNAME}" commit -m "release ${VERSION}"
```
## 检查发布结果
### 检查sha512哈希
```shell
# Check sha512
shasum -c apache-dolphinscheduler-"${VERSION}"-src.tar.gz.sha512
shasum -c apache-dolphinscheduler-"${VERSION}"-bin.tar.gz.sha512
```
### 检查gpg签名
首先导入发布人公钥。从svn仓库导入KEYS到本地环境。(发布版本的人不需要再导入,帮助做验证的人需要导入,用户名填发版人的即可)
```shell
curl https://dist.apache.org/repos/dist/release/dolphinscheduler/KEYS >> KEYS
gpg --import KEYS
gpg --edit-key "${A_USERNAME}"
> trust
Please decide how far you trust this user to correctly verify other users' keys
(by looking at passports, checking fingerprints from different sources, etc.)
1 = I don't know or won't say
2 = I do NOT trust
3 = I trust marginally
4 = I trust fully
5 = I trust ultimately
m = back to the main menu
Your decision? 5
> save
```
然后进行gpg签名检查。
```shell
# Check gpg signature
gpg --verify apache-dolphinscheduler-"${VERSION}"-src.tar.gz.asc
gpg --verify apache-dolphinscheduler-"${VERSION}"-bin.tar.gz.asc
# Commit to Apache SVN
svn add *
svn --username="${A_USERNAME}" commit -m "release ${VERSION}"
```
> 注意:当你找不到你的 `asc` 文件时,你必须手动创建 gpg 签名,命令
> `gpg --armor --detach-sign --digest-algo=SHA512 apache-dolphinscheduler-"${VERSION}"-bin.tar.gz`
> `gpg --armor --detach-sign --digest-algo=SHA512 apache-dolphinscheduler-"${VERSION}"-src.tar.gz` 将创建它们
将源码包和二进制包添加至SVN工作目录。
### 检查发布文件内容
#### 检查源码包的文件内容
@ -375,14 +381,14 @@ changelog.
### 投票阶段
1. DolphinScheduler社区投票,发起投票邮件到`dev@dolphinscheduler.apache.org`。PMC需要先按照文档检查版本的正确性,然后再进行投票。
经过至少72小时并统计到至少3个`+1 并且没有-1 PMC member`票后,即可进入下一阶段。
DolphinScheduler社区投票,发起投票邮件到`dev@dolphinscheduler.apache.org`。PMC需要先按照文档检查版本的正确性,然后再进行投票。 经过
至少72小时并统计到至少3个`+1 并且没有-1 PMC member`票后,即可进入下一阶段。
2. 宣布投票结果,发起投票结果邮件到`dev@dolphinscheduler.apache.org`。
宣布投票结果,发起投票结果邮件到`dev@dolphinscheduler.apache.org`。
### 投票模板
### 模板
1. DolphinScheduler社区投票模板
### VOTE 模板
标题:
@ -429,9 +435,13 @@ Checklist for reference:
[ ] No compiled archives bundled in source archive.
```
2.宣布投票结果模板
#### RESULT 模版
正文:
Title:
```txt
[RESULT][VOTE] Release Apache DolphinScheduler <VERSION>
```
```txt
The vote to release Apache DolphinScheduler <VERSION> has passed.Here is the vote result,
@ -449,14 +459,24 @@ xxx
Thanks everyone for taking time to check this release and help us.
```
## 完成发布
## Announce
### 将源码和二进制包从svn的dev目录移动到release目录
### Move Packages to Release
```shell
# move to release directory
svn mv -m "release ${VERSION}" https://dist.apache.org/repos/dist/dev/dolphinscheduler/"${VERSION}" https://dist.apache.org/repos/dist/release/dolphinscheduler/
# remove old release directory
svn delete -m "remove old release" https://dist.apache.org/repos/dist/release/dolphinscheduler/<PREVIOUS-RELEASE-VERSION>
# Remove prepare branch
cd "${SOURCE_CODE_DIR}"
git push --delete "${GH_REMOTE}" "${VERSION}-prepare"
```
在 [apache staging repositories](https://repository.apache.org/#stagingRepositories) 仓库找到 DolphinScheduler 并点击`Release`
### 更新文档
官网应该在您发送通知邮件之前完成更新,本节将告诉您如何更改网站。假设发版的版本是 `<VERSION>`,需要进行以下更新(注意,当修改pull requests 被 merge 后就会生效):
@ -473,7 +493,53 @@ svn mv -m "release ${VERSION}" https://dist.apache.org/repos/dist/dev/dolphinsch
- `.github/ISSUE_TEMPLATE/bug-report.yml`: DolphinScheduler 在 GitHub bug report 的 issue 中有版本选择,当有新的版本发版后,需要更新
[bug-report](https://github.com/apache/dolphinscheduler/blob/dev/.github/ISSUE_TEMPLATE/bug-report.yml) 中的 **Version** 部分。
### 在 [apache staging repositories](https://repository.apache.org/#stagingRepositories) 仓库找到 DolphinScheduler 并点击`Release`
### 发布 Docker Image
我们已经有了现有的 CI,可以将最新的 Docker image 发布到 GitHub 容器注册 [config](https://github.com/apache/dolphinscheduler/blob/d80cf21456265c9d84e642bdb4db4067c7577fc6/.github/workflows/publish-docker.yaml#L55- L63).
我们可以重用 CI 运行的主要命令,并通过单个命令将 Docker iamge 发布到 DockerHub。
强烈建议在推送到 docker hub 之前先在本地构建和测试 docker 镜像
```shell
# Checkout and create to target tag
git checkout -b "${VERSION}" "${VERSION}"
# Build docker images locally
./mvnw -B clean package \
-Dmaven.test.skip \
-Dmaven.javadoc.skip \
-Dmaven.checkstyle.skip \
-Dmaven.deploy.skip \
-Ddocker.tag="${VERSION}" \
-Pdocker,release
# You should test whether the standalone-server images work or not
docker run --name dolphinscheduler-standalone-server -p 12345:12345 -p 25333:25333 -d apache/dolphinscheduler-standalone-server:"${DOLPHINSCHEDULER_VERSION}"
# If success, push to dockerhub
docker push apache/dolphinscheduler-tools:"${VERSION}"
docker push apache/dolphinscheduler-standalone-server:"${VERSION}"
docker push apache/dolphinscheduler-master:"${VERSION}"
docker push apache/dolphinscheduler-worker:"${VERSION}"
docker push apache/dolphinscheduler-api:"${VERSION}"
docker push apache/dolphinscheduler-alert-server:"${VERSION}"
```
> 注意:推送到dockerhub,必须有dockerhub的Apache组织权限。 如果你不需要,你需要向 Apache infra Jira 申请。 您可以参考
> [此处](https://issues.apache.org/jira/projects/INFRA/issues/INFRA-23314)提交申请
>
> 如果您确保 docker image 正常工作,您也可以通过单个命令构建和推送 docker
>
> ```shell
> ./mvnw -B clean deploy \
> -Dmaven.test.skip \
> -Dmaven.javadoc.skip \
> -Dmaven.checkstyle.skip \
> -Dmaven.deploy.skip \
> -Ddocker.tag="${VERSION}" \
> -Ddocker.hub=apache \
> -Pdocker,release
> ```
### 发送公告邮件通知社区
@ -510,3 +576,15 @@ DolphinScheduler Resources:
- Documents: https://dolphinscheduler.apache.org/zh-cn/docs/<VERSION>/user_doc/about/introduction.html
```
## News
一切就绪后,应该写一篇文章发布到社区,它应该包括:
* 版本、功能添加、错误修复或两者的主要目的是什么
* 主要新功能及使用方法,最好有图片或gif
* 主要错误修复和与之前版本不同的地方,最好有图片或gif
* 自上一版本以来的所有贡献者
## 获取全部的贡献者
当您想要发布新版本的新闻或公告时,您可能需要当前版本的所有贡献者,您可以在 `tools/release` 中使用命令 `python release.py contributor` 自动生成贡献者 Github id。

24
docs/docs/zh/contribute/release/release-post.md

@ -1,24 +0,0 @@
# 发版后续
发送公告邮件后,我们还有一些发布任务要做,目前我们必须将 Docker 镜像发布到 Docker Hub。
## 发布 Docker 镜像
我们已经有 CI 发布最新的 Docker 镜像到 GitHub container register [点击查看详情](https://github.com/apache/dolphinscheduler/blob/d80cf21456265c9d84e642bdb4db4067c7577fc6/.github/workflows/publish-docker.yaml#L55-L63)。
我们可以稍微修改 CI 的主要命令实现单个命令发布 Docker 镜像发布到 Docker Hub。
```bash
# 请将 <VERSION> 修改成你要发版的版本
./mvnw -B clean deploy \
-Dmaven.test.skip \
-Dmaven.javadoc.skip \
-Dmaven.checkstyle.skip \
-Dmaven.deploy.skip \
-Ddocker.tag=<VERSION> \
-Ddocker.hub=apache \
-Pdocker,release
```
## 获取全部的贡献者
当您想要发布新版本的新闻或公告时,您可能需要当前版本的所有贡献者,您可以在 `tools/release` 中使用命令 `python release.py contributor` 自动生成贡献者 Github id。

32
docs/docs/zh/contribute/release/release-prepare.md

@ -1,32 +0,0 @@
# 发版准备
## 检查 release-docs
和上一个版本比较,如果有依赖及版本发生了变化,当前版本的 `release-docs` 需要被更新到最新
- `dolphinscheduler-dist/release-docs/LICENSE`
- `dolphinscheduler-dist/release-docs/NOTICE`
- `dolphinscheduler-dist/release-docs/licenses`
## 更新版本
例如要发版 `x.y.z`,需要先进行以下修改:
- 修改代码中的版本号:
- `sql`:
- `dolphinscheduler_mysql.sql`: `t_ds_version` 版本更新为 x.y.z
- `dolphinscheduler_postgre.sql`: `t_ds_version` 版本更新为 x.y.z
- `dolphinscheduler_h2.sql`: `t_ds_version` 版本更新为 x.y.z
- `upgrade`: 是否新增 `x.y.z_schema`
- `soft_version`: 版本更新为 x.y.z
- `deploy/docker/.env`: `HUB` 改为 `apache`,`TAG` 改为 `x.y.z`
- `deploy/kubernetes/dolphinscheduler`:
- `Chart.yaml`: `appVersion``version` 版本更新为 x.y.z
- `values.yaml`: `image.tag` 版本更新为 x.y.z
- 修改文档(docs模块)中的版本号:
- 将 `docs` 文件夹下文件的占位符 `<version>` (除了 pom.xml 相关的) 修改成 `x.y.z`
- 新增历史版本
- `docs/docs/en/history-versions.md``docs/docs/zh/history-versions.md`: 增加新的历史版本为 `x.y.z`
- 修改文档 sidebar
- `docs/configs/docsdev.js`: 将里面的 `/dev/` 修改成 `/x.y.z/`,**不要**修改文件名称,website 仓库的 shell 脚本会对他进行修改
Loading…
Cancel
Save