Jiajie Zhong
2 years ago
committed by
GitHub
73 changed files with 3921 additions and 101 deletions
@ -0,0 +1,85 @@
|
||||
# DSIP |
||||
|
||||
DolphinScheduler Improvement Proposal (DSIP) 是对 Apache DolphinScheduler 代码库进行的重大改进。它不是为了小修小补存在的， |
||||
DSIP 的目的是通知社区完成或即将完成的重大变更。 |
||||
|
||||
## 怎样的修改应该被认定为 DSIP |
||||
|
||||
- 任何重大的新功能、重大改进、引入或删除组件 |
||||
- 任何公共接口的任何重大变化，例如 API接口、web ui 巨大变化 |
||||
|
||||
当一个 PR 或者 Issue 是否应该被认定为 DSIP 存疑时，如果有 committer 认为他应该纳入 DAIP 的范畴，那它就应该是 DSIP。 |
||||
|
||||
我们使用 GitHub Issue 和 Apache 邮件列表来记录和保存 DSIP，想要了解更多相关信息，您可以跳转到 当前的 DSIPs 以及 past DSIPs |
||||
|
||||
作为 DSIP，它应该包含如下部分： |
||||
|
||||
- 在 [dev@dolphinscheduler.apache.org][mail-to-dev] 中有一个以 `[DISCUSS][DSIP` 为开头的邮件。 |
||||
- 有一个打了 "DSIP" 标签的 GitHub Issue，并在描述中包含邮链接。 |
||||
|
||||
### 当前的 DSIPs |
||||
|
||||
当前的 DSIP 包括所有仍在进行中的 DSIP，您可以在 [当前的 DSIPs][current-DSIPs] 中找到他们 |
||||
|
||||
### 完结的 DSIPs |
||||
|
||||
完结的 DSIP，包括所有已完成或因某种原因终止的 DSIP，您可以在 [完结的 DSIPs][past-DSIPs] 中找到他们 |
||||
|
||||
## DSIP 的步骤 |
||||
|
||||
### 创建 GitHub Issue |
||||
|
||||
所有 DSIP 都应该起源于 GitHub Issue |
||||
|
||||
- 如果您确定你的问题是 DSIP，你可以在 [GitHub Issue][github-issue-choose] 中点击并选择 "DSIP" |
||||
- 如果您不确定您的问题是否是 DSIP，您可以在 [GitHub Issue][github-issue-choose] 单击并选择 "Feature request"。当DolphinScheduler |
||||
维护团队在查看 Issue 时认为他是 DSIP 时，会为 Issue 增加标签 "DSIP"。 |
||||
|
||||
You should and special prefix `[DSIP-XXX]`, `XXX` stand for the id DSIP. It's auto increment, and you could find the next |
||||
integer in [All DSIPs][all-DSIPs] issues. |
||||
在您的问题被标记成 DSIP 后，您应该特殊前缀 `[DSIP-XXX]`，其中`XXX` 代表 id DSIP。它是自动递增的，你可以在 [All DSIPs][all-DSIPs] |
||||
找到下一个 DSIP 的整数编号。 |
||||
|
||||
### 发送讨论邮件 |
||||
|
||||
在您的问题被标记为 "DSIP" 后，您应该发送电子邮件至 [dev@dolphinscheduler.apache.org][mail-to-dev] 描述提案的目的，以及设计草案。 |
||||
|
||||
下面是邮件的模板 |
||||
|
||||
- 标题: `[DISCUSS][DSIP-XXX] <CHANGE-TO-YOUR-LOVELY-PROPOSAL-TITLE>`, 将 `XXX` 修改为 to special integer you just change in |
||||
GitHub Issue, and also change proposal title. |
||||
- 内容: |
||||
|
||||
```text |
||||
Hi community, |
||||
|
||||
<CHANGE-TO-YOUR-PROPOSAL-DETAIL> |
||||
|
||||
I already add a GitHub Issue for my proposal, which you could see in <CHANGE-TO-YOUR-GITHUB-ISSUE-LINK>. |
||||
|
||||
Looking forward any feedback for this thread. |
||||
``` |
||||
|
||||
在社区讨论并且所有人都认为它值得作为 DSIP 之后，您可以去到下节正式开始工作。但是如果社区认为它不应该是 DSIP，维护者需要终止邮件讨论并 |
||||
删除 GitHub Issue 中的 "DSIP" 标签。如果当这个修改不应该合并到 DolphinScheduler 中时，维护者除了除了移除标签外，还要关闭 GitHub Issue。 |
||||
|
||||
### 开始开发或者为他创建子任务 |
||||
|
||||
当您的提案通过邮件讨论时，您可以开始工作。你可以提交一个相关的 pull requests 如果更改应该在一次提交中进行。如果提案太大，已经超过了单次 |
||||
提交的范畴，你可以在 GitHub Issue 中创建子任务，如 [DSIP-1][DSIP-1]，并分成多个 pull requests 提交任务。 |
||||
|
||||
### 关闭 DSIP |
||||
|
||||
当 DSIP 完成并合并所有相关 PR 后，您应该回复您在第二步创建的邮件讨论，通知社区 DSIP 的结果。在这之后，这个 DSIP GitHub Issue 将会被 |
||||
关闭，并从 [当前的 DSIPs][current-DSIPs] 转移到 [完结的 DSIPs][past-DSIPs]，但您仍然可以在 [All DSIPs][all-DSIPs] 中找到它 |
||||
|
||||
## DSIP的例子 |
||||
|
||||
* [[DSIP-1][Feature][Parent] Add Python API for DolphinScheduler][DSIP-1]: 有多个子任务和项目。 |
||||
|
||||
[all-DSIPs]: https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue+label%3A%22DSIP%22+ |
||||
[current-DSIPs]: https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue+is%3Aopen+label%3A%22DSIP%22 |
||||
[past-DSIPs]: https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue+is%3Aclosed+label%3A%22DSIP%22+ |
||||
[github-issue-choose]: https://github.com/apache/dolphinscheduler/issues/new/choose |
||||
[mail-to-dev]: mailto:dev@dolphinscheduler.apache.org |
||||
[DSIP-1]: https://github.com/apache/dolphinscheduler/issues/6407 |
||||
@ -0,0 +1,42 @@
|
||||
# License Notice |
||||
|
||||
As we know that DolphinScheduler is an open-source undergoing project at The Apache Software Foundation (ASF), which means that you have to follow the Apache way to become the DolphinScheduler contributor. Furthermore, Apache has extremely strict rules according to the License. This passage will explain the ASF license and how to avoid License risks at the early stage when you participate in DolphinScheduler. |
||||
|
||||
Note: This article only applies to the Apache projects. |
||||
|
||||
### Licenses Could be Accepted to the Apache Project |
||||
|
||||
You have to pay attention to the following open-source software protocols which Apache projects support when you intend to add a new feature to the DolphinScheduler (or other Apache projects), which functions refers to other open-source software references. |
||||
|
||||
[ASF 3RD PARTY LICENSE POLICY](https://apache.org/legal/resolved.html) |
||||
|
||||
If the 3rd party software is not present at the above policy, we are sorry that your code can not pass the audit and we suggest searching for other substitute plans. |
||||
|
||||
Besides, when you demand new dependencies in the project, please email us about the reason and the outcome of the influence to dev@dolphinscheduler.apache.org to discuss. Besides, you need at least 3 positive votes from the PPMC to finish the whole step. |
||||
|
||||
### How to Legally Use 3rd Party Open-source Software in the DolphinScheduler |
||||
|
||||
Moreover, when we intend to refer a new software ( not limited to 3rd party jar, text, CSS, js, pics, icons, audios etc and modifications based on 3rd party files) to our project, we need to use them legally in addition to the permission of ASF. Refer to the following article: |
||||
|
||||
* [COMMUNITY-LED DEVELOPMENT "THE APACHE WAY"](https://apache.org/dev/licensing-howto.html) |
||||
|
||||
|
||||
For example, we should contain the NOTICE file (every open-source project has NOTICE file, generally under root directory) of ZooKeeper in our project when we are using ZooKeeper. As the Apache explains, "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work. |
||||
|
||||
We are not going to dive into every 3rd party open-source license policy, you may look up them if interested. |
||||
|
||||
### DolphinScheduler-License Check Rules |
||||
|
||||
In general, we would have our License-check scripts to our project. DolphinScheduler-License is provided by [kezhenxu94](https://github.com/kezhenxu94) which differ a bit from other open-source projects. All in all, we are trying to make sure avoiding the license issues at the first time. |
||||
|
||||
We need to follow the following steps when we need to add new jars or external resources: |
||||
|
||||
* Add the name and the version of the jar file in the known-dependencies.txt |
||||
* Add relevant maven repository address under 'dolphinscheduler-dist/release-docs/LICENSE' directory |
||||
* Append relevant NOTICE files under 'dolphinscheduler-dist/release-docs/NOTICE' directory and make sure they are no different to the original repository |
||||
* Add relevant source code protocols under 'dolphinscheduler-dist/release-docs/license/' directory and the file name should be named as license+filename.txt. Eg: license-zk.txt |
||||
|
||||
### References |
||||
|
||||
* [COMMUNITY-LED DEVELOPMENT "THE APACHE WAY"](https://apache.org/dev/licensing-howto.html) |
||||
* [ASF 3RD PARTY LICENSE POLICY](https://apache.org/legal/resolved.html) |
||||
@ -0,0 +1,11 @@
|
||||
# How to Become DolphinScheduler Committer |
||||
|
||||
Anyone can be a contributor to an Apache project. Being a contributor simply means that you take an interest in the project and contribute in some way, ranging from asking sensible questions (which documents the project and provides feedback to developers) through to providing new features as patches. |
||||
|
||||
If you become a valuable contributor to the project you may well be invited to become a committer. Committer is a term used at the ASF to signify someone who is committed to a particular project. It brings with it the privilege of write access to the project repository and resources. |
||||
|
||||
In Dolphinscheduler community, if a committer who have earned even more merit, can be invited to be a part of the Project Management Committee (PMC). |
||||
|
||||
One thing that is sometimes hard to understand when you are new to the open development process used at the ASF, is that we value the community more than the code. A strong and healthy community will be respectful and be a fun and rewarding place. More importantly, a diverse and healthy community can continue to support the code over the longer term, even as individual companies come and go from the field. |
||||
|
||||
More details could be found [here](https://community.apache.org/contributors/). |
||||
@ -0,0 +1,68 @@
|
||||
# Code of Conduct |
||||
|
||||
The following Code of Conduct is based on full compliance with the [Apache Software Foundation Code of Conduct](https://www.apache.org/foundation/policies/conduct.html). |
||||
|
||||
## Development philosophy |
||||
- **Consistent** code style, naming, and usage are consistent. |
||||
- **Easy to read** code is obvious, easy to read and understand, when debugging one knows the intent of the code. |
||||
- **Neat** agree with the concepts of《Refactoring》and《Code Cleanliness》and pursue clean and elegant code. |
||||
- **Abstract** hierarchy is clear and the concepts are refined and reasonable. Keep methods, classes, packages, and modules at the same level of abstraction. |
||||
- **Heart** Maintain a sense of responsibility and continue to be carved in the spirit of artisans. |
||||
|
||||
## Development specifications |
||||
|
||||
- Executing `mvn -U clean package -Prelease` can compile and test through all test cases. |
||||
- The test coverage tool checks for no less than dev branch coverage. |
||||
- In the root directory, use Checkstyle to check your code for special reasons for violating validation rules. The template location is located at ds_check_style.xml. |
||||
- Follow the coding specifications. |
||||
|
||||
## Coding specifications |
||||
|
||||
- Use linux line breaks. |
||||
- Indentation (including empty lines) is consistent with the last line. |
||||
- An empty line is required between the class declaration and the following variable or method. |
||||
- There should be no meaningless empty lines. |
||||
- Classes, methods, and variables should be named as the name implies and abbreviations should be avoided. |
||||
- Return value variables are named after `result`; `each` is used in loops to name loop variables; and `entry` is used in map instead of `each`. |
||||
- The cached exception is called `e`; Catch the exception and do nothing, and the exception is named `ignored`. |
||||
- Configuration Files are named in camelCase, and file names are lowercase with uppercase initial/starting letter. |
||||
- Code that requires comment interpretation should be as small as possible and interpreted by method name. |
||||
- `equals` and `==` In a conditional expression, the constant is left, the variable is on the right, and in the expression greater than less than condition, the variable is left and the constant is right. |
||||
- In addition to the abstract classes used for inheritance, try to design the class as `final`. |
||||
- Nested loops are as much a method as possible. |
||||
- The order in which member variables are defined and the order in which parameters are passed is consistent across classes and methods. |
||||
- Priority is given to the use of guard statements. |
||||
- Classes and methods have minimal access control. |
||||
- The private method used by the method should follow the method, and if there are multiple private methods, the writing private method should appear in the same order as the private method in the original method. |
||||
- Method entry and return values are not allowed to be `null`. |
||||
- The return and assignment statements of if else are preferred with the tri-objective operator. |
||||
- Priority is given to `LinkedList` and only use `ArrayList` if you need to get element values in the collection through the index. |
||||
- Collection types such as `ArrayList`，`HashMap` that may produce expansion must specify the initial size of the collection to avoid expansion. |
||||
- Logs and notes are always in English. |
||||
- Comments can only contain `javadoc`, `todo` and `fixme`. |
||||
- Exposed classes and methods must have javadoc, other classes and methods and methods that override the parent class do not require javadoc. |
||||
|
||||
## Unit test specifications |
||||
|
||||
- Test code and production code are subject to the same code specifications. |
||||
- Unit tests are subject to AIR (Automatic, Independent, Repeatable) Design concept. |
||||
- Automatic: Unit tests should be fully automated, not interactive. Manual checking of output results is prohibited, `System.out`, `log`, etc. are not allowed, and must be verified with assertions. |
||||
- Independent: It is prohibited to call each other between unit test cases and to rely on the order of execution. Each unit test can be run independently. |
||||
- Repeatable: Unit tests cannot be affected by the external environment and can be repeated. |
||||
- Unit tests are subject to BCDE（Border, Correct, Design, Error) Design principles. |
||||
- Border (Boundary value test): The expected results are obtained by entering the boundaries of loop boundaries, special values, data order, etc. |
||||
- Correct (Correctness test): The expected results are obtained with the correct input. |
||||
- Design (Rationality Design): Design high-quality unit tests in combination with production code design. |
||||
- Error (Fault tolerance test): The expected results are obtained through incorrect input such as illegal data, abnormal flow, etc. |
||||
- If there is no special reason, the test needs to be fully covered. |
||||
- Each test case needs to be accurately asserted. |
||||
- Prepare the environment for code separation from the test code. |
||||
- Only jUnit `Assert`，hamcrest `CoreMatchers`，Mockito Correlation can use static import. |
||||
- Single-data assertions should use `assertTrue`，`assertFalse`，`assertNull` and `assertNotNull`. |
||||
- Multi-data assertions should use `assertThat`. |
||||
- Accurate assertion, try not to use `not`，`containsString` assertion. |
||||
- The true value of the test case should be named actualXXX, and the expected value should be named expectedXXX. |
||||
- Classes and Methods with `@Test` labels do not require javadoc. |
||||
|
||||
- Public specifications. |
||||
- Each line is no longer than `200` in length, ensuring that each line is semantically complete for easy understanding. |
||||
@ -0,0 +1,94 @@
|
||||
# Commit Message Notice |
||||
|
||||
### Preface |
||||
|
||||
A good commit message can help other developers (or future developers) quickly understand the context of related changes, and can also help project managers determine whether the commit is suitable for inclusion in the release. But when we checked the commit logs of many open source projects, we found an interesting problem. Some developers have very good code quality, but the commit message record is rather confusing. When other contributors or learners are viewing the code, it can’t be intuitively understood through commit log. |
||||
The purpose of the changes before and after the submission, as Peter Hutterer said：Re-establishing the context of a piece of code is wasteful. We can’t avoid it completely, so our efforts should go to reducing it as much as possible. Commit messages can do exactly that and as a result, a commit message shows whether a developer is a good collaborator. Therefore, DolphinScheduler developed the protocol in conjunction with other communities and official Apache documents. |
||||
|
||||
### Commit Message RIP |
||||
|
||||
#### 1：Clearly modify the content |
||||
|
||||
A commit message should clearly state what issues (bug fixes, function enhancements, etc.) the submission solves, so that other developers can better track the issues and clarify the optimization during the version iteration process. |
||||
|
||||
#### 2：Associate the corresponding Pull Request or Issue |
||||
|
||||
When our changes are large, the commit message should best be associated with the relevant Issue or Pull Request on GitHub, so that our developers can quickly understand the context of the code submission through the associated information when reviewing the code. If the current commit is for an issue, then the issue can be closed in the Footer section. |
||||
|
||||
#### 3：Unified format |
||||
|
||||
The formatted CommitMessage can help provide more historical information for quick browsing, and it can also generate a Change Log directly from commit. |
||||
|
||||
Commit message should include three parts: Header, Body and Footer. Among them, Header is required, Body and Footer can be omitted. |
||||
|
||||
##### Header |
||||
|
||||
The header part has only one line, including three fields: type (required), scope (optional), and subject (required). |
||||
|
||||
[DS-ISSUE number][type] subject |
||||
|
||||
(1) Type is used to indicate the category of commit, and only the following 7 types are allowed. |
||||
|
||||
- feat：New features |
||||
- fix：Bug fixes |
||||
- docs：Documentation |
||||
- style： Format (does not affect changes in code operation) |
||||
- refactor：Refactoring (It is not a new feature or a code change to fix a bug) |
||||
- test：Add test |
||||
- chore：Changes in the build process or auxiliary tools |
||||
|
||||
If the type is feat and fix, the commit will definitely appear in the change log. Other types (docs, chore, style, refactor, test) are not recommended. |
||||
|
||||
(2) Scope |
||||
|
||||
Scope is used to indicate the scope of commit impact, such as server, remote, etc. If there is no suitable scope, you can use \*. |
||||
|
||||
(3) subject |
||||
|
||||
Subject is a short description of the purpose of the commit, no more than 50 characters. |
||||
|
||||
##### Body |
||||
|
||||
The body part is a detailed description of this commit, which can be divided into multiple lines, and the line break will wrap with 72 characters to avoid automatic line wrapping affecting the appearance. |
||||
|
||||
Note the following points in the Body section: |
||||
|
||||
- Use the verb-object structure, note the use of present tense. For example, use change instead of changed or changes |
||||
|
||||
- Don't capitalize the first letter |
||||
|
||||
- The end of the sentence does not need a ‘.’ (period) |
||||
|
||||
##### Footer |
||||
|
||||
Footer only works in two situations |
||||
|
||||
(1) Incompatible changes |
||||
|
||||
If the current code is not compatible with the previous version, the Footer part starts with BREAKING CHANGE, followed by a description of the change, the reason for the change, and the migration method. |
||||
|
||||
(2) Close Issue |
||||
|
||||
If the current commit is for a certain issue, you can close the issue in the Footer section, or close multiple issues at once. |
||||
|
||||
##### For Example |
||||
|
||||
``` |
||||
[DS-001][docs-en] add commit message |
||||
|
||||
- commit message RIP |
||||
- build some conventions |
||||
- help the commit messages become clean and tidy |
||||
- help developers and release managers better track issues |
||||
and clarify the optimization in the version iteration |
||||
|
||||
This closes #001 |
||||
``` |
||||
|
||||
### Reference documents |
||||
|
||||
[Commit message format](https://cwiki.apache.org/confluence/display/GEODE/Commit+Message+Format) |
||||
|
||||
[On commit messages-Peter Hutterer](http://who-t.blogspot.com/2009/12/on-commit-messages.html) |
||||
|
||||
[RocketMQ Community Operation Conventions](https://mp.weixin.qq.com/s/LKM4IXAY-7dKhTzGu5-oug) |
||||
@ -0,0 +1,40 @@
|
||||
# Participate in Contributing |
||||
|
||||
First of all, thank you very much for choosing and using DolphinScheduler, and welcome to join the DolphinScheduler family! |
||||
|
||||
We encourage any form of participation in the community that will eventually become Committer or PPMC Such as: |
||||
* Problems will be encountered via github on the [issue](https://github.com/apache/dolphinscheduler/issues) form feedback out. |
||||
* Answer the issue questions that others are asking. |
||||
* Help improve the documentation. |
||||
* Help your project add test cases. |
||||
* Add comments to the code. |
||||
* Submit a PR that fixes the bug or Feature. |
||||
* Publish application case practice, scheduling process analysis, or technical articles related to scheduling. |
||||
* Help promote DolphinScheduler, participate in technical conferences or meetup, sharing and more. |
||||
|
||||
Welcome to the contributing team and join open source starting with submitting your first PR. |
||||
- For example, add code comments or find "easy to fix" tags or some very simple issue (misspellings, etc.) and so on, first familiarize yourself with the submission process through the first simple PR. |
||||
|
||||
Note: Contributions are not limited to PR Only, but contribute to the development of the project. |
||||
|
||||
I'm sure you'll benefit from open source by participating in DolphinScheduler! |
||||
|
||||
### 1. Participate in documentation contributions. |
||||
|
||||
Refer to the [Submit Guide-Document Notice](./document.md) |
||||
|
||||
### 2. Participate in code contributions. |
||||
|
||||
Refer to the [Submit Guide-Issue Notice](./issue.md), [Submit Guide-Pull Request Notice](./pull-request.md), [Submit Guide-Commit Message Notice](./commit-message.md) |
||||
|
||||
### 3. How to pick up an Issue and submit a Pull Request. |
||||
|
||||
If you want to implement a Feature or fix a Bug. Please refer to the following: |
||||
|
||||
* All Bugs and the new Features are recommended and managed using the Issues Page. |
||||
* If you want to develop a Feature, first reply to the Issue associated with that feature, indicating that you are currently working on it. And set yourself a "deadline" when to Submit the Feature, and add it in the reply comment. |
||||
* It's a good idea to find a mentor (or an instructor) in the core contributors who gives immediate feedback on design and functional implementation. |
||||
* You should create a new branch to start your work, to get the name of the branch refer to the [Submit Guide-Pull Request Notice](./pull-request.md). For example, if you want to complete the feature and submit Issue 111, your branch name should be feature-111. The feature name can be determined after discussion with the instructor. |
||||
* When you're done, send a Pull Request to dolphinscheduler, please refer to the《[Submit Guide-Submit Pull Request Process](./submit-code.md)》 |
||||
|
||||
If you want to submit a Pull Request to complete a Feature or fix a Bug, it is recommended that you start with the `good first issue`, `easy-to-fix` issues, complete a small function to submit, do not change too many files at a time, changing too many files will also put a lot of pressure on Reviewers, it is recommended to submit them through multiple Pull Requests, not all at once. |
||||
@ -0,0 +1,62 @@
|
||||
# Documentation Notice |
||||
|
||||
Good documentation is critical for any type of software. Any contribution that can improve the DolphinScheduler documentation is welcome. |
||||
|
||||
### Get the document project |
||||
|
||||
Documentation for the DolphinScheduler project is maintained in a separate [git repository](https://github.com/apache/dolphinscheduler-website). |
||||
|
||||
First you need to fork the document project into your own github repository, and then clone the document to your local computer. |
||||
|
||||
``` |
||||
git clone https://github.com/<your-github-user-name>/dolphinscheduler-website |
||||
``` |
||||
|
||||
### The document environment |
||||
|
||||
The DolphinScheduler website is supported by [docsite](https://github.com/chengshiwen/docsite-ext) |
||||
|
||||
Make sure that your node version is 10+, docsite does not yet support versions higher than 10.x. |
||||
|
||||
### Document build guide |
||||
|
||||
1. Run `npm install` in the root directory to install the dependencies. |
||||
|
||||
2. Run commands to collect resources 2.1.Run `export PROTOCOL_MODE=ssh` tells Git clone resource via SSH protocol instead of HTTPS protocol. 2.2.Run `./scripts/prepare_docs.sh` prepare all related resources, for more information you could see [how prepare script work](https://github.com/apache/dolphinscheduler-website/blob/master/HOW_PREPARE_WOKR.md). |
||||
|
||||
3. Run `npm run start` in the root directory to start a local server, you will see the website in 'http://localhost:8080'. |
||||
|
||||
4. Run `npm run build` to build source code into dist directory. |
||||
|
||||
5. Verify your change locally: `python -m SimpleHTTPServer 8000`, when your python version is 3 use :`python3 -m http.server 8000` instead. |
||||
|
||||
If the latest version of node is installed locally, consider using `nvm` to allow different versions of `node` to run on your computer. |
||||
|
||||
1. Refer to the [Instructions](http://nvm.sh) to install nvm. |
||||
|
||||
2. Run `nvm install v10.23.1` to install node v10. |
||||
|
||||
3. Run `nvm use v10.23.1` to switch the current working environment to node v10. |
||||
|
||||
Now you can run and build the website in your local environment. |
||||
|
||||
### The document specification |
||||
|
||||
1. ** Spaces are Required ** between Chinese characters and English or numbers and ** Spaces are not required ** between Chinese punctuation marks and English or numbers, to enhance the aesthetics and readability of the Chinese-English mix. |
||||
|
||||
2. It is recommended that you use "you" in general. Of course, you can use the term when necessary, such as when there is a warning prompt. |
||||
|
||||
### How to submit a document Pull Request |
||||
|
||||
1. Do not use "git add." to commit all changes. |
||||
|
||||
2. Simply push the changed files, for example: |
||||
|
||||
* `*.md` |
||||
* `blog.js or docs.js or site.js` |
||||
|
||||
3. Submit the Pull Request to the **master** branch. |
||||
|
||||
### Reference to the documentation |
||||
|
||||
[Apache Flink Translation Specifications](https://cwiki.apache.org/confluence/display/FLINK/Flink+Translation+Specifications) |
||||
@ -0,0 +1,36 @@
|
||||
# DolphinScheduler E2E Test Contribution Guide |
||||
|
||||
The main purpose of E2E test is to verify the integration and data integrity of the system, and its components, by simulating real user scenarios. This makes it possible to extend the scope of testing and ensure the health and stability of the system. To a certain extent, the test workload and costs are reduced. In simple terms, E2E testing is about treating a program as a black box and simulating the access behaviour of a real system from the user's point of view to see if the input to the test (user behaviour/simulated data) gives the expected results. Therefore, the community decided to add E2E automated testing to DolphinScheduler. |
||||
|
||||
The current community E2E test has not yet reached full coverage, so this document has been written with the aim of guiding more partners to get involved. |
||||
|
||||
### How to find the corresponding ISSUE? |
||||
|
||||
The E2E test pages are currently divided into four pages: Project Management, Resource Center, DataSource, and Security Center. |
||||
|
||||
Contributors can find the task by going to GitHub, searching for [apace/dolphinscheduler](https://github.com/apache/dolphinscheduler), and then searching for `e2e test cases` in the [issue list](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue+is%3Aopen+e2e+test+cases). As shown below. |
||||
|
||||
 |
||||
|
||||
In each issue, we list the content to be tested and the expected results, which can be seen in the Description. Once you are on the current page, you can select the issue you are interested in, for example, to participate in the Security Center test, just leave a comment under the corresponding issue about the case you want to test. |
||||
|
||||
- Task status: If the test has been completed, it is considered finished and as a contributor you need to find the outstanding tests. |
||||
- number: Test case serial number. |
||||
- function module: Functional modules to be tested, a functional module containing multiple test cases. |
||||
- test point: Specific examples of what needs to be tested. For example button click actions on pages, page jump functions. |
||||
- priority: The priority of the test case, **which is recommended to find a high priority**. |
||||
- service: The service that needs to be started during the test process. |
||||
- test steps: The test steps to be performed for each test case. |
||||
- expected results: The expected test results. |
||||
- actual results: The actual results. |
||||
- remarks: The precautions needed during the test process. |
||||
|
||||
 |
||||
|
||||
### How to write test code? |
||||
|
||||
After taking up the corresponding task, the next step is to get to the crux of writing the code. Many partners may not be familiar with E2E test codes, so you can refer to this page: [e2e-test](https://dolphinscheduler.apache.org/en-us/development/e2e-test.html). |
||||
|
||||
### How to submit a Pull Request? |
||||
|
||||
Participation in the open source community can take many forms, including issue, pull request and translation. To participate in the E2E testing process, contributors are first required to understand the simple process of submitting a pull request, see: [Pull Request](./pull-request.md). |
||||
@ -0,0 +1,136 @@
|
||||
# Issue Notice |
||||
|
||||
## Preface |
||||
Issues function is used to track various Features, Bugs, Functions, etc. The project maintainer can organize the tasks to be completed through issues. |
||||
|
||||
Issue is an important step in drawing out a feature or bug, |
||||
and the contents that can be discussed in an issue are not limited to the features, the causes of the existing bugs, the research on preliminary scheme, and the corresponding implementation design and code design. |
||||
|
||||
And only when the Issue is approved, the corresponding Pull Request should be implemented. |
||||
|
||||
If an issue corresponds to a large feature, it is recommended to divide it into multiple small issues according to the functional modules and other dimensions. |
||||
|
||||
## Specification |
||||
|
||||
### Issue title |
||||
|
||||
Title Format: [`Issue Type`][`Module Name`] `Issue Description` |
||||
|
||||
The `Issue Type` is as follows: |
||||
|
||||
<table> |
||||
<thead> |
||||
<tr> |
||||
<th style="width: 10%; text-align: center;">Issue Type</th> |
||||
<th style="width: 20%; text-align: center;">Description</th> |
||||
<th style="width: 20%; text-align: center;">Example</th> |
||||
</tr> |
||||
</thead> |
||||
<tbody> |
||||
<tr> |
||||
<td style="text-align: center;">Feature</td> |
||||
<td style="text-align: center;">Include expected new features and functions</td> |
||||
<td style="text-align: center;">[Feature][api] Add xxx api in xxx controller</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">Bug</td> |
||||
<td style="text-align: center;">Bugs in the program</td> |
||||
<td style="text-align: center;">[Bug][api] Throw exception when xxx</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">Improvement</td> |
||||
<td style="text-align: center;">Some improvements of the current program, not limited to code format, program performance, etc</td> |
||||
<td style="text-align: center;">[Improvement][server] Improve xxx between Master and Worker</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">Test</td> |
||||
<td style="text-align: center;">Specifically for the test case</td> |
||||
<td style="text-align: center;">[Test][server] Add xxx e2e test</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">Sub-Task</td> |
||||
<td style="text-align: center;">Those generally are subtasks of feature class. For large features, they can be divided into many small subtasks to complete one by one</td> |
||||
<td style="text-align: center;">[Sub-Task][server] Implement xxx in xxx</td> |
||||
</tr> |
||||
</tbody> |
||||
</table> |
||||
|
||||
The `Module Name` is as follows: |
||||
|
||||
<table> |
||||
<thead> |
||||
<tr> |
||||
<th style="width: 10%; text-align: center;">Module Name</th> |
||||
<th style="width: 20%; text-align: center;">Description</th> |
||||
</tr> |
||||
</thead> |
||||
<tbody> |
||||
<tr> |
||||
<td style="text-align: center;">alert</td> |
||||
<td style="text-align: center;">Alert module</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">api</td> |
||||
<td style="text-align: center;">Application program interface layer module</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">service</td> |
||||
<td style="text-align: center;">Application service layer module</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">dao</td> |
||||
<td style="text-align: center;">Application data access layer module</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">plugin</td> |
||||
<td style="text-align: center;">Plugin module</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">remote</td> |
||||
<td style="text-align: center;">Communication module</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">server</td> |
||||
<td style="text-align: center;">Server module</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">ui</td> |
||||
<td style="text-align: center;">Front end module</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">docs-zh</td> |
||||
<td style="text-align: center;">Chinese document module</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">docs</td> |
||||
<td style="text-align: center;">English document module</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">...</td> |
||||
<td style="text-align: center;">-</td> |
||||
</tr> |
||||
</tbody> |
||||
</table> |
||||
|
||||
### Issue content template |
||||
|
||||
https://github.com/apache/dolphinscheduler/tree/dev/.github/ISSUE_TEMPLATE |
||||
|
||||
### Contributor |
||||
|
||||
Except for some special cases, it is recommended to discuss under issue or mailing list to determine the design scheme or provide the design scheme, |
||||
as well as the code implementation design before completing the issue. |
||||
|
||||
If there are many different solutions, it is suggested to make a decision through mailing list or voting under issue. |
||||
The issue can be implemented after final scheme and code implementation design being approved. |
||||
The main purpose of this is to avoid wasting time caused by different opinions on implementation design or reconstruction in the pull request review stage. |
||||
|
||||
### Question |
||||
|
||||
- How to deal with the user who raises an issue does not know the module corresponding to the issue. |
||||
|
||||
It is true that most users when raising issue do not know which module the issue belongs to. |
||||
In fact, this is very common in many open source communities. In this case, the committer / contributor actually knows the module affected by the issue. |
||||
If the issue is really valuable after being approved by committer and contributor, then the committer can modify the issue title according to the specific module involved in the issue, |
||||
or leave a message to the user who raises the issue to modify it into the corresponding title. |
||||
|
||||
@ -0,0 +1,100 @@
|
||||
# Micro BenchMark Notice |
||||
|
||||
All optimization must be based on data verification, and blind optimization is rejected. Based on this, we provide the MicroBench module. |
||||
|
||||
The MicroBench module is based on the OpenJDK JMH component (HotSpot's recommended benchmark test program). When you start benchmarking, you don't need additional dependencies. |
||||
|
||||
JMH, the Java MicroBenchmark Harness, is a tool suite dedicated to code microbenchmark testing. What is Micro Benchmark? Simply put, it is based on method-level benchmark testing, with an accuracy of microseconds. When you locate a hot method and want to further optimize the performance of the method, you can use JMH to quantitatively analyze the optimized results. |
||||
|
||||
### Several points to note in Java benchmark testing: |
||||
|
||||
- Prevent useless code from entering the test method. |
||||
|
||||
- Concurrent testing. |
||||
|
||||
- The test results are presented. |
||||
|
||||
### Typical application scenarios of JMH are: |
||||
|
||||
- 1: Quantitatively analyze the optimization effect of a hotspot function |
||||
|
||||
- 2: Want to quantitatively know how long a function needs to be executed, and the correlation between execution time and input variables |
||||
|
||||
- 3: Compare multiple implementations of a function |
||||
|
||||
DolphinScheduler-MicroBench provides AbstractBaseBenchmark, you can inherit from it, write your benchmark code, AbstractMicroBenchmark can guarantee to run in JUnit mode. |
||||
|
||||
### Customized operating parameters |
||||
|
||||
The default AbstractMicrobenchmark configuration is |
||||
|
||||
Warmup times 10 (warmupIterations) |
||||
|
||||
Number of tests 10 (measureIterations) |
||||
|
||||
Fork quantity 2 (forkCount) |
||||
|
||||
You can specify these parameters at startup，-DmeasureIterations, -DperfReportDir (output benchmark test result file directory), -DwarmupIterations, -DforkCount |
||||
|
||||
### DolphinScheduler-MicroBench Introduction |
||||
|
||||
It is generally not recommended to use fewer cycles when running tests. However, a smaller number of tests helps to verify the work during the benchmark test. After the verification is over, run a large number of benchmark tests. |
||||
|
||||
```java |
||||
@Warmup(iterations = 2, time = 1) |
||||
@Measurement(iterations = 4, time = 1) |
||||
@State(Scope.Benchmark) |
||||
public class EnumBenchMark extends AbstractBaseBenchmark { |
||||
|
||||
} |
||||
``` |
||||
|
||||
This can run benchmarks at the method level or the class level. Command line parameters will override the parameters on the annotation. |
||||
|
||||
```java |
||||
@Benchmark // Method annotation, indicating that the method is an object that needs to be benchmarked. |
||||
@BenchmarkMode(Mode.AverageTime) // Optional benchmark test mode is obtained through enumeration |
||||
@OutputTimeUnit(TimeUnit.MICROSECONDS) // Output time unit |
||||
public void enumStaticMapTest() { |
||||
TestTypeEnum.newGetNameByType(testNum); |
||||
} |
||||
``` |
||||
|
||||
When your benchmark test is written, you can run it to view the specific test conditions: (The actual results depend on your system configuration) |
||||
|
||||
First, it will warm up our code, |
||||
|
||||
```java |
||||
# Warmup Iteration 1: 0.007 us/op |
||||
# Warmup Iteration 2: 0.008 us/op |
||||
Iteration 1: 0.004 us/op |
||||
Iteration 2: 0.004 us/op |
||||
Iteration 3: 0.004 us/op |
||||
Iteration 4: 0.004 us/op |
||||
``` |
||||
|
||||
After warmup, we usually get the following results |
||||
|
||||
```java |
||||
Benchmark (testNum) Mode Cnt Score Error Units |
||||
EnumBenchMark.simpleTest 101 thrpt 8 428750972.826 ± 66511362.350 ops/s |
||||
EnumBenchMark.simpleTest 108 thrpt 8 299615240.337 ± 290089561.671 ops/s |
||||
EnumBenchMark.simpleTest 103 thrpt 8 288423221.721 ± 130542990.747 ops/s |
||||
EnumBenchMark.simpleTest 104 thrpt 8 236811792.152 ± 155355935.479 ops/s |
||||
EnumBenchMark.simpleTest 105 thrpt 8 472247775.246 ± 45769877.951 ops/s |
||||
EnumBenchMark.simpleTest 103 thrpt 8 455473025.252 ± 61212956.944 ops/s |
||||
EnumBenchMark.enumStaticMapTest 101 avgt 8 0.006 ± 0.003 us/op |
||||
EnumBenchMark.enumStaticMapTest 108 avgt 8 0.005 ± 0.002 us/op |
||||
EnumBenchMark.enumStaticMapTest 103 avgt 8 0.006 ± 0.005 us/op |
||||
EnumBenchMark.enumStaticMapTest 104 avgt 8 0.006 ± 0.004 us/op |
||||
EnumBenchMark.enumStaticMapTest 105 avgt 8 0.004 ± 0.001 us/op |
||||
EnumBenchMark.enumStaticMapTest 103 avgt 8 0.004 ± 0.001 us/op |
||||
EnumBenchMark.enumValuesTest 101 avgt 8 0.011 ± 0.004 us/op |
||||
EnumBenchMark.enumValuesTest 108 avgt 8 0.025 ± 0.016 us/op |
||||
EnumBenchMark.enumValuesTest 103 avgt 8 0.019 ± 0.010 us/op |
||||
EnumBenchMark.enumValuesTest 104 avgt 8 0.018 ± 0.018 us/op |
||||
EnumBenchMark.enumValuesTest 105 avgt 8 0.014 ± 0.012 us/op |
||||
EnumBenchMark.enumValuesTest 103 avgt 8 0.012 ± 0.009 us/op |
||||
``` |
||||
|
||||
OpenJDK officially gave a lot of sample codes, interested students can query and learn JMH by themselves:[OpenJDK-JMH-Example](http://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/) |
||||
@ -0,0 +1,94 @@
|
||||
# Pull Request Notice |
||||
|
||||
## Preface |
||||
Pull Request is a way of software cooperation, which is a process of bringing code involving different functions into the trunk. During this process, the code can be discussed, reviewed, and modified. |
||||
|
||||
In Pull Request, we try not to discuss the implementation of the code. The general implementation of the code and its logic should be determined in Issue. In the Pull Request, we only focus on the code format and code specification, so as to avoid wasting time caused by different opinions on implementation. |
||||
|
||||
## Specification |
||||
|
||||
### Pull Request Title |
||||
|
||||
Title Format: [`Pull Request Type`-`Issue No`][`Module Name`] `Pull Request Description` |
||||
|
||||
The corresponding relationship between `Pull Request Type` and `Issue Type` is as follows: |
||||
|
||||
<table> |
||||
<thead> |
||||
<tr> |
||||
<th style="width: 10%; text-align: center;">Issue Type</th> |
||||
<th style="width: 20%; text-align: center;">Pull Request Type</th> |
||||
<th style="width: 20%; text-align: center;">Example(Suppose Issue No is 3333)</th> |
||||
</tr> |
||||
</thead> |
||||
<tbody> |
||||
<tr> |
||||
<td style="text-align: center;">Feature</td> |
||||
<td style="text-align: center;">Feature</td> |
||||
<td style="text-align: center;">[Feature-3333][server] Implement xxx</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">Bug</td> |
||||
<td style="text-align: center;">Fix</td> |
||||
<td style="text-align: center;">[Fix-3333][server] Fix xxx</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">Improvement</td> |
||||
<td style="text-align: center;">Improvement</td> |
||||
<td style="text-align: center;">[Improvement-3333][alert] Improve the performance of xxx</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">Test</td> |
||||
<td style="text-align: center;">Test</td> |
||||
<td style="text-align: center;">[Test-3333][api] Add the e2e test of xxx</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">Sub-Task</td> |
||||
<td style="text-align: center;">(Parent type corresponding to Sub-Task)</td> |
||||
<td style="text-align: center;">[Feature-3333][server] Implement xxx</td> |
||||
</tr> |
||||
</tbody> |
||||
</table> |
||||
|
||||
`Issue No` refers to the Issue number corresponding to the current Pull Request to be resolved, `Module Name` is the same as the `Module Name` of Issue. |
||||
|
||||
### Pull Request Branch |
||||
|
||||
Branch name format: `Pull Request type`-`Issue number`. e.g. Feature-3333 |
||||
|
||||
### Pull Request Content |
||||
|
||||
Please refer to the commit message section. |
||||
|
||||
### Pull Request Code Style |
||||
|
||||
Code style is the thing you have to consider when you submit pull request for DolphinScheduler. We using [Checkstyle](https://checkstyle.sourceforge.io), a development tool to help programmers write Java code that adheres to a coding standard, in CI to keep DolphinScheduler codebase in the same style. Your pull request could not be merged if your code style checker failed. You could format your code by *Checkstyle* in your local environment before you submit your pull request to check code style. The activation step as below: |
||||
|
||||
1. Prepare Checkstyle configuration file: You could download it manually by [click here](https://github.com/apache/dolphinscheduler/blob/dev/style/checkstyle.xml), but find it in DolphinScheduler repository would be a better way. You could find configuration file in the path `style/checkstyle.xml` after you clone repository from Github. |
||||
|
||||
2. Download Checkstyle plugins in Intellij IDEA: Search plugin by keyword **CheckStyle-IDEA** or install in [this page](https://plugins.jetbrains.com/plugin/1065-checkstyle-idea). You could see [install plugin](https://www.jetbrains.com/help/idea/managing-plugins.html#install_plugin_from_repo) if you do not know how to install plugin in Intellij IDEA |
||||
|
||||
3. Configure and activate Checkstyle and Intellij IDEA code-style: After completing the above steps, you could configure and activate it in your environment. You could find Checkstyle plugins in the path `Preferences -> Tool -> Checkstyle`. After that you could activate Checkstyles as screenshot show |
||||
|
||||
<p align="center"> |
||||
<img src="../../../../img/contribute/join/pull-request/checkstyle-idea.png" alt="checkstyle idea configuration" /> |
||||
</p> |
||||
|
||||
For now your Checkstyle plugins are setup, it would show codes and files which out of style. We highly recommend you configure Intellij IDEA code-style for auto-formatting your code in Intellij IDEA, you could find this setting in `Preferences -> Editor -> Code Style -> Java` and then activate it as screenshot show |
||||
|
||||
<p align="center"> |
||||
<img src="../../../../img/contribute/join/pull-request/code-style-idea.png" alt="code style idea configuration" /> |
||||
</p> |
||||
|
||||
1. Format your codebase in Intellij IDEA before submit your pull request: After you done above steps, you could using Intellij IDEA shortcut `Command + L`(for Mac) or `Ctrl+L`(for Windows) to format your code. The best time to format your code is before you commit your change to your local git repository. |
||||
|
||||
### Question |
||||
|
||||
- How to deal with one Pull Request to many Issues scenario. |
||||
|
||||
First of all, there are fewer scenarios for one Pull Request to many Issues. |
||||
The root cause is that multiple issues need to do the same thing. |
||||
Usually, there are two solutions to this scenario: the first is to merge multiple issues with into the same issue, and then close the other issues; |
||||
the second is multiple issues have subtle differences. |
||||
In this scenario, the responsibilities of each issue can be clearly divided. The type of each issue is marked as Sub-Task, and then these sub task type issues are associated with one issue. |
||||
And each Pull Request is submitted should be associated with only one issue of a sub task. |
||||
@ -0,0 +1,153 @@
|
||||
# Community Review |
||||
|
||||
Beside submit Issues and pull requests to the GitHub repository mentioned in [team](/us-en/community/community.html), another important way to |
||||
contribute to DolphinScheduler is reviewing GitHub Issues or Pull Requests. You can not only know the latest new and |
||||
direction of the community, but also understand the good design in others during you reviewing. At the same time, you can |
||||
increase your exposure in the community and accumulate your honor. |
||||
|
||||
Anyone is encouraged to review Issues and Pull Requests. We also raise a Help Wanted email discussion to solicit contributors |
||||
from the community to review them. You could see detail in [mail][mail-review-wanted], we put the results of mail thread |
||||
in [GitHub Discussion][discussion-result-review-wanted]. |
||||
|
||||
> Note: It is only users mentioned in the [GitHub Discussion][discussion-result-review-wanted] can review Issues or Pull |
||||
> Requests, Community advocates **Anyone is encouraged to review Issues and Pull Requests**. Users in |
||||
> [GitHub Discussion][discussion-result-review-wanted] show their willing to review when we collect in the mail thread. |
||||
> The advantage of this list is when the community has discussion, in addition to the mention Members in [team](/us-en/community/community.html), |
||||
> you can also find some help in [GitHub Discussion][discussion-result-review-wanted] people. If you want to join the |
||||
> [GitHub Discussion][discussion-result-review-wanted], please comment in that discussion and leave a module you are interested |
||||
> in, and the maintainer will add you to the list. |
||||
|
||||
## How Reviewing |
||||
|
||||
DolphinScheduler receives community contributions through GitHub, and all its Issues and Pull Requests are hosted in GitHub. |
||||
If you want to join community by reviewing, please go to section [review Issues](#issues), if you prefer Pull Requests please |
||||
go to section [review Pull Requests](#pull-requests). |
||||
|
||||
### Issues |
||||
|
||||
Review Issues means discuss [Issues][all-issues] in GitHub and give suggestions on it. Include but are not limited to the following situations |
||||
|
||||
| Situation | Reason | Label | Action | |
||||
| ------ | ------ | ------ | ------ | |
||||
| wont fix | Has been fixed in dev branch | [wontfix][label-wontfix] | Close Issue, inform creator the fixed version if it already release | |
||||
| duplicate issue | Had the same problem before | [duplicate][label-duplicate] | Close issue, inform creator the link of same issue | |
||||
| Description not clearly | Without detail reproduce step | [need more information][label-need-more-information] | Inform creator add more description | |
||||
|
||||
In addition give suggestion, add label for issue is also important during review. The labeled issues can be retrieved |
||||
better, which convenient for further processing. An issue can with more than one label. Common issue categories are: |
||||
|
||||
| Label | Meaning | |
||||
| ------ | ------ | |
||||
| [UI][label-UI] | UI and front-end related | |
||||
| [security][label-security] | Security Issue | |
||||
| [user experience][label-user-experience] | User experience Issue | |
||||
| [development][label-development] | Development Issue | |
||||
| [Python][label-Python] | Python Issue | |
||||
| [plug-in][label-plug-in] | Plug-in Issue | |
||||
| [document][label-document] | Document Issue | |
||||
| [docker][label-docker] | Docker Issue | |
||||
| [need verify][label-need-verify] | Need verify Issue | |
||||
| [e2e][label-e2e] | E2E Issue | |
||||
| [win-os][label-win-os] | windows operating system Issue | |
||||
| [suggestion][label-suggestion] | Give suggestion to us | |
||||
|
||||
Beside classification, label could also set the priority of Issues. The higher the priority, the more attention pay |
||||
in the community, the easier it is to be fixed or implemented. The priority label are as follows |
||||
|
||||
| Label | priority | |
||||
| ------ | ------ | |
||||
| [priority:high][label-priority-high] | High priority | |
||||
| [priority:middle][label-priority-middle] | Middle priority | |
||||
| [priority:low][label-priority-low] | Low priority | |
||||
|
||||
All the labels above in common label. For all labels in this project you could see in [full label list][label-all-list] |
||||
|
||||
Before reading following content, please make sure you have labeled the Issue. |
||||
|
||||
* Remove label [Waiting for reply][label-waiting-for-reply] after replying: Label [Waiting for reply][label-waiting-for-reply] |
||||
added when [creating an Issue][issue-choose]. It makes positioning un reply issue more convenient, and you should remove |
||||
this label after you reviewed it. If you do not remove it, will cause others to waste time looking on the same issue. |
||||
* Mark [Waiting for review][label-waiting-for-review] when not sure whether issue is resolved or not: There are two situations |
||||
when you review issue. One is the problem has been located or resolved, maybe have to [Create PR](./submit-code.md) |
||||
when necessary. Secondly, you are not sure about this issue, you can labeled [Waiting for review][label-waiting-for-review] |
||||
and mention others to make a second confirmation. |
||||
|
||||
When an Issue need to create Pull Requests, you could also labeled it from below. |
||||
|
||||
| Label | Mean | |
||||
| ------ | ------ | |
||||
| [Chore][label-Chore] | Chore for project | |
||||
| [Good first issue][label-good-first-issue] | Good first issue for new contributor | |
||||
| [easy to fix][label-easy-to-fix] | Easy to fix, harder than `Good first issue` | |
||||
| [help wanted][label-help-wanted] | Help wanted | |
||||
|
||||
> Note: Only members have permission to add or delete label. When you need to add or remove lebals but are not member, |
||||
> you can `@` members to do that. But as long as you have a GitHub account, you can comment on issues and give suggestions. |
||||
> We encourage everyone in the community to comment and answer issues |
||||
|
||||
### Pull Requests |
||||
|
||||
<!-- markdown-link-check-disable --> |
||||
Review Pull mean discussing in [Pull Requests][all-PRs] in GitHub and giving suggestions to it. DolphinScheduler's |
||||
Pull Requests reviewing are the same as [GitHub's reviewing changes in pull requests][gh-review-pr]. You can give your |
||||
suggestions in Pull Requests |
||||
|
||||
* When you think the Pull Request is OK to be merged, you can agree to the Pull Request according to the "Approve" process |
||||
in [GitHub's reviewing changes in pull requests][gh-review-pr]. |
||||
* When you think Pull Request needs to be changed, you can comment it according to the "Comment" process in |
||||
[GitHub's reviewing changes in pull requests][gh-review-pr]. And when you think issues that must be fixed before they |
||||
merged, please follow "Request changes" in [GitHub's reviewing changes in pull requests][gh-review-pr] to ask contributors |
||||
modify it. |
||||
<!-- markdown-link-check-enable --> |
||||
|
||||
Labeled Pull Requests is an important part. Reasonable classification can save a lot of time for reviewers. The good news |
||||
is that the label's name and usage of Pull Requests are the same in [Issues](#issues), which can reduce the memory. For |
||||
example, if there is a Pull Request is related to docker and block deployment. We can label it with [docker][label-docker] |
||||
and [priority:high][label-priority-high]. |
||||
|
||||
Pull Requests have some unique labels of it own |
||||
|
||||
| Label | Mean | |
||||
| ------ | ------ | |
||||
| [miss document][label-miss-document] | Pull Requests miss document, and should be add | |
||||
| [first time contributor][label-first-time-contributor] | Pull Requests submit by first time contributor | |
||||
| [don't merge][label-do-not-merge] | Pull Requests have some problem and should not be merged | |
||||
|
||||
> Note: Only members have permission to add or delete label. When you need to add or remove lebals but are not member, |
||||
> you can `@` members to do that. But as long as you have a GitHub account, you can comment on Pull Requests and give suggestions. |
||||
> We encourage everyone in the community to review Pull Requests |
||||
|
||||
[mail-review-wanted]: https://lists.apache.org/thread/9flwlzrp69xjn6v8tdkbytq8glqp2k51 |
||||
[discussion-result-review-wanted]: https://github.com/apache/dolphinscheduler/discussions/7545 |
||||
[label-wontfix]: https://github.com/apache/dolphinscheduler/labels/wontfix |
||||
[label-duplicate]: https://github.com/apache/dolphinscheduler/labels/duplicate |
||||
[label-need-more-information]: https://github.com/apache/dolphinscheduler/labels/need%20more%20information |
||||
[label-win-os]: https://github.com/apache/dolphinscheduler/labels/win-os |
||||
[label-waiting-for-reply]: https://github.com/apache/dolphinscheduler/labels/Waiting%20for%20reply |
||||
[label-waiting-for-review]: https://github.com/apache/dolphinscheduler/labels/Waiting%20for%20review |
||||
[label-user-experience]: https://github.com/apache/dolphinscheduler/labels/user%20experience |
||||
[label-development]: https://github.com/apache/dolphinscheduler/labels/development |
||||
[label-UI]: https://github.com/apache/dolphinscheduler/labels/UI |
||||
[label-suggestion]: https://github.com/apache/dolphinscheduler/labels/suggestion |
||||
[label-security]: https://github.com/apache/dolphinscheduler/labels/security |
||||
[label-Python]: https://github.com/apache/dolphinscheduler/labels/Python |
||||
[label-plug-in]: https://github.com/apache/dolphinscheduler/labels/plug-in |
||||
[label-document]: https://github.com/apache/dolphinscheduler/labels/document |
||||
[label-docker]: https://github.com/apache/dolphinscheduler/labels/docker |
||||
[label-all-list]: https://github.com/apache/dolphinscheduler/labels |
||||
[label-Chore]: https://github.com/apache/dolphinscheduler/labels/Chore |
||||
[label-good-first-issue]: https://github.com/apache/dolphinscheduler/labels/good%20first%20issue |
||||
[label-help-wanted]: https://github.com/apache/dolphinscheduler/labels/help%20wanted |
||||
[label-easy-to-fix]: https://github.com/apache/dolphinscheduler/labels/easy%20to%20fix |
||||
[label-priority-high]: https://github.com/apache/dolphinscheduler/labels/priority%3Ahigh |
||||
[label-priority-middle]: https://github.com/apache/dolphinscheduler/labels/priority%3Amiddle |
||||
[label-priority-low]: https://github.com/apache/dolphinscheduler/labels/priority%3Alow |
||||
[label-miss-document]: https://github.com/apache/dolphinscheduler/labels/miss%20document |
||||
[label-first-time-contributor]: https://github.com/apache/dolphinscheduler/labels/first%20time%20contributor |
||||
[label-do-not-merge]: https://github.com/apache/dolphinscheduler/labels/don%27t%20merge |
||||
[label-e2e]: https://github.com/apache/dolphinscheduler/labels/e2e |
||||
[label-need-verify]: https://github.com/apache/dolphinscheduler/labels/need%20to%20verify |
||||
[issue-choose]: https://github.com/apache/dolphinscheduler/issues/new/choose |
||||
[all-issues]: https://github.com/apache/dolphinscheduler/issues |
||||
[all-PRs]: https://github.com/apache/dolphinscheduler/pulls |
||||
[gh-review-pr]: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/about-pull-request-reviews |
||||
@ -0,0 +1,8 @@
|
||||
# Security |
||||
|
||||
The Apache Software Foundation takes a rigorous stance on eliminating security issues in its software projects. Apache DolphinScheduler is also very concerned Security issues related to its features and functionality. |
||||
|
||||
If you have apprehensions regarding DolphinScheduler’s security or you discover vulnerability or potential threat, don’t hesitate to get in touch with the Apache Security Team by dropping a mail at [security@apache.org](mailto:security@apache.org). Please specify the project name as DolphinScheduler in the email and provide a description of the relevant problem or potential threat. You are also urged to recommend the way to reproduce and replicate the issue. The apache security team and the DolphinScheduler community will get back to you after assessing and analysing the findings. |
||||
|
||||
Please pay attention to report the security issue on the security email before disclosing it on public domain. |
||||
|
||||
@ -0,0 +1,23 @@
|
||||
# Subscribe Mailing Lists |
||||
|
||||
It is highly recommended to subscribe to the development mailing list to keep up-to-date with the community. |
||||
|
||||
In the process of using DolphinScheduler, if you have any questions or ideas, suggestions, you can participate in the DolphinScheduler community building through the Apache mailing list. Sending a subscription email is also very simple, the steps are as follows: |
||||
|
||||
1. Send an email to dev-subscribe@dolphinscheduler.apache.org with your own email address, subject and content are arbitrary. |
||||
|
||||
2. Receive confirmation email and reply. After completing step 1, you will receive a confirmation email from dev-help@dolphinscheduler.apache.org (if not received, please confirm whether the email is automatically classified as spam, promotion email, subscription email, etc.) . Then reply directly to the email, or click on the link in the email to reply quickly, the subject and content are arbitrary. |
||||
|
||||
3. Receive a welcome email. After completing the above steps, you will receive a welcome email with the subject WELCOME to dev@dolphinscheduler.apache.org, and you have successfully subscribed to the Apache DolphinScheduler mailing list. |
||||
|
||||
# Unsubscribe Mailing Lists |
||||
|
||||
If you do not need to know what's going on with DolphinScheduler, you can unsubscribe from the mailing list. |
||||
|
||||
Unsubscribe from the mailing list steps are as follows: |
||||
|
||||
1. Send an email to dev-unsubscribe@dolphinscheduler.apache.org with your subscribed email address, subject and content are arbitrary. |
||||
|
||||
2. Receive confirmation email and reply. After completing step 1, you will receive a confirmation email from dev-help@dolphinscheduler.apache.org (if not received, please confirm whether the email is automatically classified as spam, promotion email, subscription email, etc.) . Then reply directly to the email, or click on the link in the email to reply quickly, the subject and content are arbitrary. |
||||
|
||||
3. Receive a goodbye email. After completing the above steps, you will receive a goodbye email with the subject GOODBYE from dev@dolphinscheduler.apache.org, and you have successfully unsubscribed to the Apache DolphinScheduler mailing list, and you will not receive emails from dev@dolphinscheduler.apache.org. |
||||
@ -0,0 +1,32 @@
|
||||
# Release Post |
||||
|
||||
We still have some publish task to do after we send the announcement mail, currently we have to publish Docker images to |
||||
Docker Hub and also publish pydolphinscheduler to PyPI. |
||||
|
||||
## Publish Docker Image |
||||
|
||||
we already have the exists CI to publish the latest Docker image to GitHub container register with [config](https://github.com/apache/dolphinscheduler/blob/d80cf21456265c9d84e642bdb4db4067c7577fc6/.github/workflows/publish-docker.yaml#L55-L63). |
||||
We could reuse the main command the CI run and publish our Docker images to Docker Hub by single command. |
||||
|
||||
```bash |
||||
# Please change the <VERSION> place hold to the version you release |
||||
./mvnw -B clean deploy \ |
||||
-Dmaven.test.skip \ |
||||
-Dmaven.javadoc.skip \ |
||||
-Dmaven.checkstyle.skip \ |
||||
-Dmaven.deploy.skip \ |
||||
-Ddocker.tag=<VERSION> \ |
||||
-Ddocker.hub=apache \ |
||||
-Pdocker,release |
||||
``` |
||||
|
||||
## Publish pydolphinscheduler to PyPI |
||||
|
||||
Python API need to release to PyPI for easier download and use, you can see more detail in [Python API release](https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-python/pydolphinscheduler/RELEASE.md#to-pypi) |
||||
to finish PyPI release. |
||||
|
||||
## Get All Contributors |
||||
|
||||
You might need all contributors in current release when you want to publish the release news or announcement, you could |
||||
use the git command `git log --pretty="%an" <PREVIOUS-RELEASE-SHA>..<CURRENT-RELEASE-SHA> | sort | uniq` to auto generate |
||||
the git author name. |
||||
@ -0,0 +1,31 @@
|
||||
# Release Preparation |
||||
|
||||
## Check release-docs |
||||
|
||||
Compared with the last release, the `release-docs` of the current release needs to be updated to the latest, if there are dependencies and versions changes |
||||
|
||||
- `dolphinscheduler-dist/release-docs/LICENSE` |
||||
- `dolphinscheduler-dist/release-docs/NOTICE` |
||||
- `dolphinscheduler-dist/release-docs/licenses` |
||||
|
||||
## Update Version |
||||
|
||||
For example, to release `x.y.z`, the following updates are required: |
||||
|
||||
- Version in the code: |
||||
- `sql`: |
||||
- `dolphinscheduler_mysql.sql`: `t_ds_version` needs to be updated to x.y.z |
||||
- `dolphinscheduler_postgre.sql`: `t_ds_version` needs to be updated to x.y.z |
||||
- `dolphinscheduler_h2.sql`: `t_ds_version` needs to be updated to x.y.z |
||||
- `upgrade`: whether to add`x.y.z_schema` |
||||
- `soft_version`: need to be updated to x.y.z |
||||
- `deploy/docker/.env`: `HUB` change to `apache`，`TAG` change to `x.y.z` |
||||
- `deploy/kubernetes/dolphinscheduler`: |
||||
- `Chart.yaml`: `appVersion` needs to be updated to x.y.z (`version` is helm chart version，incremented and different from x.y.z) |
||||
- `values.yaml`: `image.tag` needs to be updated to x.y.z |
||||
- `dolphinscheduler-python/pydolphinscheduler/setup.py`: change `version` to x.y.z |
||||
- Version in the docs: |
||||
- Change the placeholder `<version>`(except `pom`) to the `x.y.z` in directory `docs` |
||||
- Add new history version |
||||
- `docs/docs/en/history-versions.md` and `docs/docs/zh/history-versions.md`: Add the new version and link for `x.y.z` |
||||
- `docs/configs/docsdev.js`: change `/dev/` to `/x.y.z/` |
||||
@ -0,0 +1,540 @@
|
||||
# Release Guide |
||||
|
||||
## Check Your Environment |
||||
|
||||
To make sure you could successfully complete the release for DolphinScheduler, you should check your environment and make sure |
||||
all conditions are met, if any or them are missing, you should install them and make sure them work. |
||||
|
||||
```shell |
||||
# JDK 1.8 above is requests |
||||
java -version |
||||
# Maven requests |
||||
mvn -version |
||||
# Python 3.6 above is requests, and you have to make keyword `python` work in your terminal and version match |
||||
python --version |
||||
``` |
||||
|
||||
## GPG Settings |
||||
|
||||
### Install GPG |
||||
|
||||
Download installation package on [official GnuPG website](https://www.gnupg.org/download/index.html). |
||||
The command of GnuPG 1.x version can differ a little from that of 2.x version. |
||||
The following instructions take `GnuPG-2.1.23` version for example. |
||||
|
||||
After the installation, execute the following command to check the version number. |
||||
|
||||
```shell |
||||
gpg --version |
||||
``` |
||||
|
||||
### Create Key |
||||
|
||||
After the installation, execute the following command to create key. |
||||
|
||||
This command indicates `GnuPG-2.x` can be used: |
||||
|
||||
```shell |
||||
gpg --full-gen-key |
||||
``` |
||||
|
||||
This command indicates `GnuPG-1.x` can be used: |
||||
|
||||
```shell |
||||
gpg --gen-key |
||||
``` |
||||
|
||||
Finish the key creation according to instructions, **Notice: Please use Apache mails and its password for key creation.** |
||||
|
||||
```shell |
||||
gpg (GnuPG) 2.0.12; Copyright (C) 2009 Free Software Foundation, Inc. |
||||
This is free software: you are free to change and redistribute it. |
||||
There is NO WARRANTY, to the extent permitted by law. |
||||
|
||||
Please select what kind of key you want: |
||||
(1) RSA and RSA (default) |
||||
(2) DSA and Elgamal |
||||
(3) DSA (sign only) |
||||
(4) RSA (sign only) |
||||
Your selection? 1 |
||||
RSA keys may be between 1024 and 4096 bits long. |
||||
What keysize do you want? (2048) 4096 |
||||
Requested keysize is 4096 bits |
||||
Please specify how long the key should be valid. |
||||
0 = key does not expire |
||||
<n> = key expires in n days |
||||
<n>w = key expires in n weeks |
||||
<n>m = key expires in n months |
||||
<n>y = key expires in n years |
||||
Key is valid for? (0) |
||||
Key does not expire at all |
||||
Is this correct? (y/N) y |
||||
|
||||
GnuPG needs to construct a user ID to identify your key. |
||||
|
||||
Real name: ${Input username} |
||||
Email address: ${Input email} |
||||
Comment: ${Input comment} |
||||
You selected this USER-ID: |
||||
"${Inputed username} (${Inputed comment}) <${Inputed email}>" |
||||
|
||||
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O |
||||
You need a Passphrase to protect your secret key. # Input your Apache mail passwords |
||||
``` |
||||
|
||||
### Check Generated Key |
||||
|
||||
```shell |
||||
gpg --list-keys |
||||
``` |
||||
|
||||
Execution Result: |
||||
|
||||
```shell |
||||
pub 4096R/85E11560 2019-11-15 |
||||
uid ${Username} (${Comment}) <{Email}> |
||||
sub 4096R/A63BC462 2019-11-15 |
||||
``` |
||||
|
||||
Among them, 85E11560 is public key ID. |
||||
|
||||
### Upload the Public Key to Key Server |
||||
|
||||
The command is as follow: |
||||
|
||||
```shell |
||||
gpg --keyserver hkp://pool.sks-keyservers.net --send-key 85E11560 |
||||
``` |
||||
|
||||
`pool.sks-keyservers.net` is randomly chosen from [public key server](https://sks-keyservers.net/status/). |
||||
Each server will automatically synchronize with one another, so it would be okay to choose any one, a backup keys servers |
||||
is `gpg --keyserver hkp://keyserver.ubuntu.com --send-key <YOUR_KEY_ID>` |
||||
|
||||
## Apache Maven Central Repository Release |
||||
|
||||
### Set `settings-security.xml` and `settings.xml` |
||||
|
||||
In this section, we add Apache server maven configuration to prepare the release, we have to add `settings-security.xml` according |
||||
to [here](http://maven.apache.org/guides/mini/guide-encryption.html) firstly and then change your `~/.m2/settings.xml` like below |
||||
|
||||
```xml |
||||
<settings> |
||||
<servers> |
||||
<server> |
||||
<id>apache.snapshots.https</id> |
||||
<username> <!-- APACHE LDAP username --> </username> |
||||
<password> <!-- APACHE LDAP encrypted password --> </password> |
||||
</server> |
||||
<server> |
||||
<id>apache.releases.https</id> |
||||
<username> <!-- APACHE LDAP username --> </username> |
||||
<password> <!-- APACHE LDAP encrypted password --> </password> |
||||
</server> |
||||
</servers> |
||||
</settings> |
||||
``` |
||||
|
||||
### Set Release in Environment |
||||
|
||||
We will use the release version, your github name and your Apache username below several times, so it is better to store |
||||
it to bash variable for easier use. |
||||
|
||||
```shell |
||||
VERSION=<THE-VERSION-YOU-RELEASE> |
||||
GH_USERNAME=<YOUR-GITHUB-USERNAME> |
||||
A_USERNAME=<YOUR-APACHE-USERNAME> |
||||
``` |
||||
|
||||
> Note: We can use the variable directly in you bash after we set environment, without changing anything. For example, we |
||||
> can use command `git clone -b "${VERSION}"-prepare https://github.com/apache/dolphinscheduler.git` to clone the release branch |
||||
> and it can be success by covert the `"${VERSION}"` to `<THE-VERSION-YOU-RELEASE>`. But you have to change `<VERSION>` manually in |
||||
> some of not bash step like [vote mail](#vote-procedure), we using `<VERSION>` instead of `"${VERSION}"` to notice release |
||||
> manager they have to change by hand. |
||||
|
||||
### Create Release Branch |
||||
|
||||
In this section, we dwonload source code from github and create new branch to release |
||||
|
||||
```shell |
||||
git clone -b "${VERSION}"-prepare https://github.com/apache/dolphinscheduler.git |
||||
cd ~/dolphinscheduler/ |
||||
git pull |
||||
git checkout -b "${VERSION}"-release |
||||
git push origin "${VERSION}"-release |
||||
``` |
||||
|
||||
### Pre-Release Check |
||||
|
||||
```shell |
||||
# make gpg command could be run in maven correct |
||||
export GPG_TTY=$(tty) |
||||
|
||||
mvn release:prepare -Prelease,python -Darguments="-Dmaven.test.skip=true -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true" -DautoVersionSubmodules=true -DdryRun=true -Dusername="${GH_USERNAME}" |
||||
``` |
||||
|
||||
* `-Prelease,python`: choose release and python profile, which will pack all the source codes, jar files and executable binary packages, and Python distribute package. |
||||
* `-DautoVersionSubmodules=true`: it can make the version number is inputted only once and not for each sub-module. |
||||
* `-DdryRun=true`: dry run which means not to generate or submit new version number and new tag. |
||||
|
||||
### Prepare for the Release |
||||
|
||||
First, clean local pre-release check information. |
||||
|
||||
```shell |
||||
mvn release:clean |
||||
``` |
||||
|
||||
Then, prepare to execute the release. |
||||
|
||||
```shell |
||||
mvn release:prepare -Prelease,python -Darguments="-Dmaven.test.skip=true -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true" -DautoVersionSubmodules=true -DpushChanges=false -Dusername="${GH_USERNAME}" |
||||
``` |
||||
|
||||
It is basically the same as the previous rehearsal command, but deleting `-DdryRun=true` parameter. |
||||
|
||||
* `-DpushChanges=fals`: do not submit the edited version number and tag to GitHub automatically. |
||||
|
||||
> Note: You have to config your git `user.name` and `user.password` by command `git config --global user.email "you@example.com"` |
||||
> and `git config --global user.name "Your Name"` if you meet some mistake like **Please tell me who you are.** |
||||
> from git. |
||||
|
||||
After making sure there is no mistake in local files, submit them to GitHub. |
||||
|
||||
```shell |
||||
git push -u origin "${VERSION}"-release |
||||
git push origin --tags |
||||
``` |
||||
|
||||
<!-- markdown-link-check-disable --> |
||||
|
||||
> Note1: In this step, you should use github token for password because native password no longer supported, you can see |
||||
> https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token for more |
||||
> detail about how to create token about it. |
||||
|
||||
> Note2: After the command done, it will auto-created `release.properties` file and `*.Backup` files, their will be need |
||||
> in the following command and DO NOT DELETE THEM |
||||
|
||||
<!-- markdown-link-check-enable --> |
||||
|
||||
### Deploy the Release |
||||
|
||||
```shell |
||||
mvn release:perform -Prelease,python -Darguments="-Dmaven.test.skip=true -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true" -DautoVersionSubmodules=true -Dusername="${GH_USERNAME}" |
||||
``` |
||||
|
||||
After that command is executed, the version to be released will be uploaded to Apache staging repository automatically. |
||||
Go to [apache staging repositories](https://repository.apache.org/#stagingRepositories) and login by Apache LDAP. then you can see the uploaded version, the content of `Repository` column is the `${STAGING.REPOSITORY}`. |
||||
Click `Close` to tell Nexus that the construction is finished, because only in this way, this version can be usable. |
||||
If there is any problem in gpg signature, `Close` will fail, but you can see the failure information through `Activity`. |
||||
|
||||
## Apache SVN Repository Release |
||||
|
||||
### Checkout dolphinscheduler Release Directory |
||||
|
||||
If there is no local work directory, create one at first. |
||||
|
||||
```shell |
||||
mkdir -p ~/ds_svn/dev/ |
||||
cd ~/ds_svn/dev/ |
||||
``` |
||||
|
||||
After the creation, checkout dolphinscheduler release directory from Apache SVN. |
||||
|
||||
```shell |
||||
svn --username="${A_USERNAME}" co https://dist.apache.org/repos/dist/dev/dolphinscheduler |
||||
cd ~/ds_svn/dev/dolphinscheduler |
||||
``` |
||||
|
||||
### Add gpg Public Key |
||||
|
||||
Only the account in its first deployment needs to add that. |
||||
It is alright for `KEYS` to only include the public key of the deployed account. |
||||
|
||||
```shell |
||||
gpg -a --export <YOUR-GPG-KEY-ID> >> KEYS |
||||
``` |
||||
|
||||
### Add the Release Content to SVN Directory |
||||
|
||||
Create folder by version number. |
||||
|
||||
```shell |
||||
mkdir -p ~/ds_svn/dev/dolphinscheduler/"${VERSION}" |
||||
mkdir -p ~/ds_svn/dev/dolphinscheduler/"${VERSION}"/python |
||||
cd ~/ds_svn/dev/dolphinscheduler/"${VERSION}" |
||||
``` |
||||
|
||||
Add source code packages, binary packages and executable binary packages to SVN working directory. |
||||
|
||||
```shell |
||||
# Source and binary tarball for main code |
||||
cp -f ~/dolphinscheduler/dolphinscheduler-dist/target/*.tar.gz ~/ds_svn/dev/dolphinscheduler/"${VERSION}" |
||||
cp -f ~/dolphinscheduler/dolphinscheduler-dist/target/*.tar.gz.asc ~/ds_svn/dev/dolphinscheduler/"${VERSION}" |
||||
|
||||
# Source and binary tarball for Python API |
||||
cp -f ~/dolphinscheduler/dolphinscheduler-dist/target/python/* ~/ds_svn/dev/dolphinscheduler/"${VERSION}"/python |
||||
``` |
||||
|
||||
### Generate sign files |
||||
|
||||
```shell |
||||
shasum -a 512 apache-dolphinscheduler-"${VERSION}"-src.tar.gz >> apache-dolphinscheduler-"${VERSION}"-src.tar.gz.sha512 |
||||
shasum -b -a 512 apache-dolphinscheduler-"${VERSION}"-bin.tar.gz >> apache-dolphinscheduler-"${VERSION}"-bin.tar.gz.sha512 |
||||
cd python |
||||
shasum -a 512 apache-dolphinscheduler-python-"${VERSION}".tar.gz >> apache-dolphinscheduler-python-"${VERSION}".tar.gz.sha512 |
||||
shasum -b -a 512 apache_dolphinscheduler-python-"${VERSION}"-py3-none-any.whl >> apache_dolphinscheduler-python-"${VERSION}"-py3-none-any.whl.sha512 |
||||
cd ../ |
||||
``` |
||||
|
||||
### Commit to Apache SVN |
||||
|
||||
```shell |
||||
cd ~/ds_svn/dev/dolphinscheduler |
||||
svn add * |
||||
svn --username="${A_USERNAME}" commit -m "release ${VERSION}" |
||||
``` |
||||
## Check Release |
||||
|
||||
### Check sha512 hash |
||||
|
||||
```shell |
||||
shasum -c apache-dolphinscheduler-"${VERSION}"-src.tar.gz.sha512 |
||||
shasum -c apache-dolphinscheduler-"${VERSION}"-bin.tar.gz.sha512 |
||||
cd python |
||||
shasum -c apache-dolphinscheduler-python-"${VERSION}".tar.gz.sha512 |
||||
shasum -c apache_dolphinscheduler-python-"${VERSION}"-py3-none-any.whl.sha512 |
||||
cd ../ |
||||
``` |
||||
|
||||
### Check gpg Signature |
||||
|
||||
First, import releaser's public key. |
||||
Import KEYS from SVN repository to local. (The releaser does not need to import again; the checking assistant needs to import it, with the user name filled as the releaser's. ) |
||||
|
||||
```shell |
||||
curl https://dist.apache.org/repos/dist/dev/dolphinscheduler/KEYS >> KEYS |
||||
gpg --import KEYS |
||||
gpg --edit-key "${A_USERNAME}" |
||||
> trust |
||||
|
||||
Please decide how far you trust this user to correctly verify other users' keys |
||||
(by looking at passports, checking fingerprints from different sources, etc.) |
||||
|
||||
1 = I don't know or won't say |
||||
2 = I do NOT trust |
||||
3 = I trust marginally |
||||
4 = I trust fully |
||||
5 = I trust ultimately |
||||
m = back to the main menu |
||||
|
||||
Your decision? 5 |
||||
|
||||
> save |
||||
``` |
||||
|
||||
Then, check the gpg signature. |
||||
|
||||
```shell |
||||
gpg --verify apache-dolphinscheduler-"${VERSION}"-src.tar.gz.asc |
||||
gpg --verify apache-dolphinscheduler-"${VERSION}"-bin.tar.gz.asc |
||||
cd python |
||||
gpg --verify apache-dolphinscheduler-python-"${VERSION}".tar.gz.asc |
||||
gpg --verify apache_dolphinscheduler-python-"${VERSION}"-py3-none-any.whl.asc |
||||
cd ../ |
||||
``` |
||||
|
||||
> Note: You have to create gpg signature manually when you can not find your `asc` file, the command |
||||
> `gpg --armor --detach-sign --digest-algo=SHA512 apache-dolphinscheduler-"${VERSION}"-bin.tar.gz` and |
||||
> `gpg --armor --detach-sign --digest-algo=SHA512 apache-dolphinscheduler-"${VERSION}"-src.tar.gz` will create them |
||||
|
||||
### Check Released Files |
||||
|
||||
#### Check source package |
||||
|
||||
Decompress `apache-dolphinscheduler-<VERSION>-src.tar.gz` and `python/apache-dolphinscheduler-python-<VERSION>.tar.gz` then check the following items: |
||||
|
||||
* Check whether source tarball is oversized for including nonessential files |
||||
* `LICENSE` and `NOTICE` files exist |
||||
* Correct year in `NOTICE` file |
||||
* There is only text files but no binary files |
||||
* All source files have ASF headers |
||||
* Codes can be compiled and pass the unit tests (mvn install) |
||||
* The contents of the release match with what's tagged in version control (diff -r a verify_dir tag_dir) |
||||
* Check if there is any extra files or folders, empty folders for example |
||||
|
||||
#### Check binary packages |
||||
|
||||
Decompress `apache-dolphinscheduler-<VERSION>-src.tar.gz` and `python/apache-dolphinscheduler-python-<VERSION>-bin.tar.gz` |
||||
to check the following items: |
||||
|
||||
- `LICENSE` and `NOTICE` files exist |
||||
- Correct year in `NOTICE` file |
||||
- Check the third party dependency license: |
||||
- The software have a compatible license |
||||
- All software licenses mentioned in `LICENSE` |
||||
- All the third party dependency licenses are under `licenses` folder |
||||
- If it depends on Apache license and has a `NOTICE` file, that `NOTICE` file need to be added to `NOTICE` file of the release |
||||
|
||||
## Call for a Vote |
||||
|
||||
### Update Release Notes |
||||
|
||||
You should create a release note in GitHub by [new release note](https://github.com/apache/dolphinscheduler/releases/new). |
||||
It should be done before vote mail because we need the release note in the mail. You could use command |
||||
`git log --pretty="- %s" <PREVIOUS-RELEASE-SHA>..<CURRENT-RELEASE-SHA> > changelog.md` to creat the changelog(some log |
||||
maybe not correct, you should filter them by yourself) and classify them and paste them to GitHub release note page |
||||
|
||||
### Vote procedure |
||||
|
||||
1. DolphinScheduler community vote: send the vote e-mail to `dev@dolphinscheduler.apache.org`. |
||||
PMC needs to check the rightness of the version according to the document before they vote. |
||||
After at least 72 hours and with at least 3 `+1 and no -1 PMC member` votes, it can come to the next stage of the vote. |
||||
|
||||
2. Announce the vote result: send the result vote e-mail to `dev@dolphinscheduler.apache.org`。 |
||||
|
||||
### Vote Templates |
||||
|
||||
#### DolphinScheduler Community Vote Template |
||||
|
||||
Title： |
||||
|
||||
```txt |
||||
[VOTE] Release Apache DolphinScheduler <VERSION> |
||||
``` |
||||
|
||||
Body： |
||||
|
||||
```txt |
||||
Hello DolphinScheduler Community, |
||||
|
||||
This is a call for vote to release Apache DolphinScheduler version <VERSION> |
||||
|
||||
Release notes: https://github.com/apache/dolphinscheduler/releases/tag/<VERSION> |
||||
|
||||
The release candidates: https://dist.apache.org/repos/dist/dev/dolphinscheduler/<VERSION>/ |
||||
|
||||
Maven 2 staging repository: https://repository.apache.org/content/repositories/<VERSION>/org/apache/dolphinscheduler/ |
||||
|
||||
Git tag for the release: https://github.com/apache/dolphinscheduler/tree/<VERSION> |
||||
|
||||
Release Commit ID: https://github.com/apache/dolphinscheduler/commit/<SHA-VALUE> |
||||
|
||||
Keys to verify the Release Candidate: https://dist.apache.org/repos/dist/dev/dolphinscheduler/KEYS |
||||
|
||||
Look at here for how to verify this release candidate: https://dolphinscheduler.apache.org/en-us/community/release.html |
||||
|
||||
The vote will be open for at least 72 hours or until necessary number of votes are reached. |
||||
|
||||
Please vote accordingly: |
||||
|
||||
[ ] +1 approve |
||||
[ ] +0 no opinion |
||||
[ ] -1 disapprove with the reason |
||||
|
||||
Checklist for reference: |
||||
|
||||
[ ] Download links are valid. |
||||
[ ] Checksums and PGP signatures are valid. |
||||
[ ] Source code artifacts have correct names matching the current release. |
||||
[ ] LICENSE and NOTICE files are correct for each DolphinScheduler repo. |
||||
[ ] All files have license headers if necessary. |
||||
[ ] No compiled archives bundled in source archive. |
||||
``` |
||||
|
||||
2. Announce the vote result: |
||||
|
||||
Body： |
||||
|
||||
```txt |
||||
The vote to release Apache DolphinScheduler <VERSION> has passed.Here is the vote result, |
||||
|
||||
4 PMC member +1 votes: |
||||
|
||||
xxx |
||||
xxx |
||||
xxx |
||||
xxx |
||||
|
||||
1 community +1 vote: |
||||
xxx |
||||
|
||||
Thanks everyone for taking time to check this release and help us. |
||||
``` |
||||
|
||||
## Finish the Release |
||||
|
||||
### Move source packages, binary packages from the `dev` directory to `release` directory |
||||
|
||||
```shell |
||||
svn mv https://dist.apache.org/repos/dist/dev/dolphinscheduler/"${VERSION}" https://dist.apache.org/repos/dist/release/dolphinscheduler/ |
||||
``` |
||||
|
||||
### Export you new gpg KEYS from dev to release(optional) |
||||
|
||||
Only if the first time you release with this gpg KEY, including it is you first release or you change your KEY |
||||
|
||||
```shell |
||||
mkdir -p ~/ds_svn/release/ |
||||
cd ~/ds_svn/release/ |
||||
svn --username="${A_USERNAME}" co https://dist.apache.org/repos/dist/release/dolphinscheduler |
||||
gpg -a --export <YOUR-GPG-KEY-ID> >> KEYS |
||||
svn add * |
||||
svn --username="${A_USERNAME}" commit -m "new key <YOUR-GPG-KEY-ID> add" |
||||
``` |
||||
|
||||
### Update Document |
||||
|
||||
Website should be present before you send the announce mail this section will tell you how to change the website. For example, |
||||
the release version is `<VERSION>`, the following updates are required(note it will take effect immediately when the PR is merged): |
||||
|
||||
- Repository **apache/dolphinscheduler-website**: |
||||
- `download/en-us/download.md` and `download/zh-cn/download.md`: add the download of the `<VERSION>` release package |
||||
- `scripts/conf.sh`: Add new release version `<VERSION>` key-value pair to variable `DEV_RELEASE_DOCS_VERSIONS` |
||||
- Repository **apache/dolphinscheduler**: |
||||
- `docs/configs/site.js`: |
||||
- `docsLatest`: update to `<VERSION>` |
||||
- `docs0`: The `text` of two places of `en-us/zh-cn` needs to be updated to `latest(<VERSION>)` |
||||
- `docsxyz`: Add a drop-down menu with `key` as `docsxyz` and `text` as `<VERSION>` in `children` of two places of `en-us/zh-cn` |
||||
- `docs/configs/index.md.jsx`: Add `<VERSION>: docsxyzConfig` |
||||
- `docs/docs/en/history-versions.md` and `docs/docs/zh/history-versions.md`: Add new `<VERSION>` release docs. |
||||
- `.github/ISSUE_TEMPLATE/bug-report.yml`: DolphinScheduler's GitHub [bug-report](https://github.com/apache/dolphinscheduler/blob/dev/.github/ISSUE_TEMPLATE/bug-report.yml) |
||||
issue template have **Version** selection bottom. So after we release DolphinScheduler we should and the new `<VERSION>` to |
||||
bug-report.yml |
||||
|
||||
### Find DolphinScheduler in [apache staging repositories](https://repository.apache.org/#stagingRepositories) and click `Release` |
||||
|
||||
### Send Announcement E-mail Community |
||||
|
||||
You should send announcement E-mail after release process finished. The E-mail should send to `dev@dolphinscheduler.apache.org` |
||||
and cc to `announce@apache.org`. |
||||
|
||||
Announcement e-mail template as below： |
||||
|
||||
Title： |
||||
|
||||
```txt |
||||
[ANNOUNCE] Release Apache DolphinScheduler <VERSION> |
||||
``` |
||||
|
||||
Body： |
||||
|
||||
```txt |
||||
Hi all, |
||||
|
||||
We are glad to announce the release of Apache DolphinScheduler <VERSION>. Once again I would like to express my thanks to your help. |
||||
|
||||
Dolphin Scheduler is a distributed and easy-to-extend visual workflow scheduler system, |
||||
dedicated to solving the complex task dependencies in data processing, making the scheduler system out of the box for data processing. |
||||
|
||||
|
||||
Download Links: https://dolphinscheduler.apache.org/en-us/download/download.html |
||||
|
||||
Release Notes: https://github.com/apache/dolphinscheduler/releases/tag/<VERSION> |
||||
|
||||
Website: https://dolphinscheduler.apache.org/ |
||||
|
||||
DolphinScheduler Resources: |
||||
- Issue: https://github.com/apache/dolphinscheduler/issues/ |
||||
- Mailing list: dev@dolphinscheduler.apache.org |
||||
- Documents: https://dolphinscheduler.apache.org/zh-cn/docs/<VERSION>/user_doc/about/introduction.html |
||||
``` |
||||
@ -0,0 +1,91 @@
|
||||
# DSIP |
||||
|
||||
DolphinScheduler Improvement Proposal(DSIP) introduce major improvements to the Apache DolphinScheduler codebase. It is |
||||
not for small incremental improvements, and the purpose of DSIP is to notice and inform community the finished or coming |
||||
big feature for Apache DolphinScheduler. |
||||
|
||||
## What is considered as DSIP |
||||
|
||||
- Any major new feature, major improvement, introduce or remove components |
||||
- Any major change of public interfaces, such as API endpoints, web ui huge change |
||||
|
||||
When the change in doubt and any committer thinks it should be DSIP, it does. |
||||
|
||||
We use GitHub Issue and Apache mail thread to record and hold DSIP, for more detail you could go to section |
||||
[current DSIPs](#current-dsips) and [past DSIPs](#past-dsips). |
||||
|
||||
As a DSIP, it should: |
||||
|
||||
- Have a mail thread title started with `[DISCUSS]` in [dev@dolphinscheduler.apache.org][mail-to-dev] |
||||
- Have a GitHub Issue labeled with `DSIP`, and including the mail thread link in the description. |
||||
|
||||
### Current DSIPs |
||||
|
||||
Current DSIPs including all DSIP still work-in-progress, you could see in [current DSIPs][current-DSIPs] |
||||
|
||||
### Past DSIPs |
||||
|
||||
Past DSIPs including all DSIP already done or retired for some reason, you could see in [past DSIPs][past-DSIPs] |
||||
|
||||
## DSIP Process |
||||
|
||||
### Create GitHub Issue |
||||
|
||||
All DSIP should start with GitHub Issue |
||||
|
||||
- If you pretty sure your issue is DSIP, you could click and choose "DSIP" in |
||||
[GitHub Issue][github-issue-choose] |
||||
- If you not sure about your issue is DSIP or not, you could click and choose "Feature request" in |
||||
[GitHub Issue][github-issue-choose]. DolphinScheduler maintainer team would add label `DSIP`, mention you in the |
||||
issue and lead you to this document when they think it should be DSIP. |
||||
|
||||
You should add special prefix `[DSIP-XXX]`, `XXX` stand for the id DSIP. It's auto increment, and you could find the next |
||||
integer in [All DSIPs][all-DSIPs] issues. |
||||
|
||||
### Send Discuss Mail |
||||
|
||||
After issue labeled with "DSIP", you should send an email to [dev@dolphinscheduler.apache.org][mail-to-dev]. |
||||
Describe the purpose, and the draft design about your idea. |
||||
|
||||
Here is the template for mail |
||||
|
||||
- Title: `[DISCUSS][DSIP-XXX] <CHANGE-TO-YOUR-LOVELY-PROPOSAL-TITLE>`, change `XXX` to special integer you just change in |
||||
[GitHub Issue](#create-github-issue), and also change proposal title. |
||||
- Content: |
||||
|
||||
```text |
||||
Hi community, |
||||
|
||||
<CHANGE-TO-YOUR-PROPOSAL-DETAIL> |
||||
|
||||
I already add a GitHub Issue for my proposal, which you could see in <CHANGE-TO-YOUR-GITHUB-ISSUE-LINK>. |
||||
|
||||
Looking forward any feedback for this thread. |
||||
``` |
||||
|
||||
After community discuss and all of them think it worth as DSIP, you could [work on it](#work-on-it-or-create-subtask-for-it). |
||||
But if community think it should not be DSIP or even this change should not be included to DolphinScheduler, maintainers |
||||
terminate mail thread and remove label "DSIP" for GitHub Issue, or even close issue if it should not change. |
||||
|
||||
### Work On It, Or Create Subtask For It |
||||
|
||||
When your proposal pass in the mail thread, you could make your hand dirty and start the work. You could submit related |
||||
pull requests in GitHub if change should in one single commit. What's more, if proposal is too huge in single commit, you |
||||
could create subtasks in GitHub Issue like [DSIP-1][DSIP-1], and separate into multiple commit. |
||||
|
||||
### Close After It Done |
||||
|
||||
When DSIP is finished and all related PR were merged, you should reply the mail thread you created in |
||||
[step two](#send-discuss-mail) to notice community the result of the DSIP. After that, this DSIP GitHub Issue would be |
||||
closed and transfer from [current DSIPs][current-DSIPs] to [past DSIPs][past-DSIPs], but you could still find it in [All DSIPs][all-DSIPs] |
||||
|
||||
## An Example For DSIP |
||||
|
||||
* [[DSIP-1][Feature][Parent] Add Python API for DolphinScheduler][DSIP-1]: Have multiple subtasks and Projects on it. |
||||
|
||||
[all-DSIPs]: https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue+label%3A%22DSIP%22+ |
||||
[current-DSIPs]: https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue+is%3Aopen+label%3A%22DSIP%22 |
||||
[past-DSIPs]: https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue+is%3Aclosed+label%3A%22DSIP%22+ |
||||
[github-issue-choose]: https://github.com/apache/dolphinscheduler/issues/new/choose |
||||
[mail-to-dev]: mailto:dev@dolphinscheduler.apache.org |
||||
[DSIP-1]: https://github.com/apache/dolphinscheduler/issues/6407 |
||||
@ -0,0 +1,104 @@
|
||||
# License 须知 |
||||
|
||||
如您所知，DolphinScheduler现属于ASF(Apache基金会)下的开源项目，这意味着当您想要成为DolphinScheduler的贡献者的时候，就必须按照Apache的规则来，而Apache对于License有着极其严苛的规则，为了避免贡献者在License上浪费过多的时间， |
||||
本文将为您讲解ASF—License以及参与DolphinScheduler如何过早的规避掉License风险。 |
||||
|
||||
注：本文仅适用于Apache项目。 |
||||
|
||||
### Apache项目可接受的License |
||||
|
||||
当您想要为DolphinScheduler（亦或其他Apache项目）增添一个新的功能，这个功能涉及到其他开源软件的引用，那么您必须注意，目前Apache项目支持遵从以下协议的开源软件（如果有遗漏，欢迎补充）： |
||||
|
||||
[ASF第三方许可证策](https://apache.org/legal/resolved.html) |
||||
|
||||
如果您所使用的第三方软件并不在以上协议之中，那么很抱歉，您的代码将无法通过审核，建议您找寻其他替代方案。 |
||||
|
||||
另外，当您需要使用新的软件的时候，请将您这样做的原因、最终产出结果发邮件至dev@dolphinscheduler.apache.org讨论，当得到至少3票PPMC认同的时候，您方可以引入。 |
||||
|
||||
### 如何在DolphinScheduler合法的使用第三方开源软件 |
||||
|
||||
当我们想要引入一个新的第三方软件(包含但不限于第三方的jar、文本、css、js、图片、图标、音视频等及在第三方基础上做的修改)至我们的项目中的时候，除了他们所遵从的协议是Apache允许的，另外一点很重要，就是合法的使用。您可以参考以下文章 |
||||
|
||||
* [COMMUNITY-LED DEVELOPMENT "THE APACHE WAY"](https://apache.org/dev/licensing-howto.html) |
||||
|
||||
|
||||
以Apache为例，当我们使用了ZooKeeper，那么ZooKeeper的NOTICE文件（每个开源项目都会有NOTICE文件，一般位于根目录）则必须在我们的项目中体现，用Apache的话来讲，就是"Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a |
||||
copyright notice that is included in or attached to the work. |
||||
|
||||
关于具体的各个开源协议使用协议，在此不做过多篇幅一一介绍，有兴趣可以自行查询了解。 |
||||
|
||||
### DolphinScheduler-License 检测规则 |
||||
|
||||
一般来讲，我们都会为自己的项目建立License-check脚本，DolphinScheduler-License是由[kezhenxu94](https://github.com/kezhenxu94)提供，其他开源软件略有不同，但最终结果都是为了确保我们在使用过程中能够第一时间避免License的问题。 |
||||
|
||||
当我们需要添加新的Jar或其他外部资源的时候，我们需要按照以下步骤： |
||||
|
||||
* 在known-dependencies.txt中添加你所需要的jar名称+版本。 |
||||
* 在dolphinscheduler-dist/release-docs/LICENSE中添加相关的maven仓库地址。 |
||||
* 在dolphinscheduler-dist/release-docs/NOTICE中追加相关的NOTICE文件，此文件请务必和原代码仓库地址中的NOTICE文件一致。 |
||||
* 在dolphinscheduler-dist/release-docs/license/下添加相关源代码的协议，文件命名为license+文件名.txt。 |
||||
#### check dependency license fail |
||||
``` |
||||
--- /dev/fd/63 2020-12-03 03:08:57.191579482 +0000 |
||||
+++ /dev/fd/62 2020-12-03 03:08:57.191579482 +0000 |
||||
@@ -1,0 +2 @@ |
||||
+HikariCP-java6-2.3.13.jar |
||||
@@ -16,0 +18 @@ |
||||
+c3p0-0.9.5.2.jar |
||||
@@ -149,0 +152 @@ |
||||
+mchange-commons-java-0.2.11.jar |
||||
Error: Process completed with exit code 1. |
||||
``` |
||||
一般来讲，添加一个jar的工作往往不会如此轻易的结束，因为它往往依赖了其它各种各样的jar，这些jar我们同样需要添加相应的license。 |
||||
这种情况下，我们会在check里面得到 check dependency license fail的错误信息，如上，我们缺少了HikariCP-java6-2.3.13、c3p0等的license声明， |
||||
按照添加jar的步骤补充即可，提示还是蛮友好的（哈哈）。 |
||||
### 附件 |
||||
|
||||
<!-- markdown-link-check-disable --> |
||||
附件：新jar的邮件格式 |
||||
``` |
||||
[VOTE][New Jar] jetcd-core(registry plugin support etcd3 ) |
||||
|
||||
|
||||
（说明目的，以及需要添加的 jar 是什么）Hi, the registry SPI will provide the implementation of etcd3. Therefore, we need to introduce a new jar (jetcd-core, jetcd-launcher (test)), which complies with the Apache-2.0 License. I checked his related dependencies to make sure it complies with the license of the Apache project. |
||||
|
||||
new jar : |
||||
|
||||
jetcd-core version -x.x.x license apache2.0 |
||||
|
||||
jetcd-launcher (test) version -x.x.x license apache2.0 |
||||
|
||||
dependent jar（它依赖了哪些jar，最好附带版本,以及相关采用的license协议）: |
||||
grpc-core version -x.x.x license XXX |
||||
grpc-netty version -x.x.x license XXX |
||||
grpc-protobuf version -x.x.x license XXX |
||||
grpc-stub version -x.x.x license XXX |
||||
grpc-grpclb version -x.x.x license XXX |
||||
netty-all version -x.x.x license XXX |
||||
failsafe version -x.x.x license XXX |
||||
|
||||
相关地址：主要有github地址、license文件地址、notice 文件地址、maven中央仓库地址 |
||||
|
||||
github address:https://github.com/etcd-io/jetcd |
||||
|
||||
|
||||
license:https://github.com/etcd-io/jetcd/blob/master/LICENSE |
||||
|
||||
|
||||
notice:https://github.com/etcd-io/jetcd/blob/master/NOTICE |
||||
|
||||
|
||||
Maven repository: |
||||
|
||||
|
||||
https://mvnrepository.com/artifact/io.etcd/jetcd-core |
||||
|
||||
|
||||
https://mvnrepository.com/artifact/io.etcd/jetcd-launcher |
||||
``` |
||||
<!-- markdown-link-check-enable --> |
||||
|
||||
### 参考文章： |
||||
* [COMMUNITY-LED DEVELOPMENT "THE APACHE WAY"](https://apache.org/dev/licensing-howto.html) |
||||
* [ASF 3RD PARTY LICENSE POLICY](https://apache.org/legal/resolved.html) |
||||
|
||||
@ -0,0 +1,12 @@
|
||||
# 如何成为 DolphinScheduler Committer |
||||
|
||||
每个人都可以成为Apache项目的贡献者。作为一个贡献者只是意味着你对项目感兴趣并以某种方式做出贡献，从提出合理的问题（这些问题记录了项目并向开发人员提供反馈）到提供新的特性作为补丁。 |
||||
|
||||
如果你成为对一个项目有价值的贡献者，你有可能被邀请成为一个committer。committer是ASF（Apache软件基金会）中用来表示提交特定项目的人的术语。它给你带来对项目仓库和资源写的权限。 |
||||
|
||||
在Dolphinscheduler社区，如果一个committer获得大量的优秀成绩，就可以被邀请加入项目管理委员会（PMC）。 |
||||
|
||||
当您不熟悉ASF使用的开源的开发过程时，有时难以理解的一点，就是我们更重视社区而不是代码。一个强大而健康的社区将受到尊重，成为一个有趣和有益的地方。更重要的是，一个多元化和健康的社区 |
||||
可以长时间的持续支持代码，即使个别公司在这个领域来来往往，也是如此。 |
||||
|
||||
更多详细信息可以在[这里](https://community.apache.org/contributors/)找到 |
||||
@ -0,0 +1,68 @@
|
||||
# 行为准则 |
||||
|
||||
以下行为准则以完全遵循[Apache软件基金会行为准则](https://www.apache.org/foundation/policies/conduct.html)为前提。 |
||||
|
||||
## 开发理念 |
||||
- **一致** 代码风格、命名以及使用方式保持一致。 |
||||
- **易读** 代码无歧义，易于阅读和理解而非调试手段才知晓代码意图。 |
||||
- **整洁** 认同《重构》和《代码整洁之道》的理念，追求整洁优雅代码。 |
||||
- **抽象** 层次划分清晰，概念提炼合理。保持方法、类、包以及模块处于同一抽象层级。 |
||||
- **用心** 保持责任心，持续以工匠精神雕琢。 |
||||
|
||||
## 开发规范 |
||||
|
||||
- 执行`mvn -U clean package -Prelease`可以编译和测试通过全部测试用例。 |
||||
- 测试覆盖率工具检查不低于dev分支覆盖率。 |
||||
- 请使用Checkstyle检查代码，违反验证规则的需要有特殊理由。模板位置在根目录下ds_check_style.xml。 |
||||
- 遵守编码规范。 |
||||
|
||||
## 编码规范 |
||||
|
||||
- 使用linux换行符。 |
||||
- 缩进（包含空行）和上一行保持一致。 |
||||
- 类声明后与下面的变量或方法之间需要空一行。 |
||||
- 不应有无意义的空行。 |
||||
- 类、方法和变量的命名要做到顾名思义，避免使用缩写。 |
||||
- 返回值变量使用`result`命名；循环中使用`each`命名循环变量；map中使用`entry`代替`each`。 |
||||
- 捕获的异常名称命名为`e`；捕获异常且不做任何事情，异常名称命名为`ignored`。 |
||||
- 配置文件使用驼峰命名，文件名首字母小写。 |
||||
- 需要注释解释的代码尽量提成小方法，用方法名称解释。 |
||||
- `equals`和`==`条件表达式中，常量在左，变量在右；大于小于等条件表达式中，变量在左，常量在右。 |
||||
- 除了用于继承的抽象类之外，尽量将类设计为`final`。 |
||||
- 嵌套循环尽量提成方法。 |
||||
- 成员变量定义顺序以及参数传递顺序在各个类和方法中保持一致。 |
||||
- 优先使用卫语句。 |
||||
- 类和方法的访问权限控制为最小。 |
||||
- 方法所用到的私有方法应紧跟该方法，如果有多个私有方法，书写私有方法应与私有方法在原方法的出现顺序相同。 |
||||
- 方法入参和返回值不允许为`null`。 |
||||
- 优先使用三目运算符代替if else的返回和赋值语句。 |
||||
- 优先考虑使用`LinkedList`，只有在需要通过下标获取集合中元素值时再使用`ArrayList`。 |
||||
- `ArrayList`，`HashMap`等可能产生扩容的集合类型必须指定集合初始大小，避免扩容。 |
||||
- 日志与注释一律使用英文。 |
||||
- 注释只能包含javadoc，todo和fixme。 |
||||
- 公开的类和方法必须有javadoc，其他类和方法以及覆盖自父类的方法无需javadoc。 |
||||
|
||||
## 单元测试规范 |
||||
|
||||
- 测试代码和生产代码需遵守相同代码规范。 |
||||
- 单元测试需遵循AIR（Automatic, Independent, Repeatable）设计理念。 |
||||
- 自动化（Automatic）：单元测试应全自动执行，而非交互式。禁止人工检查输出结果，不允许使用`System.out`，`log`等，必须使用断言进行验证。 |
||||
- 独立性（Independent）：禁止单元测试用例间的互相调用，禁止依赖执行的先后次序。每个单元测试均可独立运行。 |
||||
- 可重复执行（Repeatable）：单元测试不能受到外界环境的影响，可以重复执行。 |
||||
- 单元测试需遵循BCDE（Border, Correct, Design, Error）设计原则。 |
||||
- 边界值测试（Border）：通过循环边界、特殊数值、数据顺序等边界的输入，得到预期结果。 |
||||
- 正确性测试（Correct）：通过正确的输入，得到预期结果。 |
||||
- 合理性设计（Design）：与生产代码设计相结合，设计高质量的单元测试。 |
||||
- 容错性测试（Error）：通过非法数据、异常流程等错误的输入，得到预期结果。 |
||||
- 如无特殊理由，测试需全覆盖。 |
||||
- 每个测试用例需精确断言。 |
||||
- 准备环境的代码和测试代码分离。 |
||||
- 只有junit `Assert`，hamcrest `CoreMatchers`，Mockito相关可以使用static import。 |
||||
- 单数据断言，应使用`assertTrue`，`assertFalse`，`assertNull`和`assertNotNull`。 |
||||
- 多数据断言，应使用`assertThat`。 |
||||
- 精确断言，尽量不使用`not`，`containsString`断言。 |
||||
- 测试用例的真实值应名为为actualXXX，期望值应命名为expectedXXX。 |
||||
- 测试类和`@Test`标注的方法无需javadoc。 |
||||
|
||||
- 公共规范 |
||||
- 每行长度不超过`200`个字符，保证每一行语义完整以便于理解。 |
||||
@ -0,0 +1,89 @@
|
||||
# Commit Message 须知 |
||||
|
||||
### 前言 |
||||
一个好的 commit message 是能够帮助其他的开发者（或者未来的开发者）快速理解相关变更的上下文，同时也可以帮助项目管理人员确定该提交是否适合包含在发行版中。但当我们在查看了很多开源项目的 commit log 后，发现一个有趣的问题，一部分开发者，代码质量很不错，但是 commit message 记录却比较混乱，当其他贡献者或者学习者在查看代码的时候，并不能通过 commit log 很直观的了解 |
||||
该提交前后变更的目的，正如 Peter Hutterer 所言：Re-establishing the context of a piece of code is wasteful. We can’t avoid it completely, so our efforts should go to reducing it as much as possible. Commit messages can do exactly that and as a result, a commit message shows whether a developer is a good collaborator. 因此，DolphinScheduler 结合其他社区以及 Apache 官方文档制定了该规约。 |
||||
|
||||
### Commit Message RIP |
||||
|
||||
#### 1：明确修改内容 |
||||
|
||||
commit message 应该明确说明该提交解决了哪些问题（bug 修复、功能增强等），以便于用户开发者更好的跟踪问题，明确版本迭代过程中的优化情况。 |
||||
|
||||
#### 2：关联相应的Pull Request 或者Issue |
||||
|
||||
当我们的改动较大的时候，commit message 最好能够关联 GitHub 上的相关 Issue 或者 Pull Request，这样，我们的开发者在查阅代码的时候能够通过关联信息较为迅速的了解改代码提交的上下文情景，如果当前 commit 针对某个 issue，那么可以在 Footer 部分关闭这个 issue。 |
||||
|
||||
#### 3：统一的格式 |
||||
|
||||
格式化后的 CommitMessage 能够帮助我们提供更多的历史信息，方便快速浏览，同时也可以直接从 commit 生成 Change Log。 |
||||
|
||||
Commit message 应该包括三个部分：Header，Body 和 Footer。其中，Header 是必需的，Body 和 Footer 可以省略。 |
||||
|
||||
##### header |
||||
Header 部分只有一行，包括三个字段：type（必需）、scope（可选）和 subject（必需）。 |
||||
|
||||
[DS-ISSUE编号][type] subject |
||||
|
||||
(1) type 用于说明 commit 的类别，只允许使用下面7个标识。 |
||||
|
||||
* feat：新功能（feature） |
||||
* fix：修补bug |
||||
* docs：文档（documentation） |
||||
* style： 格式（不影响代码运行的变动） |
||||
* refactor：重构（即不是新增功能，也不是修改bug的代码变动） |
||||
* test：增加测试 |
||||
* chore：构建过程或辅助工具的变动 |
||||
|
||||
如果 type 为 feat 和 fix，则该 commit 将肯定出现在 Change log 之中。其他情况（docs、chore、style、refactor、test）建议不放入。 |
||||
|
||||
(2）scope |
||||
|
||||
scope 用于说明 commit 影响的范围，比如 server、remote 等，如果没有更合适的范围，你可以用 *。 |
||||
|
||||
(3) subject |
||||
|
||||
subject 是 commit 目的的简短描述，不超过50个字符。 |
||||
|
||||
##### Body |
||||
|
||||
Body 部分是对本次 commit 的详细描述，可以分成多行，换行符将以72个字符换行，避免自动换行影响美观。 |
||||
|
||||
Body 部分需要注意以下几点： |
||||
|
||||
* 使用动宾结构，注意使用现在时，比如使用 change 而非 changed 或 changes |
||||
|
||||
* 首字母不要大写 |
||||
|
||||
* 语句最后不需要 ‘.’ (句号) 结尾 |
||||
|
||||
|
||||
##### Footer |
||||
|
||||
Footer只适用于两种情况 |
||||
|
||||
(1) 不兼容变动 |
||||
|
||||
如果当前代码与上一个版本不兼容，则 Footer 部分以 BREAKING CHANGE 开头，后面是对变动的描述、以及变动理由和迁移方法。 |
||||
|
||||
(2) 关闭 Issue |
||||
|
||||
如果当前 commit 针对某个issue，那么可以在 Footer 部分关闭这个 issue,也可以一次关闭多个 issue 。 |
||||
|
||||
##### 举个例子 |
||||
[DS-001][docs-zh] add commit message |
||||
|
||||
* commit message RIP |
||||
* build some conventions |
||||
* help the commit messages become clean and tidy |
||||
* help developers and release managers better track issues |
||||
and clarify the optimization in the version iteration |
||||
|
||||
This closes #001 |
||||
|
||||
### 参考文档 |
||||
[提交消息格式](https://cwiki.apache.org/confluence/display/GEODE/Commit+Message+Format) |
||||
|
||||
[On commit messages-Peter Hutterer](http://who-t.blogspot.com/2009/12/on-commit-messages.html) |
||||
|
||||
[RocketMQ Community Operation Conventions](https://mp.weixin.qq.com/s/LKM4IXAY-7dKhTzGu5-oug) |
||||
@ -0,0 +1,42 @@
|
||||
# 参与贡献 |
||||
|
||||
首先非常感谢大家选择和使用 DolphinScheduler，非常欢迎大家加入 DolphinScheduler 大家庭，融入开源世界！ |
||||
|
||||
我们鼓励任何形式的参与社区，最终成为 Committer 或 PPMC，如： |
||||
* 将遇到的问题通过 github 上 [issue](https://github.com/apache/dolphinscheduler/issues) 的形式反馈出来 |
||||
* 回答别人遇到的 issue 问题 |
||||
* 帮助完善文档 |
||||
* 帮助项目增加测试用例 |
||||
* 为代码添加注释 |
||||
* 提交修复 Bug 或者 Feature 的 PR |
||||
* 发表应用案例实践、调度流程分析或者与调度相关的技术文章 |
||||
* 帮助推广 DolphinScheduler，参与技术大会或者 meetup 的分享等 |
||||
|
||||
欢迎加入贡献的队伍，加入开源从提交第一个 PR 开始 |
||||
- 比如添加代码注释或找到带有 ”easy to fix” 标记或一些非常简单的 issue(拼写错误等) 等等，先通过第一个简单的 PR 熟悉提交流程 |
||||
|
||||
注：贡献不仅仅限于 PR 哈，对促进项目发展的都是贡献 |
||||
|
||||
相信参与 DolphinScheduler，一定会让您从开源中受益！ |
||||
|
||||
### 1. 参与文档贡献 |
||||
|
||||
参考[参与贡献-文档需知](./document.md) |
||||
|
||||
### 2. 参与代码贡献 |
||||
|
||||
参考[参与贡献 Issue 需知](./issue.md)，[参与贡献 Pull Request 需知](./pull-request.md)，[参与贡献 CommitMessage 需知](./commit-message.md) |
||||
|
||||
|
||||
### 3. 如何领取 Issue，提交 Pull Request |
||||
|
||||
如果你想实现某个 Feature 或者修复某个 Bug。请参考以下内容： |
||||
|
||||
* 所有的 Bug 与新 Feature 建议使用 Issues Page 进行管理。 |
||||
* 如果想要开发实现某个 Feature 功能，请先回复该功能所关联的 Issue，表明你当前正在这个 Issue 上工作。 并在回复的时候为自己设置一个 **deadline**，并添加的回复内容中。 |
||||
* 最好在核心贡献者找到一个导师（指导者），导师会在设计与功能实现上给予即时的反馈。 |
||||
* 你应该新建一个分支来开始你的工作，分支的名字参考[参与贡献 Pull Request 需知](./pull-request.md)。比如，你想完成 feature 功能并提交了 Issue 111，那么你的 branch 名字应为 feature-111。 功能名称可与导师讨论后确定。 |
||||
* 完成后，发送一个 Pull Request 到 dolphinscheduler，提交过程具体请参考下面《[提交代码流程](./submit-code.md)》。 |
||||
|
||||
如果是想提交 Pull Request 完成某一个 Feature 或者修复某个 Bug，这里都建议大家从小处做起，完成一个小功能就提交一次，每次别改动太多文件，改动文件太多也会给 Reviewer 造成很大的心理压力，建议通过多次 Pull Request 的方式完成。 |
||||
|
||||
@ -0,0 +1,62 @@
|
||||
# 文档须知 |
||||
|
||||
良好的使用文档对任何类型的软件都是至关重要的。欢迎任何可以改进 DolphinScheduler 文档的贡献。 |
||||
|
||||
### 获取文档项目 |
||||
|
||||
DolphinScheduler 项目的文档维护在独立的 [git 仓库](https://github.com/apache/dolphinscheduler-website)中。 |
||||
|
||||
首先你需要先将文档项目 fork 到自己的 github 仓库中，然后将 fork 的文档克隆到本地计算机中。 |
||||
|
||||
``` |
||||
git clone https://github.com/<your-github-user-name>/dolphinscheduler-website |
||||
``` |
||||
|
||||
### 文档环境 |
||||
|
||||
DolphinScheduler 网站由 [docsite](https://github.com/chengshiwen/docsite-ext) 提供支持。 |
||||
|
||||
请确保你的 node 版本是 10+，docsite 尚不支持高于 10.x 的版本。 |
||||
|
||||
### 文档构建指南 |
||||
|
||||
1. 在根目录中运行 `npm install` 以安装依赖项。 |
||||
|
||||
2. 运行命令收集资源：2.1.运行 `export PROTOCOL_MODE=ssh` 告诉Git克隆资源，通过SSH协议而不是HTTPS协议。 2.2.运行 `./scripts/prepare_docs.sh` 准备所有相关资源，关更多信息，您可以查看[how prepare script work](https://github.com/apache/dolphinscheduler-website/blob/master/HOW_PREPARE_WOKR.md)。 |
||||
|
||||
3. 在根目录下运行 `npm run start` 启动本地服务器，其将允许在 http://localhost:8080。 |
||||
|
||||
4. 运行 `npm run build` 可以生成文档网站源代码。 |
||||
|
||||
5. 在本地验证你的更改：`python -m SimpleHTTPServer 8000`，当 python 版本为 3 时，请使用：`python3 -m http.server 8000`。 |
||||
|
||||
如果本地安装了更高版本的 node，可以考虑使用 `nvm` 来允许不同版本的 `node` 在你的计算机上运行。 |
||||
|
||||
1. 参考[说明](http://nvm.sh)安装 nvm |
||||
|
||||
2. 运行 `nvm install v10.23.1` 安装 node v10 |
||||
|
||||
3. 运行 `nvm use v10.23.1` 将当前工作环境切换到 node v10 |
||||
|
||||
然后你就可以在本地环境运行和建立网站了。 |
||||
|
||||
### 文档规范 |
||||
|
||||
1. 汉字与英文、数字之间**需空格**，中文标点符号与英文、数字之间**不需空格**，以增强中英文混排的美观性和可读性。 |
||||
|
||||
2. 建议在一般情况下使用 “你” 即可。当然必要的时候可以使用 “您” 来称呼，比如有 warning 提示的时候。 |
||||
|
||||
### 怎样提交文档 Pull Request |
||||
|
||||
1. 不要使用 “git add.” 提交所有更改。 |
||||
|
||||
2. 只需推送更改的文件，例如： |
||||
|
||||
* `*.md` |
||||
* `blog.js or docs.js or site.js` |
||||
|
||||
3. 向 **master** 分支提交 Pull Request。 |
||||
|
||||
### 参考文档 |
||||
|
||||
[Apache Flink 中文文档规范](https://cwiki.apache.org/confluence/display/FLINK/Flink+Translation+Specifications) |
||||
@ -0,0 +1,217 @@
|
||||
# Issue 须知 |
||||
|
||||
## 前言 |
||||
Issues 功能被用来追踪各种特性，Bug，功能等。项目维护者可以通过 Issues 来组织需要完成的任务。 |
||||
|
||||
Issue 是引出一个 Feature 或 Bug 等的重要步骤，在单个 |
||||
Issue 中可以讨论的内容包括但不限于 Feature 的包含的功能，存在的 Bug 产生原因，前期方案的调研，以及其对应的实现设计和代码思路。 |
||||
|
||||
并且只有当 Issue 被 approve 之后才需要有对应的 Pull Request 去实现。 |
||||
|
||||
如果是一个 Issue 对应的是一个大 Feature，建议先将其按照功能模块等维度分成多个小的 Issue。 |
||||
|
||||
## 规范 |
||||
|
||||
### Issue 标题 |
||||
|
||||
标题格式：[`Issue 类型`][`模块名`] `Issue 描述` |
||||
|
||||
其中`Issue 类型`如下： |
||||
|
||||
<table> |
||||
<thead> |
||||
<tr> |
||||
<th style="width: 10%; text-align: center;">Issue 类型</th> |
||||
<th style="width: 20%; text-align: center;">描述</th> |
||||
<th style="width: 20%; text-align: center;">样例</th> |
||||
</tr> |
||||
</thead> |
||||
<tbody> |
||||
<tr> |
||||
<td style="text-align: center;">Feature</td> |
||||
<td style="text-align: center;">包含所期望的新功能和新特性</td> |
||||
<td style="text-align: center;">[Feature][api] Add xxx api in xxx controller</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">Bug</td> |
||||
<td style="text-align: center;">程序中存在的 Bug</td> |
||||
<td style="text-align: center;">[Bug][api] Throw exception when xxx</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">Improvement</td> |
||||
<td style="text-align: center;">针对目前程序的一些改进，不限于代码格式，程序性能等</td> |
||||
<td style="text-align: center;">[Improvement][server] Improve xxx between Master and Worker</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">Test</td> |
||||
<td style="text-align: center;">专门针对测试用例部分</td> |
||||
<td style="text-align: center;">[Test][server] Add xxx e2e test</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">Sub-Task</td> |
||||
<td style="text-align: center;">一般都是属于 Feature 类的子任务，针对大 Feature，可以将其分成很多个小的子任务来一一完成</td> |
||||
<td style="text-align: center;">[Sub-Task][server] Implement xxx in xxx</td> |
||||
</tr> |
||||
</tbody> |
||||
</table> |
||||
|
||||
其中`模块名`如下： |
||||
|
||||
<table> |
||||
<thead> |
||||
<tr> |
||||
<th style="width: 10%; text-align: center;">模块名</th> |
||||
<th style="width: 20%; text-align: center;">描述</th> |
||||
</tr> |
||||
</thead> |
||||
<tbody> |
||||
<tr> |
||||
<td style="text-align: center;">alert</td> |
||||
<td style="text-align: center;">报警模块</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">api</td> |
||||
<td style="text-align: center;">应用程序接口层模块</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">service</td> |
||||
<td style="text-align: center;">应用程序服务层模块</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">dao</td> |
||||
<td style="text-align: center;">应用程序数据访问层模块</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">plugin</td> |
||||
<td style="text-align: center;">插件模块</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">remote</td> |
||||
<td style="text-align: center;">通信模块</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">server</td> |
||||
<td style="text-align: center;">服务器模块</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">ui</td> |
||||
<td style="text-align: center;">前端界面模块</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">docs-zh</td> |
||||
<td style="text-align: center;">中文文档</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">docs</td> |
||||
<td style="text-align: center;">英文文档</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">待补充...</td> |
||||
<td style="text-align: center;">-</td> |
||||
</tr> |
||||
</tbody> |
||||
</table> |
||||
|
||||
### Issue 内容模板 |
||||
|
||||
https://github.com/apache/dolphinscheduler/tree/dev/.github/ISSUE_TEMPLATE |
||||
|
||||
### Bug 类 Issue |
||||
|
||||
当您发现一个 Bug 时，请提交一个 Issue 类的 Bug，提交前： |
||||
* 请先在 issue 列表里查找一下是否该 Bug 已经提交，如果已经有此 Bug，请在此 Bug 下接着回复。 |
||||
* 如果该 Bug 是可以复现的。请尽量提供完整的重现步骤。 |
||||
|
||||
请在 issues 页面中提交 Bug。 |
||||
|
||||
一个高质量的 Bug 通常有以下特征： |
||||
|
||||
* 使用一个清晰并有描述性的标题来定义 Bug。 |
||||
* 详细的描述复现 Bug 的步骤。包括您的配置情况，预计产生的结果，实际产生的结果。并附加详细的 TRACE 日志。 |
||||
* 如果程序抛出异常，请附加完整的堆栈日志。 |
||||
* 如有可能，请附上屏幕截图或动态的 GIF 图，这些图片能帮助演示整个问题的产生过程。 |
||||
* 哪个版本。 |
||||
* 需要修复的优先级(危急、重大、次要、细微)。 |
||||
|
||||
下面是 **Bug 的 Markdown 内容模板**，请按照该模板填写 issue。 |
||||
|
||||
```shell |
||||
**标题** |
||||
标题格式: [BUG][Priority] bug标题 |
||||
Priority分为四级: Critical、Major、Minor、Trivial |
||||
|
||||
**问题描述** |
||||
[清晰准确描述遇到的问题] |
||||
|
||||
**问题复现步骤:** |
||||
1. [第一步] |
||||
2. [第二步] |
||||
3. [...] |
||||
|
||||
**期望的表现:** |
||||
[在这里描述期望的表现] |
||||
|
||||
**观察到的表现:** |
||||
[在这里描述观察到的表现] |
||||
|
||||
**屏幕截图和动态GIF图** |
||||
 |
||||
|
||||
**DolphinScheduler版本:(以1.1.0为例)** |
||||
-[1.1.0] |
||||
|
||||
**补充的内容:** |
||||
[请描述补充的内容，比如] |
||||
|
||||
**需求或者建议** |
||||
[请描述你的需求或者建议] |
||||
``` |
||||
|
||||
### Feature 类 Issue |
||||
|
||||
提交前： |
||||
* 请确定这不是一个重复的功能增强建议。 查看 Issue Page 列表，搜索您要提交的功能增强建议是否已经被提交过。 |
||||
|
||||
请在 issues 页面中提交 Feature。 |
||||
|
||||
一个高质量的 Feature 通常有以下特征： |
||||
* 一个清晰的标题来定义 Feature |
||||
* 详细描述 Feature 的行为模式 |
||||
* 说明为什么该 Feature 对大多数用户是有用的。新功能应该具有广泛的适用性。 |
||||
* 尽量列出其他调度已经具备的类似功能。商用与开源软件均可。 |
||||
|
||||
以下是 **Feature 的 Markdown 内容模板**，请按照该模板填写 issue 内容。 |
||||
```shell |
||||
**标题** |
||||
标题格式: [Feature][Priority] feature标题 |
||||
Priority分为四级: Critical、Major、Minor、Trivial |
||||
|
||||
**Feature的描述** |
||||
[描述新Feature应实现的功能] |
||||
|
||||
**为什么这个新功能是对大多数用户有用的** |
||||
[解释这个功能为什么对大多数用户是有用的] |
||||
|
||||
**补充的内容** |
||||
[列出其他的调度是否包含该功能，是如何实现的] |
||||
|
||||
``` |
||||
|
||||
|
||||
### Contributor |
||||
|
||||
除一些特殊情况之外，在开始完成 |
||||
Issue 之前，建议先在 Issue 下或者邮件列表中和大家讨论确定设计方案或者提供设计方案，以及代码实现思路。 |
||||
|
||||
如果存在多种不同的方案，建议通过邮件列表或者在 |
||||
Issue 下进行投票决定，最终方案和代码实现思路被 |
||||
approve 之后，再去实现，这样做的主要目的是避免在 |
||||
Pull Request review 阶段针对实现思路的意见不同或需要重构而导致 waste time。 |
||||
|
||||
### 相关问题 |
||||
|
||||
- 当出现提出 Issue 的用户不清楚该 Issue 对应的模块时的处理方式。 |
||||
|
||||
确实存在大多数提出 Issue 用户不清楚这个 Issue 是属于哪个模块的，其实这在很多开源社区都是很常见的。在这种情况下，其实 |
||||
committer/contributor 是知道这个 Issue 影响的模块的，如果之后这个 Issue 被 committer 和 contributor approve |
||||
确实有价值，那么 committer 就可以按照 Issue 涉及到的具体的模块去修改 Issue 标题，或者留言给提出 Issue 的用户去修改成对应的标题。 |
||||
@ -0,0 +1,98 @@
|
||||
# 微基准测试须知 |
||||
|
||||
所有的优化必须建立在数据印证的基础上，拒绝盲目优化。基于此，我们提供了MicroBench模块。 |
||||
|
||||
MicroBench模块是基于OpenJDK JMH构件的（HotSpot的推荐基准测试方案）。当你开始基准测试时，你不需要额外的依赖。 |
||||
|
||||
JMH，即Java MicroBenchmark Harness，是专门用于代码微基准测试的工具套件。何谓Micro Benchmark呢？简单的来说就是基于方法层面的基准测试，精度可以达到微秒级。当你定位到热点方法，希望进一步优化方法性能的时候，就可以使用JMH对优化的结果进行量化的分析。 |
||||
|
||||
### Java基准测试需要注意的几个点： |
||||
|
||||
* 防止无用代码进入测试方法中。 |
||||
|
||||
* 并发测试。 |
||||
|
||||
* 测试结果呈现。 |
||||
|
||||
### JMH比较典型的应用场景有： |
||||
|
||||
* 1:定量分析某个热点函数的优化效果 |
||||
|
||||
* 2:想定量地知道某个函数需要执行多长时间，以及执行时间和输入变量的相关性 |
||||
|
||||
* 3:对比一个函数的多种实现方式 |
||||
|
||||
|
||||
DolphinScheduler-MicroBench提供了AbstractBaseBenchmark,你可以在其基础上继承，编写你的基准测试代码，AbstractMicroBenchmark能保证以JUnit的方式运行。 |
||||
|
||||
### 定制运行参数 |
||||
|
||||
默认的AbstractMicrobenchmark配置是 |
||||
|
||||
Warmup次数 10（warmupIterations） |
||||
|
||||
测试次数 10（measureIterations） |
||||
|
||||
Fork数量 2 （forkCount） |
||||
|
||||
你可以在启动的时候指定这些参数，-DmeasureIterations、-DperfReportDir（输出基准测试结果文件目录）、-DwarmupIterations、-DforkCount |
||||
|
||||
### DolphinScheduler-MicroBench 介绍 |
||||
|
||||
|
||||
通常并不建议跑测试时，用较少的循环次数，但是较少的次数有助于确认基准测试时工作的，在确认结束后，再运行大量的基准测试。 |
||||
```java |
||||
@Warmup(iterations = 2, time = 1) |
||||
@Measurement(iterations = 4, time = 1) |
||||
@State(Scope.Benchmark) |
||||
public class EnumBenchMark extends AbstractBaseBenchmark { |
||||
|
||||
} |
||||
``` |
||||
这可以以方法级别或者类级别来运行基准测试，命令行的参数会覆盖annotation上的参数。 |
||||
|
||||
```java |
||||
@Benchmark //方法注解，表示该方法是需要进行 benchmark 的对象。 |
||||
@BenchmarkMode(Mode.AverageTime) //可选基准测试模式通过枚举Mode得到 |
||||
@OutputTimeUnit(TimeUnit.MICROSECONDS) // 输出的时间单位 |
||||
public void enumStaticMapTest() { |
||||
TestTypeEnum.newGetNameByType(testNum); |
||||
} |
||||
``` |
||||
|
||||
当你的基准测试编写完成后，你可以运行它查看具体的测试情况：（实际结果取决于你的系统配置情况） |
||||
|
||||
首先它会对我们的代码进行预热， |
||||
|
||||
``` |
||||
# Warmup Iteration 1: 0.007 us/op |
||||
# Warmup Iteration 2: 0.008 us/op |
||||
Iteration 1: 0.004 us/op |
||||
Iteration 2: 0.004 us/op |
||||
Iteration 3: 0.004 us/op |
||||
Iteration 4: 0.004 us/op |
||||
``` |
||||
在经过预热后，我们通常会得到如下结果 |
||||
```java |
||||
Benchmark (testNum) Mode Cnt Score Error Units |
||||
EnumBenchMark.simpleTest 101 thrpt 8 428750972.826 ± 66511362.350 ops/s |
||||
EnumBenchMark.simpleTest 108 thrpt 8 299615240.337 ± 290089561.671 ops/s |
||||
EnumBenchMark.simpleTest 103 thrpt 8 288423221.721 ± 130542990.747 ops/s |
||||
EnumBenchMark.simpleTest 104 thrpt 8 236811792.152 ± 155355935.479 ops/s |
||||
EnumBenchMark.simpleTest 105 thrpt 8 472247775.246 ± 45769877.951 ops/s |
||||
EnumBenchMark.simpleTest 103 thrpt 8 455473025.252 ± 61212956.944 ops/s |
||||
EnumBenchMark.enumStaticMapTest 101 avgt 8 0.006 ± 0.003 us/op |
||||
EnumBenchMark.enumStaticMapTest 108 avgt 8 0.005 ± 0.002 us/op |
||||
EnumBenchMark.enumStaticMapTest 103 avgt 8 0.006 ± 0.005 us/op |
||||
EnumBenchMark.enumStaticMapTest 104 avgt 8 0.006 ± 0.004 us/op |
||||
EnumBenchMark.enumStaticMapTest 105 avgt 8 0.004 ± 0.001 us/op |
||||
EnumBenchMark.enumStaticMapTest 103 avgt 8 0.004 ± 0.001 us/op |
||||
EnumBenchMark.enumValuesTest 101 avgt 8 0.011 ± 0.004 us/op |
||||
EnumBenchMark.enumValuesTest 108 avgt 8 0.025 ± 0.016 us/op |
||||
EnumBenchMark.enumValuesTest 103 avgt 8 0.019 ± 0.010 us/op |
||||
EnumBenchMark.enumValuesTest 104 avgt 8 0.018 ± 0.018 us/op |
||||
EnumBenchMark.enumValuesTest 105 avgt 8 0.014 ± 0.012 us/op |
||||
EnumBenchMark.enumValuesTest 103 avgt 8 0.012 ± 0.009 us/op |
||||
``` |
||||
|
||||
OpenJDK官方给了很多样例代码，有兴趣的同学可以自己查询并学习JMH：[OpenJDK-JMH-Example](http://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/) |
||||
@ -0,0 +1,95 @@
|
||||
# Pull Request 须知 |
||||
|
||||
## 前言 |
||||
Pull Request 本质上是一种软件的合作方式，是将涉及不同功能的代码，纳入主干的一种流程。这个过程中，可以进行讨论、审核和修改代码。 |
||||
|
||||
在 Pull Request 中尽量不讨论代码的实现方案，代码及其逻辑的大体实现方案应该尽量在 |
||||
Issue 或者邮件列表中被讨论确定，在 Pull Request 中我们尽量只关注代码的格式以及代码规范等信息，从而避免实现方式的意见不同而导致 |
||||
waste time。 |
||||
|
||||
## 规范 |
||||
|
||||
### Pull Request 标题 |
||||
|
||||
标题格式：[`Pull Request 类型`-`Issue 号`][`模块名`] `Pull Request 描述` |
||||
|
||||
其中`Pull Request 类型`和`Issue 类型`的对应关系如下： |
||||
|
||||
<table> |
||||
<thead> |
||||
<tr> |
||||
<th style="width: 10%; text-align: center;">Issue 类型</th> |
||||
<th style="width: 20%; text-align: center;">Pull Request 类型</th> |
||||
<th style="width: 20%; text-align: center;">样例（假设 Issue 号为 3333）</th> |
||||
</tr> |
||||
</thead> |
||||
<tbody> |
||||
<tr> |
||||
<td style="text-align: center;">Feature</td> |
||||
<td style="text-align: center;">Feature</td> |
||||
<td style="text-align: center;">[Feature-3333][server] Implement xxx</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">Bug</td> |
||||
<td style="text-align: center;">Fix</td> |
||||
<td style="text-align: center;">[Fix-3333][server] Fix xxx</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">Improvement</td> |
||||
<td style="text-align: center;">Improvement</td> |
||||
<td style="text-align: center;">[Improvement-3333][alert] Improve the performance of xxx</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">Test</td> |
||||
<td style="text-align: center;">Test</td> |
||||
<td style="text-align: center;">[Test-3333][api] Add the e2e test of xxx</td> |
||||
</tr> |
||||
<tr> |
||||
<td style="text-align: center;">Sub-Task</td> |
||||
<td style="text-align: center;">Sub-Task 对应的父类型</td> |
||||
<td style="text-align: center;">[Feature-3333][server] Implement xxx</td> |
||||
</tr> |
||||
</tbody> |
||||
</table> |
||||
|
||||
其中 `Issue 号`是指当前 Pull Request 对应要解决的 Issue 号，`模块名`同 Issue 的模块名。 |
||||
|
||||
### Pull Request 分支名 |
||||
|
||||
分支名格式：`Pull Request 类型`-`Issue 号`，举例：Feature-3333。 |
||||
|
||||
### Pull Request 内容 |
||||
|
||||
请参阅到 commit message 篇。 |
||||
|
||||
### Pull Request Code Style |
||||
|
||||
当你向 DolphinScheduler 提交 pull request 的时候 code-style 是你不得不考虑的问题。我们在 CI 中使用 Checkstyle [参考](https://checkstyle.sourceforge.io/)来保持代码风格的统一，它是一种帮助开发者编写遵循编码规范的 Java 代码开发工具。如果你的 pull request 没有通过 Checkstyle 的检测，那它将不会被合并到主库中。你可以在提交 pull request 前使用 Checkstyle 来检测或者格式化你的代码。如下的步骤将引领你配置并激活 Checkstyle |
||||
|
||||
1. 准备 Checkstyle 配置文件：你可以点击[这里](https://github.com/apache/dolphinscheduler/blob/dev/style/checkstyle.xml)手动下载，但是我们更加推荐在 DolphinScheduler 代码库中找到它。当你将代码库克隆下来后，你可以在路径 `style/checkstyle.xml` 下找到配置文件 |
||||
2. 下载 Intellij IDEA Checkstyle 插件：通过关键字**CheckStyle-IDEA**或者通过[这个页面](https://plugins.jetbrains.com/plugin/1065-checkstyle-idea)安装均可。如果你不清楚如何安装Intellij IDEA插件，可以参考[这个连接](https://www.jetbrains.com/help/idea/managing-plugins.html#install_plugin_from_repo) |
||||
3. 配置并激活 Checkstyles 以及 Intellij IDEA 代码风格：当完成上面几步后，你就可以配置并激活他们了。你可以在路径`Preferences -> Tool -> Checkstyle`中找到 Checkstyle，请参照下图完成其配置 |
||||
|
||||
<p align="center"> |
||||
<img src="../../../../img/contribute/join/pull-request/checkstyle-idea.png" alt="checkstyle idea configuration" /> |
||||
</p> |
||||
|
||||
截止目前，Checkstyle 插件已经配置完成了，当有代码或者文件不符合风格时就会显示在 Checkstyle 中。但强烈建议同时配置 Intellij IDEA 的代码风格，完成配置后可以使用 Intellij IDEA 自动格式化功能。你可以在路径`Preferences -> Editor -> Code Style -> Java`找到配置，请参照下图完成其配置 |
||||
|
||||
<p align="center"> |
||||
<img src="../../../../img/contribute/join/pull-request/code-style-idea.png" alt="code style idea configuration" /> |
||||
</p> |
||||
|
||||
1. 在提交 pull request 前格式化你的代码：完成上面全部后，你可以使用快捷键`Command + L`(Mac用户) or `Ctrl+L`(Windows用户)在 Intellij IDEA 完成自动格式化。格式化代码的最佳时间是将你的修改提交到本地 git 版本库之前 |
||||
|
||||
### 相关问题 |
||||
|
||||
- 怎样处理一个 Pull Request 对应多个 Issue 的场景。 |
||||
|
||||
首先 Pull Request 和 Issue 一对多的场景是比较少的。Pull Request 和 Issue 一对多的根本原因就是出现了多个 |
||||
Issue 需要做大体相同的一件事情的场景，通常针对这种场景有两种解决方法：第一种就是把多个功能相同的 Issue 合并到同一个 Issue 上，然后把其他的 |
||||
Issue 进行关闭；第二种就是多个 Issue 大体上是在做一个功能，但是存在一些细微的差别，这类场景下可以把每个 Issue 的职责划分清楚，每一个 |
||||
Issue 的类型都标记为 Sub-Task，然后将这些 Sub-Task 类型的 Issue 关联到一个总 Issue 上，在提交 |
||||
Pull Request 时，每个 Pull Request 都只关联一个 Sub-Task 的 Issue。 |
||||
|
||||
尽量把一个 Pull Request 作为最小粒度。如果一个 Pull Request 只做一件事，Contributor 容易完成，Pull Request 影响的范围也会更加清晰，对 reviewer 的压力也会小。 |
||||
@ -0,0 +1,141 @@
|
||||
# 参与社区 review |
||||
|
||||
贡献 DolphinScheduler 的方式，除了向 [团队](/us-en/community/community.html) 中提到的 GitHub 仓库提交 Issues 和 pull requests 外，另一非常重要的方式是 |
||||
review 社区的 Issues 或者 Pull Requests。通过别人 Issues 和 Pull Requests，你不仅能知道社区的最新进展和发展方向，还能了解别人代码的设 |
||||
计思想，同时可以增加自己在社区的曝光、积累自己在社区的荣誉值。 |
||||
|
||||
任何人都被鼓励去 review 社区的 Issues 和 Pull Requests。我们还曾经发起过一个 Help Wanted 的邮件讨论，向社区征求贡献者协助 review Issues |
||||
以及 Pull Requests，详见 [邮件][mail-review-wanted]，并将其结果放到了 [GitHub Discussion][discussion-result-review-wanted] 中。 |
||||
|
||||
> 注意: 这里并不是说只有 [GitHub Discussion][discussion-result-review-wanted] 中提及的用户才可以协助 review Issue 或者 Pull Requests， |
||||
> 请记住社区的主张是 **任何人都被鼓励去 review 社区的 Issues 和 Pull Requests**。只是那部分用户在邮件列表意见征集的时候，表达了愿意付 |
||||
> 出更多的时间，参与社区的 review。另一个好处是，当社区有不确定的问题的时，除了可以找 [团队](/us-en/community/community.html) 中对应的 Members 外，还可以找 |
||||
> [GitHub Discussion][discussion-result-review-wanted] 中提及的人解答对应的问题。如果你要想要加入到 [GitHub Discussion][discussion-result-review-wanted] |
||||
> 中，请在该 discussion 中评论并留下你感兴趣的模块，维护者会将你加入到对应的名单中。 |
||||
|
||||
## 怎么参与社区 review |
||||
|
||||
DolphinScheduler 主要通过 GitHub 接收社区的贡献，其所有的 Issues 和 Pull Requests 都托管在 GitHub 中，如果你想参与 Issues 的 review |
||||
具体请查看 [review Issues](#issues) 章节，如果你是想要参与 Pull Requests 的 review 具体请查看 [review Pull Requests](#pull-requests) |
||||
章节。 |
||||
|
||||
### Issues |
||||
|
||||
Review Issues 是指在 GitHub 中参与 [Issues][all-issues] 的讨论，并在对应的 Issues 给出建议。给出的建议包括但不限于如下的情况 |
||||
|
||||
| 情况 | 原因 | 需增加标签 | 需要的动作 | |
||||
| ------ | ------ | ------ | ------ | |
||||
| 不需要修改 | 问题在 dev 分支最新代码中已经修复了 | [wontfix][label-wontfix] | 关闭 Issue，告知提出者将在那个版本发布，如已发布告知版本 | |
||||
| 重复的问题 | 之前已经存在相同的问题 | [duplicate][label-duplicate] | 关闭 Issue，告知提出者相同问题的连接 | |
||||
| 问题描述不清晰 | 没有明确说明问题如何复现 | [need more information][label-need-more-information] | 提醒用户需要增加缺失的描述 | |
||||
|
||||
除了个 issue 建议之外，给 Issue 分类也是非常重要的一个工作。分类后的 Issue 可以更好的被检索，为以后进一步处理提供便利。一个 Issue 可以被打上多个标签，常见的 Issue 分类有 |
||||
|
||||
| 标签 | 标签代表的情况 | |
||||
| ------ | ------ | |
||||
| [UI][label-UI] | UI以及前端相关的 Issue | |
||||
| [security][label-security] | 安全相关的 Issue | |
||||
| [user experience][label-user-experience] | 用户体验相关的 Issue | |
||||
| [development][label-development] | 开发者相关的 Issue | |
||||
| [Python][label-Python] | Python相关的 Issue | |
||||
| [plug-in][label-plug-in] | 插件相关的 Issue | |
||||
| [document][label-document] | 文档相关的 Issue | |
||||
| [docker][label-docker] | docker相关的 Issue | |
||||
| [need verify][label-need-verify] | Issue 需要被验证 | |
||||
| [e2e][label-e2e] | e2e相关的 Issue | |
||||
| [win-os][label-win-os] | windows 操作系统相关的 Issue | |
||||
| [suggestion][label-suggestion] | Issue 为项目提出了建议 | |
||||
|
||||
标签除了分类之外，还能区分 Issue 的优先级，优先级越高的标签越重要，越容易被重视，并会尽快被修复或者实现，优先级的标签如下 |
||||
|
||||
| 标签 | 优先级 | |
||||
| ------ | ------ | |
||||
| [priority:high][label-priority-high] | 高优先级 | |
||||
| [priority:middle][label-priority-middle] | 中优先级 | |
||||
| [priority:low][label-priority-low] | 低优先级 | |
||||
|
||||
以上是常见的几个标签，更多的标签请查阅项目[全部的标签列表][label-all-list] |
||||
|
||||
在阅读以下内容是，请确保你已经为 Issue 打了标签。 |
||||
|
||||
* 回复后及时去掉标签[Waiting for reply][label-waiting-for-reply]：在 [创建 Issue 的时候][issue-choose]，我们会为 Issue 打上特定的标签 |
||||
[Waiting for reply][label-waiting-for-reply]，方便定位还没有被回复的 Issue，所以当你 review 了 Issue 之后，就需要将标签 |
||||
[Waiting for reply][label-waiting-for-reply] 及时的从 Issue 中删除。 |
||||
* 打上 [Waiting for review][label-waiting-for-review] 标当你不确定这个 Issue 是否被解决：当你查阅了 Issue 后，会有两个情况出现。一是 |
||||
问题已经被定位或解决，如果创建 Pull Requests 的话，则参考 [创建PR](./submit-code.md)。二是你也不确定这个问题是否真的是 |
||||
被解决，这时你可以为 Issue 打上 [Waiting for review][label-waiting-for-review] 标签，并在 Issue 中 `@` 对应的人进行二次确认 |
||||
|
||||
当 Issue 需要被创建 Pull Requests 解决，也可以视情况打上部分标签 |
||||
|
||||
| 标签 | 标签代表的PR | |
||||
| ------ | ------ | |
||||
| [Chore][label-Chore] | 日常维护工作 | |
||||
| [Good first issue][label-good-first-issue] | 适合首次贡献者解决的 Issue | |
||||
| [easy to fix][label-easy-to-fix] | 比较容易解决 | |
||||
| [help wanted][label-help-wanted] | 向社区寻求帮忙 | |
||||
|
||||
> 注意: 上面关于增加和删除标签的操作，目前只有成员可以操作，当你遇到需要增减标签的时候，但是不是成员是，可以 `@` 对应的成员让其帮忙增减。 |
||||
> 但只要你有 GitHub 账号就能评论 Issue，并给出建议。我们鼓励社区每人都去评论并为 Issue 给出解答 |
||||
|
||||
### Pull Requests |
||||
|
||||
<!-- markdown-link-check-disable --> |
||||
Review Pull 是指在 GitHub 中参与 [Pull Requests][all-PRs] 的讨论，并在对应的 Pull Requests 给出建议。DolphinScheduler review |
||||
Pull Requests 与 [GitHub 的 reviewing changes in pull requests][gh-review-pr] 一样。你可以为 Pull Requests 提出自己的看法， |
||||
|
||||
* 当你认为这个 Pull Requests 没有问题，可以被合并的时候，可以根据 [GitHub 的 reviewing changes in pull requests][gh-review-pr] 的 |
||||
approve 流程同意这个 Pull Requests。 |
||||
* 当你觉得这个 Pull Requests 需要被修改时，可以根据 [GitHub 的 reviewing changes in pull requests][gh-review-pr] 的 comment |
||||
流程评论这个 Pull Requests。当你认为存在一定要先修复才能合并的问题，请参照 [GitHub 的 reviewing changes in pull requests][gh-review-pr] |
||||
的 Request changes 流程要求贡献者修改 Pull Requests 的内容。 |
||||
<!-- markdown-link-check-enable --> |
||||
|
||||
为 Pull Requests 打上标签也是非常重要的一个环节，合理的分类能为后来的 reviewer 节省大量的时间。值得高兴的是，Pull Requests 的标签和 [Issues](#issues) |
||||
中提及的标签和用法是一致的，这能减少 reviewer 对标签的记忆。例如这个 Pull Requests 是和 docker 并且直接影响到用户部署的，我们可以为他 |
||||
打上 [docker][label-docker] 和 [priority:high][label-priority-high] 的标签。 |
||||
|
||||
除了和 Issue 类似的标签外，Pull Requests 还有许多自己特有的标签 |
||||
|
||||
| 标签 | 含义 | |
||||
| ------ | ------ | |
||||
| [miss document][label-miss-document] | 该 Pull Requests 缺少文档 需要增加 | |
||||
| [first time contributor][label-first-time-contributor] | 该 Pull Requests 贡献者是第一次贡献项目 | |
||||
| [don't merge][label-do-not-merge] | 该 Pull Requests 有问题 暂时先不要合并 | |
||||
|
||||
> 注意: 上面关于增加和删除标签的操作，目前只有成员可以操作，当你遇到需要增减标签的时候，可以 `@` 对应的成员让其帮忙增减。但只要你有 GitHub |
||||
> 账号就能评论 Pull Requests，并给出建议。我们鼓励社区每人都去评论并为 Pull Requests 给出建议 |
||||
|
||||
[mail-review-wanted]: https://lists.apache.org/thread/9flwlzrp69xjn6v8tdkbytq8glqp2k51 |
||||
[discussion-result-review-wanted]: https://github.com/apache/dolphinscheduler/discussions/7545 |
||||
[label-wontfix]: https://github.com/apache/dolphinscheduler/labels/wontfix |
||||
[label-duplicate]: https://github.com/apache/dolphinscheduler/labels/duplicate |
||||
[label-need-more-information]: https://github.com/apache/dolphinscheduler/labels/need%20more%20information |
||||
[label-win-os]: https://github.com/apache/dolphinscheduler/labels/win-os |
||||
[label-waiting-for-reply]: https://github.com/apache/dolphinscheduler/labels/Waiting%20for%20reply |
||||
[label-waiting-for-review]: https://github.com/apache/dolphinscheduler/labels/Waiting%20for%20review |
||||
[label-user-experience]: https://github.com/apache/dolphinscheduler/labels/user%20experience |
||||
[label-development]: https://github.com/apache/dolphinscheduler/labels/development |
||||
[label-UI]: https://github.com/apache/dolphinscheduler/labels/UI |
||||
[label-suggestion]: https://github.com/apache/dolphinscheduler/labels/suggestion |
||||
[label-security]: https://github.com/apache/dolphinscheduler/labels/security |
||||
[label-Python]: https://github.com/apache/dolphinscheduler/labels/Python |
||||
[label-plug-in]: https://github.com/apache/dolphinscheduler/labels/plug-in |
||||
[label-document]: https://github.com/apache/dolphinscheduler/labels/document |
||||
[label-docker]: https://github.com/apache/dolphinscheduler/labels/docker |
||||
[label-all-list]: https://github.com/apache/dolphinscheduler/labels |
||||
[label-Chore]: https://github.com/apache/dolphinscheduler/labels/Chore |
||||
[label-good-first-issue]: https://github.com/apache/dolphinscheduler/labels/good%20first%20issue |
||||
[label-help-wanted]: https://github.com/apache/dolphinscheduler/labels/help%20wanted |
||||
[label-easy-to-fix]: https://github.com/apache/dolphinscheduler/labels/easy%20to%20fix |
||||
[label-priority-high]: https://github.com/apache/dolphinscheduler/labels/priority%3Ahigh |
||||
[label-priority-middle]: https://github.com/apache/dolphinscheduler/labels/priority%3Amiddle |
||||
[label-priority-low]: https://github.com/apache/dolphinscheduler/labels/priority%3Alow |
||||
[label-miss-document]: https://github.com/apache/dolphinscheduler/labels/miss%20document |
||||
[label-first-time-contributor]: https://github.com/apache/dolphinscheduler/labels/first%20time%20contributor |
||||
[label-do-not-merge]: https://github.com/apache/dolphinscheduler/labels/don%27t%20merge |
||||
[label-e2e]: https://github.com/apache/dolphinscheduler/labels/e2e |
||||
[label-need-verify]: https://github.com/apache/dolphinscheduler/labels/need%20to%20verify |
||||
[issue-choose]: https://github.com/apache/dolphinscheduler/issues/new/choose |
||||
[all-issues]: https://github.com/apache/dolphinscheduler/issues |
||||
[all-PRs]: https://github.com/apache/dolphinscheduler/pulls |
||||
[gh-review-pr]: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/about-pull-request-reviews |
||||
@ -0,0 +1,8 @@
|
||||
# 安全 |
||||
|
||||
Apache Software Foundation在消除其软件项目中的安全性问题方面采取严格的立场。 Apache DolphinScheduler也非常关注与其功能有关的安全性问题。 |
||||
|
||||
如果您对DolphinScheduler的安全性有疑虑，或者发现了漏洞或潜在威胁，请发送邮件至[security@apache.org]（mailto：security@apache.org），与Apache安全团队联系。 请在电子邮件中将项目名称指定为DolphinScheduler，并提供相关问题或潜在威胁的描述。 还敦促您推荐重现和复制问题的方法。 在评估和分析调查结果之后，apache安全团队和DolphinScheduler社区将与您联系。 |
||||
|
||||
在公共领域公开该安全电子邮件之前，请注意在安全电子邮件中报告该安全问题。 |
||||
|
||||
@ -0,0 +1,71 @@
|
||||
# 提交代码 |
||||
|
||||
* 首先从远端仓库*https://github.com/apache/dolphinscheduler.git* fork一份代码到自己的仓库中 |
||||
|
||||
* 远端仓库中目前有三个分支： |
||||
* master 正常交付分支 |
||||
发布稳定版本以后，将稳定版本分支的代码合并到master上。 |
||||
|
||||
* dev 日常开发分支 |
||||
日常dev开发分支，新提交的代码都可以pull request到这个分支上。 |
||||
|
||||
* branch-1.0.0 发布版本分支 |
||||
发布版本分支，后续会有2.0...等版本分支。 |
||||
|
||||
* 把自己仓库clone到本地 |
||||
|
||||
` git clone https://github.com/apache/dolphinscheduler.git` |
||||
|
||||
* 添加远端仓库地址，命名为upstream |
||||
|
||||
` git remote add upstream https://github.com/apache/dolphinscheduler.git ` |
||||
|
||||
* 查看仓库： |
||||
|
||||
` git remote -v` |
||||
|
||||
> 此时会有两个仓库：origin(自己的仓库)和upstream（远端仓库） |
||||
|
||||
* 获取/更新远端仓库代码（已经是最新代码，就跳过） |
||||
|
||||
` git fetch upstream ` |
||||
|
||||
|
||||
* 同步远端仓库代码到本地仓库 |
||||
|
||||
``` |
||||
git checkout origin/dev |
||||
git merge --no-ff upstream/dev |
||||
``` |
||||
|
||||
如果远端分支有新加的分支比如`dev-1.0`,需要同步这个分支到本地仓库 |
||||
|
||||
``` |
||||
git checkout -b dev-1.0 upstream/dev-1.0 |
||||
git push --set-upstream origin dev-1.0 |
||||
``` |
||||
|
||||
* 新建分支 |
||||
|
||||
``` |
||||
git checkout -b xxx origin/dev |
||||
``` |
||||
|
||||
确保分支`xxx`是基于官方dev分支的最新代码 |
||||
|
||||
|
||||
* 在新建的分支上本地修改代码以后，提交到自己仓库： |
||||
|
||||
`git commit -m 'commit content'` |
||||
|
||||
`git push origin xxx --set-upstream` |
||||
|
||||
* 将修改提交到远端仓库 |
||||
|
||||
* 在github的PullRequest页面，点击"New pull request". |
||||
|
||||
* 选择修改完的本地分支和要合并的目的分支，点击"Create pull request". |
||||
|
||||
* 接着社区Committer们会做CodeReview，然后他会与您讨论一些细节（包括设计，实现，性能等）。当团队中所有人员对本次修改满意后，会将提交合并到dev分支 |
||||
|
||||
* 最后，恭喜您已经成为了dolphinscheduler的官方贡献者！ |
||||
@ -0,0 +1,25 @@
|
||||
# 订阅邮件列表 |
||||
|
||||
强烈推荐订阅开发邮件列表，与社区保持最新信息同步，这一点非常重要。 |
||||
|
||||
在使用DolphinScheduler的过程中，如果您有任何问题或者想法、建议，都可以通过Apache邮件列表参与到DolphinScheduler的社区建设中来。 |
||||
|
||||
发送订阅邮件也非常简单，步骤如下: |
||||
|
||||
1. 用自己的邮箱向[dev-subscribe@dolphinscheduler.apache.org](mailto:dev-subscribe@dolphinscheduler.apache.org)发送一封邮件，主题和内容任意。 |
||||
|
||||
2. 接收确认邮件并回复。 完成步骤1后，您将收到一封来自dev-help@dolphinscheduler.apache.org的确认邮件（如未收到，请确认邮件是否被自动归入垃圾邮件、推广邮件、订阅邮件等文件夹）。然后直接回复该邮件，或点击邮件里的链接快捷回复即可，主题和内容任意。 |
||||
|
||||
3. 接收欢迎邮件。 完成以上步骤后，您会收到一封主题为WELCOME to dev@dolphinscheduler.apache.org的欢迎邮件，至此您已成功订阅Apache DolphinScheduler的邮件列表。 |
||||
|
||||
# 取消订阅邮件列表 |
||||
|
||||
如果您不再需要了解DolphinScheduler的动态，可以取消订阅邮件列表。 |
||||
|
||||
取消订阅邮件列表步骤如下: |
||||
|
||||
1. 用已经订阅的邮箱向[dev-unsubscribe@dolphinscheduler.apache.org](mailto:dev-unsubscribe@dolphinscheduler.apache.org)发送一封邮件，主题和内容任意。 |
||||
|
||||
2. 接收确认邮件并回复。 完成步骤1后，您将收到一封来自dev-help@dolphinscheduler.apache.org的确认邮件（如未收到，请确认邮件是否被自动归入垃圾邮件、推广邮件、订阅邮件等文件夹）。然后直接回复该邮件，或点击邮件里的链接快捷回复即可，主题和内容任意。 |
||||
|
||||
3. 接收告别邮件。 完成以上步骤后，您会收到一封主题为GOODBYE from dev@dolphinscheduler.apache.org的告别邮件，至此您已成功取消订阅Apache DolphinScheduler的邮件列表，以后将不会再接收来自dev@dolphinscheduler.apache.org的邮件通知。 |
||||
@ -0,0 +1,110 @@
|
||||
## Unit Test 覆盖率 |
||||
Unit Test |
||||
### 1.写单元测试的收益 |
||||
* 单元测试能帮助每个人深入代码细节，了解代码的功能。 |
||||
* 通过测试用例我们能发现 bug，并提交代码的健壮性。 |
||||
* 测试用例同时也是代码的 demo 用法。 |
||||
### 2.单元测试用例的一些设计原则 |
||||
* 应该精心设计好步骤，颗粒度和组合条件。 |
||||
* 注意边界条件。 |
||||
* 单元测试也应该好好设计，不要写无用的代码。 |
||||
* 当你发现一个`方法`很难写单元测试时，如果可以确认这个`方法`是`臭代码`，那么就和开发者一起重构它。 |
||||
<!-- markdown-link-check-disable --> |
||||
* DolphinScheduler: [mockito](http://site.mockito.org/). 下面是一些开发向导: [mockito tutorial](https://www.baeldung.com/bdd-mockito), [mockito refcard](https://dzone.com/refcardz/mockito) |
||||
<!-- markdown-link-check-enable --> |
||||
* TDD（可选）：当你开始写一个新的功能时，你可以试着先写测试用例。 |
||||
### 3.测试覆盖率设定值 |
||||
* 在现阶段，Delta 更改代码的测试覆盖设定值为：>＝60%，越高越好。 |
||||
* 我们可以在这个页面中看到测试报告: https://codecov.io/gh/apache/dolphinscheduler |
||||
|
||||
## 单元测试基本准则 |
||||
### 1： 隔离性与单一性 |
||||
|
||||
一个测试用例应该精确到方法级别，并应该能够单独执行该测试用例。同时关注点也始终在该方法上（只测试该方法）。 |
||||
|
||||
如果方法过于复杂，开发阶段就应该将其再次进行拆分，对于测试用例来讲，最佳做法是一个用例只关注一个分支（判断）。当对其进行修改后，也仅仅影响一个测试用例的成功与否。这会极大方便我们在开发阶段验证问题和解决问题，但与此同时，也对我们覆盖率提出了极大的挑战。 |
||||
|
||||
### 2：自动性 |
||||
|
||||
单元测试能够自动化进行。强制要求：所有的单元测试必须写在 src/test 下，同时方法命名应该符合规范。基准测试除外。 |
||||
|
||||
### 3：可重复性 |
||||
|
||||
多次执行（任何环境任何时间）结果唯一，且可以重复执行。 |
||||
|
||||
### 4：轻量型 |
||||
|
||||
即任何环境都可快速执行。 |
||||
|
||||
这要求我们尽可能不要依赖太多组件，如各种 spring bean 之类的。在单元测试中，这些都是可被 mock 的，增加这些，会加大我们单测的执行速度，同时也可能会传递污染。 |
||||
|
||||
对于一些数据库、其他外部组件等。尽可能也采用模拟客户端的形式，即不依赖于外部环境，（任何外部依赖的存在都会极大的限制测试用例的可迁移性和稳定性以及结果正确性），这同时也方便开发者在任何环境都能够进行测试。 |
||||
|
||||
### 5： 可测性 |
||||
|
||||
这么多年过去了，你所看到的 mockito 已经成长为 mock 界的 NO.1 了，但他依然不支持 mock 静态方法、构造方法等。甚至官网上一直写着： "Don’t mock everything" 。因此尽量少用静态方法。 |
||||
|
||||
一般建议只在一些工具类提供静态方法，这种情况下也不需要 mock，直接使用真实类即可。如果被依赖类不是工具类，可以将静态方法重构为实例方法。这样更加符合面向对象的设计理念。 |
||||
|
||||
### 6： 完备性 |
||||
|
||||
测试覆盖率，这是个非常费劲的问题，对于核心流程，我们是希望能够达到 90% 的覆盖率，非核心流程要求 60% 以上。 |
||||
|
||||
覆盖率足够高的情况下会减少足够多的 bug 出现的概率，同时也减少了我们回归测试的成本。这是一个长久的工作，每当开发者新增或者修改代码的时候，相关测试用例与此同时也需要完善。这一点，希望开发者以及相关代码 reviewer 都能足够重视。 |
||||
|
||||
### 7：拒绝无效断言 |
||||
|
||||
无效断言让测试本身变得毫无意义，它和你的代码正确与否几乎没什么关系，且有可能会给你造成一种成功的假象，这种假象有可能持续到你的代码部署到生产环境。 |
||||
|
||||
关于无效的断言这么几种类型 |
||||
|
||||
1:不同类型的比较。 |
||||
|
||||
2:判断一个具有默认值的对象或者变量不为空。 |
||||
|
||||
这本身显得毫无意义，因此，在进行相关判断的时候应该关注一下其本身是否含有默认值。 |
||||
|
||||
3:断言尽可能采用肯定断言而非否定断言，断言尽可能在一个预知结果范围内,或者是准确的数值，（否则有可能会导致一些不符合你的实际预期但是通过了断言）除非你的代码只关心他是否为空。 |
||||
|
||||
### 8:一些单测的注意点 |
||||
1:Thread.sleep() |
||||
|
||||
测试代码中尽量不要使用 Thread.sleep，这让测试变得不稳定，可能会因为环境或者负载而意外导致失败。建议采用以下方式： |
||||
|
||||
Awaitility.await().atMost(…) |
||||
|
||||
2:忽略某些测试类 |
||||
|
||||
@Ignore 注解应该附上相关 issue 地址，方便后续开发者追踪了解该测试被忽略的历史原因。 |
||||
|
||||
如 @Ignore("see #1") |
||||
|
||||
3: try-catch 单元测试异常 |
||||
|
||||
当单元测试中的代码引发异常的时候，测试将失败,因此，不需要使用 try-catch 捕获异常。 |
||||
|
||||
``` |
||||
@Test |
||||
public void testMethod() { |
||||
try { |
||||
// Some code |
||||
} catch (MyException e) { |
||||
Assert.fail(e.getMessage()); // Noncompliant |
||||
} |
||||
} |
||||
``` |
||||
你应该这样做： |
||||
``` |
||||
@Test |
||||
public void testMethod() throws MyException { |
||||
// Some code |
||||
} |
||||
``` |
||||
4:测试异常情况 |
||||
|
||||
当你需要进行异常情况测试时，应该避免在测试代码中包含多个方法的调用（尤其是有多个可以引发相同异常的方法），同时应该明确说明你要测试什么。 |
||||
|
||||
5:拒绝使用 MockitoJUnitRunner.Silent.class |
||||
|
||||
当单测出现 UnnecessaryStubbingException 时，请不要第一时间考虑使用 @RunWith(MockitoJUnitRunner.Silent.class) 来解决它，这只是隐藏了问题， |
||||
你应该根据异常提示解决相关问题，这并不是一个困难的工作。当完成更改时，你会发现，你的代码又简洁了许多。 |
||||
@ -0,0 +1,30 @@
|
||||
# 发版后续 |
||||
|
||||
发送公告邮件后，我们还有一些发布任务要做，目前我们必须将 Docker 镜像发布到 Docker Hub 和 并且需要将 pydolphinscheduler 发布到 PyPI。 |
||||
|
||||
## 发布 Docker 镜像 |
||||
|
||||
我们已经有 CI 发布最新的 Docker 镜像到 GitHub container register [点击查看详情](https://github.com/apache/dolphinscheduler/blob/d80cf21456265c9d84e642bdb4db4067c7577fc6/.github/workflows/publish-docker.yaml#L55-L63)。 |
||||
我们可以稍微修改 CI 的主要命令实现单个命令发布 Docker 镜像发布到 Docker Hub。 |
||||
|
||||
```bash |
||||
# 请将 <VERSION> 修改成你要发版的版本 |
||||
./mvnw -B clean deploy \ |
||||
-Dmaven.test.skip \ |
||||
-Dmaven.javadoc.skip \ |
||||
-Dmaven.checkstyle.skip \ |
||||
-Dmaven.deploy.skip \ |
||||
-Ddocker.tag=<VERSION> \ |
||||
-Ddocker.hub=apache \ |
||||
-Pdocker,release |
||||
``` |
||||
|
||||
## 发布 pydolphinscheduler 到 PyPI |
||||
|
||||
需要将 Python API 发布到 PyPI，请参考 [Python API release](https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-python/pydolphinscheduler/RELEASE.md#to-pypi) |
||||
完成 PyPI 的发版 |
||||
|
||||
## 获取全部的贡献者 |
||||
|
||||
当您想要发布新版本的新闻或公告时，您可能需要当前版本的所有贡献者，您可以使用 git 命令 `git log --pretty="%an" <PREVIOUS-RELEASE-SHA>..<CURRENT-RELEASE-SHA> | sort | uniq` |
||||
（将对应的版本改成两个版本的 tag 值）自动生成 git 作者姓名。 |
||||
@ -0,0 +1,32 @@
|
||||
# 发版准备 |
||||
|
||||
## 检查 release-docs |
||||
|
||||
和上一个版本比较，如果有依赖及版本发生了变化，当前版本的 `release-docs` 需要被更新到最新 |
||||
|
||||
- `dolphinscheduler-dist/release-docs/LICENSE` |
||||
- `dolphinscheduler-dist/release-docs/NOTICE` |
||||
- `dolphinscheduler-dist/release-docs/licenses` |
||||
|
||||
## 更新版本 |
||||
|
||||
例如要发版 `x.y.z`，需要先进行以下修改: |
||||
|
||||
- 修改代码中的版本号: |
||||
- `sql`: |
||||
- `dolphinscheduler_mysql.sql`: `t_ds_version` 版本更新为 x.y.z |
||||
- `dolphinscheduler_postgre.sql`: `t_ds_version` 版本更新为 x.y.z |
||||
- `dolphinscheduler_h2.sql`: `t_ds_version` 版本更新为 x.y.z |
||||
- `upgrade`: 是否新增 `x.y.z_schema` |
||||
- `soft_version`: 版本更新为 x.y.z |
||||
- `deploy/docker/.env`: `HUB` 改为 `apache`，`TAG` 改为 `x.y.z` |
||||
- `deploy/kubernetes/dolphinscheduler`: |
||||
- `Chart.yaml`: `appVersion` 版本更新为 x.y.z (`version` 为 helm chart 版本, 增量更新但不要设置为 x.y.z) |
||||
- `values.yaml`: `image.tag` 版本更新为 x.y.z |
||||
- `dolphinscheduler-python/pydolphinscheduler/setup.py`: 修改其中的 `version` 为 x.y.z |
||||
- 修改文档（docs模块）中的版本号: |
||||
- 将 `docs` 文件夹下文件的占位符 `<version>` (除了 pom.xml 相关的) 修改成 `x.y.z` |
||||
- 新增历史版本 |
||||
- `docs/docs/en/history-versions.md` 和 `docs/docs/zh/history-versions.md`: 增加新的历史版本为 `x.y.z` |
||||
- 修改文档 sidebar |
||||
- `docs/configs/docsdev.js`: 将里面的 `/dev/` 修改成 `/x.y.z/` |
||||
@ -0,0 +1,534 @@
|
||||
# 发版指南 |
||||
|
||||
## 检查环境 |
||||
|
||||
为确保您可以成功完成 DolphinScheduler 的发布，您应该检查您的环境并确保满足所有条件，如果缺少任何条件，您应该安装它们并确保它们正常工作。 |
||||
|
||||
```shell |
||||
# 需要 JDK 1.8 及以上的版本 |
||||
java -version |
||||
# 需要 Maven |
||||
mvn -version |
||||
# 需要 Python 3.6 及以上的版本，并且需要 `python` 关键字能在命令行中运行，且版本符合条件。 |
||||
python --version |
||||
``` |
||||
|
||||
## GPG设置 |
||||
|
||||
### 安装GPG |
||||
|
||||
在[GnuPG官网](https://www.gnupg.org/download/index.html)下载安装包。 |
||||
GnuPG的1.x版本和2.x版本的命令有细微差别，下列说明以`GnuPG-2.1.23`版本为例。 |
||||
|
||||
安装完成后，执行以下命令查看版本号。 |
||||
|
||||
```shell |
||||
gpg --version |
||||
``` |
||||
|
||||
### 创建key |
||||
|
||||
安装完成后，执行以下命令创建key。 |
||||
|
||||
`GnuPG-2.x`可使用： |
||||
|
||||
```shell |
||||
gpg --full-gen-key |
||||
``` |
||||
|
||||
`GnuPG-1.x`可使用： |
||||
|
||||
```shell |
||||
gpg --gen-key |
||||
``` |
||||
|
||||
根据提示完成key，**注意：请使用Apache mail 和 对应的密码生成GPG的Key。** |
||||
|
||||
```shell |
||||
gpg (GnuPG) 2.0.12; Copyright (C) 2009 Free Software Foundation, Inc. |
||||
This is free software: you are free to change and redistribute it. |
||||
There is NO WARRANTY, to the extent permitted by law. |
||||
|
||||
Please select what kind of key you want: |
||||
(1) RSA and RSA (default) |
||||
(2) DSA and Elgamal |
||||
(3) DSA (sign only) |
||||
(4) RSA (sign only) |
||||
Your selection? 1 |
||||
RSA keys may be between 1024 and 4096 bits long. |
||||
What keysize do you want? (2048) 4096 |
||||
Requested keysize is 4096 bits |
||||
Please specify how long the key should be valid. |
||||
0 = key does not expire |
||||
<n> = key expires in n days |
||||
<n>w = key expires in n weeks |
||||
<n>m = key expires in n months |
||||
<n>y = key expires in n years |
||||
Key is valid for? (0) |
||||
Key does not expire at all |
||||
Is this correct? (y/N) y |
||||
|
||||
GnuPG needs to construct a user ID to identify your key. |
||||
|
||||
Real name: ${输入用户名} |
||||
Email address: ${输入邮件地址} |
||||
Comment: ${输入注释} |
||||
You selected this USER-ID: |
||||
"${输入的用户名} (${输入的注释}) <${输入的邮件地址}>" |
||||
|
||||
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O |
||||
You need a Passphrase to protect your secret key. # 输入apache登录密码 |
||||
``` |
||||
注意：如果遇到以下错误： |
||||
``` |
||||
gpg: cancelled by user |
||||
gpg: Key generation canceled. |
||||
``` |
||||
需要使用自己的用户登录服务器，而不是root切到自己的账户 |
||||
|
||||
### 查看生成的key |
||||
|
||||
```shell |
||||
gpg --list-keys |
||||
``` |
||||
|
||||
执行结果： |
||||
|
||||
```shell |
||||
pub 4096R/85E11560 2019-11-15 |
||||
uid ${用户名} (${注释}) <{邮件地址}> |
||||
sub 4096R/A63BC462 2019-11-15 |
||||
``` |
||||
|
||||
其中85E11560为公钥ID。 |
||||
|
||||
### 将公钥同步到服务器 |
||||
|
||||
命令如下： |
||||
|
||||
```shell |
||||
gpg --keyserver hkp://pool.sks-keyservers.net --send-key 85E11560 |
||||
``` |
||||
|
||||
`pool.sks-keyservers.net`为随意挑选的[公钥服务器](https://sks-keyservers.net/status/)，每个服务器之间是自动同步的，选任意一个即可。 |
||||
|
||||
注意：如果同步到公钥服务器，可以在服务器上查到新建的公钥 |
||||
http://keyserver.ubuntu.com:11371/pks/lookup?search=${用户名}&fingerprint=on&op=index |
||||
备用公钥服务器 gpg --keyserver hkp://keyserver.ubuntu.com --send-key ${公钥ID} |
||||
|
||||
|
||||
## 发布Apache Maven中央仓库 |
||||
|
||||
### 设置 `settings-security.xml` 和 `settings.xml` 文件 |
||||
|
||||
在本节中，我们添加 Apache 服务器 maven 配置以准备发布，请参考[这里](http://maven.apache.org/guides/mini/guide-encryption.html) 添加 |
||||
`settings-security.xml` 文件，并且像下面这样更改你的 `~/.m2/settings.xml` |
||||
|
||||
```xml |
||||
<settings> |
||||
<servers> |
||||
<server> |
||||
<id>apache.snapshots.https</id> |
||||
<username> <!-- APACHE LDAP 用户名 --> </username> |
||||
<password> <!-- APACHE LDAP 加密后的密码 --> </password> |
||||
</server> |
||||
<server> |
||||
<id>apache.releases.https</id> |
||||
<username> <!-- APACHE LDAP 用户名 --> </username> |
||||
<password> <!-- APACHE LDAP 加密后的密码 --> </password> |
||||
</server> |
||||
</servers> |
||||
</settings> |
||||
``` |
||||
|
||||
### 配置环境变量 |
||||
|
||||
我们将多次使用发布版本 `VERSION`，github名称 `GH_USERNAME`，以及 Apache 用户名 `<YOUR-APACHE-USERNAME>`，因此最好将其存储到bash变量中以便于使用。 |
||||
|
||||
```shell |
||||
VERSION=<THE-VERSION-YOU-RELEASE> |
||||
GH_USERNAME=<YOUR-GITHUB-USERNAME> |
||||
A_USERNAME=<YOUR-APACHE-USERNAME> |
||||
``` |
||||
|
||||
> 注意：设置环境变量后，我们可以直接在你的 bash 中使用该变量，而无需更改任何内容。例如，我们可以直接使用命令 `git clone -b "${VERSION}"-prepare https://github.com/apache/dolphinscheduler.git` |
||||
> 来克隆发布分支，他会自动将其中的 `"${VERSION}"` 转化成你设置的值 `<THE-VERSION-YOU-RELEASE>`。 但是您必须在一些非 bash 步骤中手动更改 |
||||
> `<VERSION>` 为对应的版本号，例如发起投票中的内容。我们使用 `<VERSION>` 而不是 `"${VERSION}"` 来提示 release manager 他们必须手动更改这部分内容 |
||||
|
||||
### 创建发布分支 |
||||
|
||||
在本节中，我们从 github 下载源代码并创建新分支以发布 |
||||
|
||||
```shell |
||||
git clone -b "${VERSION}"-prepare https://github.com/apache/dolphinscheduler.git |
||||
cd ~/dolphinscheduler/ |
||||
git pull |
||||
git checkout -b ${RELEASE.VERSION}-release |
||||
git push origin ${RELEASE.VERSION}-release |
||||
``` |
||||
|
||||
### 发布预校验 |
||||
|
||||
```shell |
||||
# 保证 python profile 的 gpg 可以正常运行 |
||||
export GPG_TTY=$(tty) |
||||
|
||||
# 运行发版校验 |
||||
mvn release:prepare -Prelease,python -Darguments="-Dmaven.test.skip=true -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true" -DautoVersionSubmodules=true -DdryRun=true -Dusername="${GH_USERNAME}" |
||||
``` |
||||
|
||||
* `-Prelease,python`: 选择release和python的profile，这个profile会打包所有源码、jar文件以及可执行二进制包，以及Python的二进制包。 |
||||
* `-DautoVersionSubmodules=true`: 作用是发布过程中版本号只需要输入一次，不必为每个子模块都输入一次。 |
||||
* `-DdryRun=true`: 演练，即不产生版本号提交，不生成新的tag。 |
||||
|
||||
### 准备发布 |
||||
|
||||
首先清理发布预校验本地信息。 |
||||
|
||||
```shell |
||||
mvn release:clean |
||||
``` |
||||
|
||||
然后准备执行发布。 |
||||
|
||||
```shell |
||||
mvn release:prepare -Prelease,python -Darguments="-Dmaven.test.skip=true -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true" -DautoVersionSubmodules=true -DpushChanges=false -Dusername="${GH_USERNAME}" |
||||
``` |
||||
|
||||
和上一步演练的命令基本相同，去掉了 `-DdryRun=true` 参数。 |
||||
|
||||
* `-DpushChanges=fals`:不要将修改后的版本号和tag自动提交至GitHub。 |
||||
|
||||
> 注意：如果你遇到来自 git 的类似 **Please tell me who you are.** 错误信息。您可以通过命令 `git config --global user.email "you@example.com"` |
||||
> 和 `git config --global user.name "Your Name"` 来配置你的用户名和邮箱如果你遇到一些错误。 |
||||
|
||||
将本地文件检查无误后，提交至github。 |
||||
|
||||
```shell |
||||
git push -u origin "${VERSION}"-release |
||||
git push origin --tags |
||||
``` |
||||
|
||||
<!-- markdown-link-check-disable --> |
||||
|
||||
> 注意1：因为 Github 不再支持在 HTTPS 协议中使用原生密码在，所以在这一步你应该使用 github token 作为密码。你可以通过 https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating -a-personal-access-token |
||||
> 了解更多如果创建 token 的信息。 |
||||
|
||||
> 注意2：命令完成后，会自动创建 `release.properties` 文件和 `*.Backup` 文件，它们在下面的命令中是需要的，不要删除它们 |
||||
|
||||
<!-- markdown-link-check-enable --> |
||||
|
||||
### 部署发布 |
||||
|
||||
```shell |
||||
mvn release:perform -Prelease,python -Darguments="-Dmaven.test.skip=true -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true" -DautoVersionSubmodules=true -Dusername="${GH_USERNAME}" |
||||
``` |
||||
|
||||
执行完该命令后，待发布版本会自动上传到Apache的临时筹备仓库(staging repository)。你可以通过访问 [apache staging repositories](https://repository.apache.org/#stagingRepositories) |
||||
, 然后使用Apache的LDAP账户登录后，就会看到上传的版本，`Repository` 列的内容即为 `${STAGING.REPOSITORY}`。 |
||||
点击 `Close` 来告诉Nexus这个构建已经完成，只有这样该版本才是可用的。如果电子签名等出现问题，`Close` 会失败，可以通过 `Activity` 查看失败信息。 |
||||
|
||||
## 发布Apache SVN仓库 |
||||
|
||||
### 检出dolphinscheduler发布目录 |
||||
|
||||
如无本地工作目录，则先创建本地工作目录。 |
||||
|
||||
```shell |
||||
mkdir -p ~/ds_svn/dev/ |
||||
cd ~/ds_svn/dev/ |
||||
``` |
||||
|
||||
创建完毕后，从Apache SVN检出dolphinscheduler发布目录。 |
||||
|
||||
```shell |
||||
svn --username="${A_USERNAME}" co https://dist.apache.org/repos/dist/dev/dolphinscheduler |
||||
cd ~/ds_svn/dev/dolphinscheduler |
||||
``` |
||||
|
||||
### 添加gpg公钥 |
||||
|
||||
仅第一次部署的账号需要添加，只要`KEYS`中包含已经部署过的账户的公钥即可。 |
||||
|
||||
```shell |
||||
gpg -a --export <YOUR-GPG-KEY-ID> >> KEYS |
||||
``` |
||||
|
||||
### 将待发布的内容添加至SVN目录 |
||||
|
||||
创建版本号目录。 |
||||
|
||||
```shell |
||||
mkdir -p ~/ds_svn/dev/dolphinscheduler/"${VERSION}" |
||||
mkdir -p ~/ds_svn/dev/dolphinscheduler/"${VERSION}"/python |
||||
cd ~/ds_svn/dev/dolphinscheduler/"${VERSION}" |
||||
``` |
||||
|
||||
将源码包和二进制包添加至SVN工作目录。 |
||||
|
||||
```shell |
||||
# 主程序源码包和二进制包 |
||||
cp -f ~/dolphinscheduler/dolphinscheduler-dist/target/*.tar.gz ~/ds_svn/dev/dolphinscheduler/"${VERSION}" |
||||
cp -f ~/dolphinscheduler/dolphinscheduler-dist/target/*.tar.gz.asc ~/ds_svn/dev/dolphinscheduler/"${VERSION}" |
||||
|
||||
# Python API 源码和二进制包 |
||||
cp -f ~/dolphinscheduler/dolphinscheduler-dist/target/python/* ~/ds_svn/dev/dolphinscheduler/"${VERSION}"/python |
||||
``` |
||||
|
||||
### 生成文件签名 |
||||
|
||||
```shell |
||||
shasum -a 512 apache-dolphinscheduler-"${VERSION}"-src.tar.gz >> apache-dolphinscheduler-"${VERSION}"-src.tar.gz.sha512 |
||||
shasum -b -a 512 apache-dolphinscheduler-"${VERSION}"-bin.tar.gz >> apache-dolphinscheduler-"${VERSION}"-bin.tar.gz.sha512 |
||||
cd python |
||||
shasum -a 512 apache-dolphinscheduler-python-"${VERSION}".tar.gz >> apache-dolphinscheduler-python-"${VERSION}".tar.gz.sha512 |
||||
shasum -b -a 512 apache_dolphinscheduler-python-"${VERSION}"-py3-none-any.whl >> apache_dolphinscheduler-python-"${VERSION}"-py3-none-any.whl.sha512 |
||||
cd ../ |
||||
``` |
||||
|
||||
### 提交Apache SVN |
||||
|
||||
```shell |
||||
cd ~/ds_svn/dev/dolphinscheduler |
||||
svn add * |
||||
svn --username="${A_USERNAME}" commit -m "release ${VERSION}" |
||||
``` |
||||
## 检查发布结果 |
||||
|
||||
### 检查sha512哈希 |
||||
|
||||
```shell |
||||
shasum -c apache-dolphinscheduler-"${VERSION}"-src.tar.gz.sha512 |
||||
shasum -c apache-dolphinscheduler-"${VERSION}"-bin.tar.gz.sha512 |
||||
cd python |
||||
shasum -c apache-dolphinscheduler-python-"${VERSION}".tar.gz.sha512 |
||||
shasum -c apache_dolphinscheduler-python-"${VERSION}"-py3-none-any.whl.sha512 |
||||
cd ../ |
||||
``` |
||||
|
||||
### 检查gpg签名 |
||||
|
||||
首先导入发布人公钥。从svn仓库导入KEYS到本地环境。（发布版本的人不需要再导入，帮助做验证的人需要导入，用户名填发版人的即可） |
||||
|
||||
```shell |
||||
curl https://dist.apache.org/repos/dist/dev/dolphinscheduler/KEYS >> KEYS |
||||
gpg --import KEYS |
||||
gpg --edit-key "${A_USERNAME}" |
||||
> trust |
||||
|
||||
Please decide how far you trust this user to correctly verify other users' keys |
||||
(by looking at passports, checking fingerprints from different sources, etc.) |
||||
|
||||
1 = I don't know or won't say |
||||
2 = I do NOT trust |
||||
3 = I trust marginally |
||||
4 = I trust fully |
||||
5 = I trust ultimately |
||||
m = back to the main menu |
||||
|
||||
Your decision? 5 |
||||
|
||||
> save |
||||
``` |
||||
|
||||
然后进行gpg签名检查。 |
||||
|
||||
```shell |
||||
gpg --verify apache-dolphinscheduler-"${VERSION}"-src.tar.gz.asc |
||||
gpg --verify apache-dolphinscheduler-"${VERSION}"-bin.tar.gz.asc |
||||
cd python |
||||
gpg --verify apache-dolphinscheduler-python-"${VERSION}".tar.gz.asc |
||||
gpg --verify apache_dolphinscheduler-python-"${VERSION}"-py3-none-any.whl.asc |
||||
cd ../ |
||||
``` |
||||
|
||||
> 注意：当你找不到你的 `asc` 文件时，你必须手动创建 gpg 签名，命令 `gpg --armor --detach-sign --digest-algo=SHA512 apache-dolphinscheduler-"${VERSION}"- bin.tar.gz` |
||||
> 和 `gpg --armor --detach-sign --digest-algo=SHA512 apache-dolphinscheduler-"${VERSION}"-src.tar.gz` 将创建它们 |
||||
|
||||
### 检查发布文件内容 |
||||
|
||||
#### 检查源码包的文件内容 |
||||
|
||||
解压缩`apache-dolphinscheduler-<VERSION>-src.tar.gz`以及Python文件夹下的`apache-dolphinscheduler-python-<VERSION>.tar.gz`，进行如下检查: |
||||
|
||||
- 检查源码包是否包含由于包含不必要文件，致使tarball过于庞大 |
||||
- 存在`LICENSE`和`NOTICE`文件 |
||||
- 只存在文本文件，不存在二进制文件 |
||||
- 所有文件的开头都有ASF许可证 |
||||
- 能够正确编译，单元测试可以通过 (mvn install) |
||||
- 版本内容与GitHub上tag的内容相符 (diff -r a verify_dir tag_dir) |
||||
- 检查是否有多余文件或文件夹，例如空文件夹等 |
||||
|
||||
#### 检查二进制包的文件内容 |
||||
|
||||
解压缩`apache-dolphinscheduler-<VERSION>-src.tar.gz`和`apache-dolphinscheduler-python-<VERSION>-bin.tar.gz` |
||||
进行如下检查: |
||||
|
||||
- 存在`LICENSE`和`NOTICE`文件 |
||||
- 所有文本文件开头都有ASF许可证 |
||||
- 检查第三方依赖许可证： |
||||
- 第三方依赖的许可证兼容 |
||||
- 所有第三方依赖的许可证都在`LICENSE`文件中声明 |
||||
- 依赖许可证的完整版全部在`license`目录 |
||||
- 如果依赖的是Apache许可证并且存在`NOTICE`文件，那么这些`NOTICE`文件也需要加入到版本的`NOTICE`文件中 |
||||
|
||||
## 发起投票 |
||||
|
||||
### 更新版本说明 |
||||
|
||||
在 GitHub 中通过 [创建新的 release note](https://github.com/apache/dolphinscheduler/releases/new) 创建一个 release note。 这要在 |
||||
投票邮件开始之前完成，因为我们需要在邮件中使用 release note。你可以通过命令 `git log --pretty="- %s" <PREVIOUS-RELEASE-SHA>..<CURRENT-RELEASE-SHA> > changelog.md` |
||||
自动生成 changelog（部分可以不太准确，需要人为过滤一遍），然后将他们分类并粘贴到 GitHub 的 release note 中 |
||||
|
||||
### 投票阶段 |
||||
|
||||
1. DolphinScheduler社区投票，发起投票邮件到`dev@dolphinscheduler.apache.org`。PMC需要先按照文档检查版本的正确性，然后再进行投票。 |
||||
经过至少72小时并统计到至少3个`+1 并且没有-1 PMC member`票后，即可进入下一阶段。 |
||||
|
||||
2. 宣布投票结果,发起投票结果邮件到`dev@dolphinscheduler.apache.org`。 |
||||
|
||||
### 投票模板 |
||||
|
||||
1. DolphinScheduler社区投票模板 |
||||
|
||||
标题： |
||||
|
||||
```txt |
||||
[VOTE] Release Apache DolphinScheduler <VERSION> |
||||
``` |
||||
|
||||
正文： |
||||
|
||||
```txt |
||||
Hello DolphinScheduler Community, |
||||
|
||||
This is a call for vote to release Apache DolphinScheduler version <VERSION> |
||||
|
||||
Release notes: https://github.com/apache/dolphinscheduler/releases/tag/<VERSION> |
||||
|
||||
The release candidates: https://dist.apache.org/repos/dist/dev/dolphinscheduler/<VERSION>/ |
||||
|
||||
Maven 2 staging repository: https://repository.apache.org/content/repositories/<VERSION>/org/apache/dolphinscheduler/ |
||||
|
||||
Git tag for the release: https://github.com/apache/dolphinscheduler/tree/<VERSION> |
||||
|
||||
Release Commit ID: https://github.com/apache/dolphinscheduler/commit/<SHA-VALUE> |
||||
|
||||
Keys to verify the Release Candidate: https://dist.apache.org/repos/dist/dev/dolphinscheduler/KEYS |
||||
|
||||
Look at here for how to verify this release candidate: https://dolphinscheduler.apache.org/en-us/community/release.html |
||||
|
||||
The vote will be open for at least 72 hours or until necessary number of votes are reached. |
||||
|
||||
Please vote accordingly: |
||||
|
||||
[ ] +1 approve |
||||
[ ] +0 no opinion |
||||
[ ] -1 disapprove with the reason |
||||
|
||||
Checklist for reference: |
||||
|
||||
[ ] Download links are valid. |
||||
[ ] Checksums and PGP signatures are valid. |
||||
[ ] Source code artifacts have correct names matching the current release. |
||||
[ ] LICENSE and NOTICE files are correct for each DolphinScheduler repo. |
||||
[ ] All files have license headers if necessary. |
||||
[ ] No compiled archives bundled in source archive. |
||||
``` |
||||
|
||||
2.宣布投票结果模板 |
||||
|
||||
正文： |
||||
|
||||
```txt |
||||
The vote to release Apache DolphinScheduler <VERSION> has passed.Here is the vote result, |
||||
|
||||
4 PMC member +1 votes: |
||||
|
||||
xxx |
||||
xxx |
||||
xxx |
||||
xxx |
||||
|
||||
1 community +1 vote: |
||||
xxx |
||||
|
||||
Thanks everyone for taking time to check this release and help us. |
||||
``` |
||||
|
||||
|
||||
## 完成发布 |
||||
|
||||
### 将源码和二进制包从svn的dev目录移动到release目录 |
||||
|
||||
```shell |
||||
svn mv https://dist.apache.org/repos/dist/dev/dolphinscheduler/"${VERSION}" https://dist.apache.org/repos/dist/release/dolphinscheduler/ |
||||
``` |
||||
|
||||
### 将 gpg KEYS svn的dev目录移动到release目录 |
||||
|
||||
只有你第一次使用该 KEY 发版时才需要，如果之前已经发过版且 KEY 没有变化则不需要 |
||||
|
||||
```shell |
||||
mkdir -p ~/ds_svn/release/ |
||||
cd ~/ds_svn/release/ |
||||
svn --username="${A_USERNAME}" co https://dist.apache.org/repos/dist/release/dolphinscheduler |
||||
gpg -a --export <YOUR-GPG-KEY-ID> >> KEYS |
||||
svn add * |
||||
svn --username="${A_USERNAME}" commit -m "new key <YOUR-GPG-KEY-ID> add" |
||||
``` |
||||
|
||||
### 更新文档 |
||||
|
||||
官网应该在您发送通知邮件之前完成更新，本节将告诉您如何更改网站。假设发版的版本是 `<VERSION>`，需要进行以下更新（注意，当修改pull requests 被 merge 后就会生效）: |
||||
|
||||
- **apache/dolphinscheduler-website** 仓库： |
||||
- `download/en-us/download.md` 和 `download/zh-cn/download.md`: 增加 `<VERSION>` 版本发布包的下载 |
||||
- `scripts/conf.sh`: 在变量 `DEV_RELEASE_DOCS_VERSIONS` 中增加版本为 `<VERSION>` 的新键值对 |
||||
- **apache/dolphinscheduler** 仓库： |
||||
- `docs/configs/site.js`: |
||||
- `docsLatest`: 更新为 `<VERSION>` |
||||
- `docs0`: 两处 `en-us/zh-cn` 的 `text` 更新为 `latest(<VERSION>)` |
||||
- `docsxyz`: 两处 `en-us/zh-cn` 的 `children` 增加 `key` 为 `docsxyz`, `text` 为 `<VERSION>` 的下拉菜单 |
||||
- `docs/configs/index.md.jsx`: 增加 `'<VERSION>': docsxyzConfig,` |
||||
- `docs/docs/en/history-versions.md` 和 `docs/docs/zh/history-versions.md`: 增加新的发版版本 `<VERSION>` 的链接 |
||||
- `.github/ISSUE_TEMPLATE/bug-report.yml`: DolphinScheduler 在 GitHub issue 中有版本选择的部分，当有新版本发版后，需要更新这部分的内容。目前与版本关联的是 |
||||
[bug-report](https://github.com/apache/dolphinscheduler/blob/dev/.github/ISSUE_TEMPLATE/bug-report.yml)，发版的时候需要 |
||||
向其中的 **Version** 部分增加内容。 |
||||
|
||||
### 在 [apache staging repositories](https://repository.apache.org/#stagingRepositories) 仓库找到 DolphinScheduler 并点击`Release` |
||||
|
||||
### 发送公告邮件通知社区 |
||||
|
||||
当完成了上述的发版流程后，需要发送一封公告邮件给社区。你需要将邮件发送到 `dev@dolphinscheduler.apache.org` 并抄送到 `announce@apache.org`。 |
||||
|
||||
通知邮件模板如下： |
||||
|
||||
标题： |
||||
|
||||
```txt |
||||
[ANNOUNCE] Release Apache DolphinScheduler <VERSION> |
||||
``` |
||||
|
||||
正文： |
||||
|
||||
```txt |
||||
Hi all, |
||||
|
||||
We are glad to announce the release of Apache DolphinScheduler <VERSION>. Once again I would like to express my thanks to your help. |
||||
|
||||
Dolphin Scheduler is a distributed and easy-to-extend visual workflow scheduler system, |
||||
dedicated to solving the complex task dependencies in data processing, making the scheduler system out of the box for data processing. |
||||
|
||||
|
||||
Download Links: https://dolphinscheduler.apache.org/en-us/download/download.html |
||||
|
||||
Release Notes: https://github.com/apache/dolphinscheduler/releases/tag/<VERSION> |
||||
|
||||
Website: https://dolphinscheduler.apache.org/ |
||||
|
||||
DolphinScheduler Resources: |
||||
- Issue: https://github.com/apache/dolphinscheduler/issues/ |
||||
- Mailing list: dev@dolphinscheduler.apache.org |
||||
- Documents: https://dolphinscheduler.apache.org/zh-cn/docs/<VERSION>/user_doc/about/introduction.html |
||||
``` |
||||
|
After Width: | Height: | Size: 191 KiB |
|
After Width: | Height: | Size: 251 KiB |
|
After Width: | Height: | Size: 125 KiB |
|
After Width: | Height: | Size: 388 KiB |
Loading…
Reference in new issue