Currently the size of our distribute package is up to
800MB, this patch is migrate python gateway server into
api server
The distribute package size before and after this patch is:
```sh
# before
796M apache-dolphinscheduler-2.0.4-SNAPSHOT-bin.tar.gz
# after
647M apache-dolphinscheduler-2.0.4-SNAPSHOT-bin.tar.gz
```
* feat(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.extend the type of storage
3.add the s3utils
to manager resource
4.automatic inject the storage in addition to your
config
* fix(resource manager): update the dependency
* fix(resource manager): extend s3 to the storage of ds
fix the constant of hadooputils
* fix(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.delete the import *
* fix(resource manager):
merge the unitTest:
1.TenantServiceImpl
2.ResourceServiceImpl
3.UserServiceImpl
* fix(resource manager): extend s3 to the storage of ds
merge the resourceServiceTest
* fix(resource manager): test cancel the test method
createTenant verifyTenant
* fix(resource manager): merge the code follow the check-result of sonar
* fix(resource manager): extend s3 to the storage of ds
fit the spell question
* fix(resource manager): extend s3 to the storage of ds
revert the common.properties
* fix(resource manager): extend s3 to the storage of ds
update the storageConfig with None
* fix(resource manager): extend s3 to the storage of ds
fix the judge of resourceType
* fix(resource manager): extend s3 to the storage of ds
undo the compile-mysql
* fix(resource manager): extend s3 to the storage of ds
delete hadoop aws
* fix(resource manager): extend s3 to the storage of ds
update the know-dependencies to delete aws 1.7.4
update the e2e
file-manager common.properties
* fix(resource manager): extend s3 to the storage of ds
update the aws-region
* fix(resource manager): extend s3 to the storage of ds
fix the storageconfig init
* fix(resource manager): update e2e docker-compose
update e2e docker-compose
* fix(resource manager): extend s3 to the storage of ds
revent the e2e common.proprites
print the resource type in propertyUtil
* fix(resource manager): extend s3 to the storage of ds
1.println the properties
* fix(resource manager): println the s3 info
* fix(resource manager): extend s3 to the storage of ds
delete the info and upgrade the s3 info to e2e
* fix(resource manager): extend s3 to the storage of ds
add the bucket init
* fix(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.delete the import *
* fix(resource manager): extend s3 to the storage of ds
upgrade the s3 endpoint
* fix(resource manager): withPathStyleAccessEnabled(true)
* fix(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.delete the import *
* fix(resource manager): upgrade the s3client builder
* fix(resource manager): correct the s3 point to s3client
* fix(resource manager): update the constant BUCKET_NAME
* fix(resource manager): e2e s3 endpoint -> s3:9000
* fix(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.delete the import *
* style(resource manager): add info to createBucket
* style(resource manager): debug the log
* ci(resource manager): test
test s3
* ci(ci): add INSERT INTO dolphinscheduler.t_ds_tenant (id, tenant_code, description, queue_id, create_time, update_time) VALUES(1, 'root', NULL, 1, NULL, NULL); to h2.sql
* fix(resource manager): update the h2 sql
* fix(resource manager): solve to delete the tenant
* style(resource manager): merge the style end delete the unuse s3 config
* fix(resource manager): extend s3 to the storage of ds
UPDATE the rename resources when s3
* fix(resource manager): extend s3 to the storage of ds
1.fix the code style of QuartzImpl
* fix(resource manager): extend s3 to the storage of ds
1.impoort restore_type to CommonUtils
* fix(resource manager): update the work thread
* fix(resource manager): update the baseTaskProcessor
* fix(resource manager): upgrade dolphinscheduler-standalone-server.xml
* fix(resource manager): add user Info to dolphinscheduler_h2.sql
* fix(resource manager): merge the resourceType to NONE
* style(upgrade the log level to info):
* fix(resource manager): sysnc the h2.sql
* fix(resource manager): update the merge the user tenant
* fix(resource manager): merge the resourcesServiceImpl
* fix(resource manager):
when the storage is s3 ,that the directory can't be renamed
* fix(resource manager): in s3 ,the directory cannot be renamed
* fix(resource manager): delete the deleteRenameDirectory in E2E
* fix(resource manager): check the style and recoverd the test
* fix(resource manager): delete the log.print(LoginUser)
* Add new profile `python` to separate build python
package from `release`. Now someone who does not interested
in python api could build package without python environment
* Add auto gpg asc sign when run `mvn instal -Ppython`
* Add property to skip gpg sign `python.sign.skip`
* Set `python.sign.skip=false` sign python dist by default
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
* Add LICENSE and NOTICE to Python API
* Add new license dir `python-api-license` to keep all
Python API's LICENSE files
* Change LICENSE in distribute package
* Add content about should add LICENSE when new dependence
adding
This patch add python api package as independent
directory in final dist directory when user run
command `mvn -U install package -Prelease`.
There have `tar.gz` and `whl` in the python dist directory
```
<the-old-dist-package>
python
|--- python-independent-pkg.tar.gz
|--- python-independent-pkg.whl
```
close: #8343
* Give dedicated control to docker build/push goals
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
* [Improve]Use hibernate validation framework
Improve email verification method
Support domain name verification
* Email is at least a second-level domain name
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* add mock getTaskLogName() for UT test pass
* make task-tis rename to task-pigeon
* rename module name 'tis' to pigeon
* rename module name 'tis' to pigeon,make sanity check pass
* remove execute console.log()
* Enhance using experience of DataX by introduce TIS [Feature-5992]
* Enhance using experience of DataX by introduce TIS [Feature-5992]
* fix stylecheck error
* make testCase pass
* make dev testCase pass
* add new java dependency Java-WebSocket
* modfiy TISParameters.java avoid to name confliction
* add InterruptedException checking
* reAdd async-http-client annotation
* in order to staisfy the coverage degree add test which has been removed
* make testCase pass
* make testCase pass
* add jacoco dependency
* make code duplications be more lower
* add Java-WebSocket dependency in root pom
* remove useless code comment
* make tis http apply path and post body configurable ,the params save in config.properites
* Remove the dangerous instance of double-checked locking
* [BUG-#5678][Registry]fix registry init node miss (#5686)
* [Improvement][UI] Update the update time after the user information is successfully modified (#5684)
* improve
edit the userinfo success, but the updatetime is not the latest.
* Improved shell task execution result log information, adding process.waitFor() and process.exitValue() information to the original log (#5691)
Co-authored-by: shenglm <shenglm840722@126.com>
* [Feature-#5565][Master Worker-Server] Global Param passed by sense dependencies (#5603)
* add globalParams new plan with varPool
* add unit test
* add python task varPoolParams
Co-authored-by: wangxj <wangxj31>
* Issue robot translation judgment changed to Chinese (#5694)
Co-authored-by: chenxingchun <438044805@qq.com>
* the update function should use post instead of get (#5703)
* enhance form verify (#5696)
* checkState only supports %s not {} (#5711)
* [Fix-5701]When deleting a user, the accessToken associated with the user should also be deleted (#5697)
* update
* fix the codestyle error
* fix the compile error
* support rollback
* [Fix-5699][UI] Fix update user error in user information (#5700)
* [Improvement] the automatically generated spi service name in alert-plugin is wrong (#5676)
* bug fix
the auto generated spi service can't be recongized
* include a new method
* [Improvement-5622][project management] Modify the title (#5723)
* [Fix-5714] When updating the existing alarm instance, the creation time should't be updated (#5715)
* add a new init method.
* [Fix#5758] There are some problems in the api documentation that need to be improved (#5759)
* add the necessary parameters
* openapi improve
* fix code style error
* [FIX-#5721][master-server] Global params parameter missing (#5757)
Co-authored-by: wangxj <wangxj31>
* [Fix-5738][UI] The cancel button in the pop-up dialog of `batch copy` and `batch move` doesn't work. (#5739)
* Update relatedItems.vue
* Update relatedItems.vue
* [Improvement#5741][Worker] Improve task process status log (#5776)
* [Improvement-5773][server] need to support two parameters related to task (#5774)
* add some new parameter for task
* restore official properties
* improve imports
* modify a variable's name
Co-authored-by: jiang hua <jiang.hua@zhaopin.com.cn>
* [FIX-5786][Improvement][Server] When the Worker turns down, the MasterServer cannot handle the Remove event correctly and throws NPE
* [Improvement][Worker] Task log may be lost #5775 (#5783)
* [Imporvement #5725][CheckStyle] upgrade checkstyle file (#5789)
* [Imporvement #5725][CheckStyle] upgrade checkstyle file
Upgrade checkstyle.xml to support checkstyle version 8.24+
* change ci checkstyle version
* [Fix-5795][Improvement][Server] The starttime field in the HttpTask log is not displayed as expected. (#5796)
* improve timestamp format
make the startime in the log of httptask to be easier to read.
* fix bad code smell and update the note.
* [Imporvement #5621][job instance] start-time and end-time (#5621) (#5797)
·the list of workflow instances is sorted by start time and end time
·This closes#5621
* fix (#5803)
Co-authored-by: shuangbofu <fusb@tuya.com>
* fix: Remove duplicate "registryClient.close" method calls (#5805)
Co-authored-by: wen-hemin <wenhemin@apache.com>
* [Improvement][SPI] support load single plugin (#5794)
change load operation of 'registry.plugin.dir'
* [Improvement][Api Module] refactor registry client, remove spring annotation (#5814)
* fix: refactor registry client, remove spring annotation
* fix UT
* fix UT
* fix checkstyle
* fix UT
* fix UT
* fix UT
* fix: Rename RegistryCenterUtils method name
Co-authored-by: wen-hemin <wenhemin@apache.com>
* [Fix-5699][UI] Fix update user error in user information introduced by #5700 (#5735)
* [Fix-5726] When we used the UI page, we found some problems such as parameter validation, parameter update shows success but actually work (#5727)
* enhance the validation in UI
* enchance form verifaction
* simplify disable condition
* fix: Remove unused class (#5833)
Co-authored-by: wen-hemin <wenhemin@apache.com>
* [fix-5737] [Bug][Datasource] datsource other param check error (#5835)
Co-authored-by: wanggang <wanggy01@servyou.com.cn>
* [Fix-5719][K8s] Fix Ingress tls: got map expected array On TLS enabled On Kubernetes
[Fix-5719][K8s] Fix Ingress tls: got map expected array On TLS enabled On Kubernetes
* [Fix-5825][BUG][WEB] the resource tree in the process definition of latest dev branch can't display correctly (#5826)
* resoures-shows-error
* fix codestyle error
* add license header for new js
* fix codesmell
* [Improvement-5852][server] Support two parameters related to task for the rest of type of tasks. (#5867)
* provide two system parameters to support the rest of type of tasks
* provide two system parameters to support the rest of type of tasks
* improve test conversion
* [Improvement][Fix-5769][UI]When we try to delete the existing dag, the console in web browser would shows exception (#5770)
* fix bug
* cache the this variable
* Avoid self name
* fix code style compile error
* [Fix-5781][UT] Fix test coverage in sonar (#5817)
* build(UT): make jacoco running in offline-instrumentation
issue: #5781
* build(UT): remove the jacoco agent dependency in microbench
issue: #5781
* [Fix-5808][Server] When we try to transfer data using datax between different types of data sources, the worker will exit with ClassCastException (#5809)
* bug fix
* fix bug
* simplify the code format
* add a new parameter to make it easier to understand.
* [Fix-5830][Improvement][UI] Improve the selection style in dag edit dialog (#5829)
* improve the selection style
* update another file
* remove unnecessary css part.
* [Fix-5904][upgrade]fix dev branch upgrade mysql sql script error (#5821)
* fix dev branch upgrade mysql sql script error.
* Update naming convention.
* [Improvement][Api Module] refactor DataSourceParam and DependentParam, remove spring annotation (#5832)
* fix: refactor api utils class, remove spring annotation.
* fix: Optimization comments
Co-authored-by: wen-hemin <wenhemin@apache.com>
* correct the wrong annotion from zk queue implemented to java priority blocking queue (#5906)
Co-authored-by: ywang46 <ywang46@paypal.com>
* Add a Gitter chat badge to README.md (#5883)
* Add Gitter badge
* Update README.md
Co-authored-by: David <dailidong66@gmail.com>
* ci: improve maven connection in CI builds (#5924)
issue: #5921
* [Improvement][Master]fix typo (#5934)
·fix typo in MasterBaseTaskExecThread
* [Fix-5886][server] Enhanced scheduler delete check (#5936)
* Add:Name verification remove the first and last spaces.
* Update: wrong word: 'WAITTING' ->'WAITING'
* Add: Strengthen verification
Co-authored-by: Squid <2824638304@qq.com>
* [Improvement-5880][api] Optimized data structure of pagination query API results (#5895)
* [5880][refactor]Optimized data structure of pagination query API results
- refactor PageInfo and delete returnDataListPaging in API
- modify the related Controller and Service and the corresponding Test
* Merge branch 'dev' of github.com:apache/dolphinscheduler into dev
Conflicts:
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ProcessDefinitionServiceImpl.java
Co-authored-by: 蔡泽华 <sorea1k@163.com>
* [IMPROVEMENT]fix mysql comment error (#5959)
* [Improvement][Api]fix typo (#5960)
* [Imporvement #5621][job instance] start-time and end-time (#5621)
·the list of workflow instances is sorted by start time and end time
·This closes#5621
* [FIX-5975]queryLastRunningProcess sql in ProcessInstanceMapper.xml (#5980)
* [NEW FEATURE][FIX-4385] compensation task add the ability to configure parallelism (#5912)
* update
* web improved
* improve the ui
* add the ability to configure the parallelism
* update variables
* enhance the ut and add necessary note
* fix code style
* fix code style issue
* ensure the complation task in parallel mode can run the right numbers of tasks.
* [Improvement][dao]When I search for the keyword description, the web UI shows empty (#5952)
* [Bug][WorkerServer] SqlTask NullPointerException #5549
* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428
* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428
* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428
* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428
* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428
* [Improvement][dao]When I search for the keyword Modify User, the web UI shows empty #5428
* [Improvement][dao]When I search for the keyword description, the web UI shows empty #5428
* fix the readme typing issue (#5998)
* Fix unchecked type conversions
* Use indentation level reported by checkstyle
* Reorganize CI workflows to fasten the wasted time and resources (#6011)
* Add standalone server module to make it easier to develop (#6022)
* Task node of SWITCH (#5939)
* [Feature-#5273][server-master] Task node of SWITCH (#5922)
Co-authored-by: wangxj <wangxj31>
* remove description of bonecp (#6030)
Co-authored-by: shaojwu <shaojwu@ebay.com>
* [Improvement][Api Module]split alert group list-paging interface (#5941)
* [Improvement][Api Module]split alert group list-paging interface
* [FIX-#6007]Wrong complement date (#6026)
* [FIX-#6007]Wrong complement date
* [style]Wrong complement date
* [Improvement-6024][dist] Remove useless packaging commands (#6029)
·Remove useless packaging commands in dolphinscheduler-bin.xml
This closes#6024
Co-authored-by: mask <liuhu@zhiyoutec.com>
* [FIX-5908][MasterServer] When executing an compensation task, the execution thread would have a NPE (#5909)
* fix the npe in MasterExec
* fix the compile error
* Add `.asf.yaml` to easily set the GitHub metadata (#6035)
* fix dead server cannot stop (#6046)
* Enhancement Translation (#6042)
* replaced Loading... with i18n
* modified Edit zh_CN translation
* Delete zh_CN.js
Co-authored-by: David <dailidong66@gmail.com>
* fix bug #6053 zh_CN.js is lost
* [Fix-6038][ui] width of "SQL Statement" in Dag FormLineModal will be shrunk if sql line is too long (#6040)
This closes#6038
* [Improvement] Fix inefficient map iterator (#6004)
* Fix inefficient map iterator
* Use forEach and remove call to valueOf
* Modify AbstractParameters
* Enhance `StandaloneServer` so that we don't need to update the version number manually (#6074)
* Remove invalid character in `.asf.yaml` (#6075)
* Remove invalid character `\n` in `.asf.yaml` (#6077)
It turns out that the invalid character is `\n`
* Add alert server into standalone-server as well and some minor polish (#6087)
* Support starting standalone server in Docker image (#6102)
Also remove unused class
* [Feature-4355][Master-Worker-API] improvements of master and scheduler module (#6095)
* [Feature-4355][Master-Worker-API] improvements of master and scheduler module (#6085)
* master refactor:
1. spi for task submit and other actions(pause, kill)
2. remove threads for process instance and task instance.
3. add events for process instance and task instance
* ut npe
* add try catch
* code style
* fix critical bugs
* fix critical bugs
* fix critical bugs
* fix critical bugs
* Remove unused params in SwitchTaskTest (#6109)
* fix ut
* fix ut
* fix ut
* fix ut
* fix server module ut
* fix ut
Co-authored-by: Kirs <acm_master@163.com>
Co-authored-by: kyoty <echohlne@gmail.com>
Co-authored-by: ji04xiaogang <ji04xiaogang@163.com>
Co-authored-by: shenglm <shenglm840722@126.com>
Co-authored-by: wangxj3 <857234426@qq.com>
Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com>
Co-authored-by: chenxingchun <438044805@qq.com>
Co-authored-by: Shiwen Cheng <chengshiwen0103@gmail.com>
Co-authored-by: Jianchao Wang <akingchao@qq.com>
Co-authored-by: Tanvi Moharir <74228962+tanvimoharir@users.noreply.github.com>
Co-authored-by: Hua Jiang <jianghuachinacom@163.com>
Co-authored-by: jiang hua <jiang.hua@zhaopin.com.cn>
Co-authored-by: Wenjun Ruan <861923274@qq.com>
Co-authored-by: Tandoy <56899730+Tandoy@users.noreply.github.com>
Co-authored-by: 傅双波 <786183073@qq.com>
Co-authored-by: shuangbofu <fusb@tuya.com>
Co-authored-by: wen-hemin <39549317+wen-hemin@users.noreply.github.com>
Co-authored-by: wen-hemin <wenhemin@apache.com>
Co-authored-by: geosmart <geosmart@hotmail.com>
Co-authored-by: wanggang <wanggy01@servyou.com.cn>
Co-authored-by: AzureCN <colorazure@163.com>
Co-authored-by: 深刻 <tsund@qq.com>
Co-authored-by: zhuangchong <37063904+zhuangchong@users.noreply.github.com>
Co-authored-by: Yao WANG <Yao.MR.CN@gmail.com>
Co-authored-by: ywang46 <ywang46@paypal.com>
Co-authored-by: The Gitter Badger <badger@gitter.im>
Co-authored-by: David <dailidong66@gmail.com>
Co-authored-by: Squidyu <1297554122@qq.com>
Co-authored-by: Squid <2824638304@qq.com>
Co-authored-by: soreak <60459867+soreak@users.noreply.github.com>
Co-authored-by: 蔡泽华 <sorea1k@163.com>
Co-authored-by: yimaixinchen <yimaixinchen@163.com>
Co-authored-by: atai-555 <74188560+atai-555@users.noreply.github.com>
Co-authored-by: didiaode18 <563646039@qq.com>
Co-authored-by: Roy <yongjuncao1213@gmail.com>
Co-authored-by: lyxell <alyxell@kth.se>
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
Co-authored-by: myangle1120 <942542838@qq.com>
Co-authored-by: gabry.wu <gabrywu@apache.org>
Co-authored-by: shaojwu <shaojwu@ebay.com>
Co-authored-by: Shukun Zhang <60541766+andream7@users.noreply.github.com>
Co-authored-by: linquan <1175687813@qq.com>
Co-authored-by: mask <39329477+Narcasserun@users.noreply.github.com>
Co-authored-by: mask <liuhu@zhiyoutec.com>
Co-authored-by: RichardStark <49977764+RichardStark@users.noreply.github.com>
Co-authored-by: lenboo <baoliang.leon@gmail.com>
Co-authored-by: lilyzhou <lj_zhou@outlook.com>
Co-authored-by: OS <29528966+lenboo@users.noreply.github.com>
Co-authored-by: junfan.zhang <zuston.shacha@gmail.com>
Co-authored-by: JinyLeeChina <297062848@qq.com>
Kerwin
Jiajie Zhong
Eric Gao
songjianet
nobolity
kezhenxu94
BaoLiang
ronyang1985
Kirs
qianli2022
zixi0825
Kirs
wind
JinYong Li
KyoYang
mask
百岁