* [fix] Enhance name pre checker in resource center
Add file name and directory checker to avoid directory traversal
* add some missing change and change docs
* change var name in directoryTraversal
* Fix ci
* [common] Make dolphinscheduler_env.sh work
* Change dist tarball `dolphinscheduler_env.sh` location
from `bin/` to `conf/`, which users could finish their
change configuration operation in one single directory.
and we only need to add `$DOLPHINSCHEDULER_HOME/conf`
when we start our sever instead of adding both
`$DOLPHINSCHEDULER_HOME/conf` and `$DOLPHINSCHEDULER_HOME/bin`
* Change the `start.sh`'s path of `dolphinscheduler_env.sh`
* Change the setting order of `dolphinscheduler_env.sh`
* `bin/env/dolphinscheduler_env.sh` will overwrite the `<server>/conf/dolphinscheduler_env.sh`
when start the server using `bin/dolphinsceduler_daemon.sh` or `bin/install.sh`
* Change the related docs
* The resource download method is selected according to the configuration and the service startup verification is added.
* common check CI fix
* Startup check changed to running check
* code smell
* Coordinate resources to increase test coverage.
* Split resource download method.
* Unit Test Coverage
Co-authored-by: WangJPLeo <wangjipeng@whaleops.com>
* feat(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.extend the type of storage
3.add the s3utils
to manager resource
4.automatic inject the storage in addition to your
config
* fix(resource manager): update the dependency
* fix(resource manager): extend s3 to the storage of ds
fix the constant of hadooputils
* fix(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.delete the import *
* fix(resource manager):
merge the unitTest:
1.TenantServiceImpl
2.ResourceServiceImpl
3.UserServiceImpl
* fix(resource manager): extend s3 to the storage of ds
merge the resourceServiceTest
* fix(resource manager): test cancel the test method
createTenant verifyTenant
* fix(resource manager): merge the code follow the check-result of sonar
* fix(resource manager): extend s3 to the storage of ds
fit the spell question
* fix(resource manager): extend s3 to the storage of ds
revert the common.properties
* fix(resource manager): extend s3 to the storage of ds
update the storageConfig with None
* fix(resource manager): extend s3 to the storage of ds
fix the judge of resourceType
* fix(resource manager): extend s3 to the storage of ds
undo the compile-mysql
* fix(resource manager): extend s3 to the storage of ds
delete hadoop aws
* fix(resource manager): extend s3 to the storage of ds
update the know-dependencies to delete aws 1.7.4
update the e2e
file-manager common.properties
* fix(resource manager): extend s3 to the storage of ds
update the aws-region
* fix(resource manager): extend s3 to the storage of ds
fix the storageconfig init
* fix(resource manager): update e2e docker-compose
update e2e docker-compose
* fix(resource manager): extend s3 to the storage of ds
revent the e2e common.proprites
print the resource type in propertyUtil
* fix(resource manager): extend s3 to the storage of ds
1.println the properties
* fix(resource manager): println the s3 info
* fix(resource manager): extend s3 to the storage of ds
delete the info and upgrade the s3 info to e2e
* fix(resource manager): extend s3 to the storage of ds
add the bucket init
* fix(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.delete the import *
* fix(resource manager): extend s3 to the storage of ds
upgrade the s3 endpoint
* fix(resource manager): withPathStyleAccessEnabled(true)
* fix(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.delete the import *
* fix(resource manager): upgrade the s3client builder
* fix(resource manager): correct the s3 point to s3client
* fix(resource manager): update the constant BUCKET_NAME
* fix(resource manager): e2e s3 endpoint -> s3:9000
* fix(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.delete the import *
* style(resource manager): add info to createBucket
* style(resource manager): debug the log
* ci(resource manager): test
test s3
* ci(ci): add INSERT INTO dolphinscheduler.t_ds_tenant (id, tenant_code, description, queue_id, create_time, update_time) VALUES(1, 'root', NULL, 1, NULL, NULL); to h2.sql
* fix(resource manager): update the h2 sql
* fix(resource manager): solve to delete the tenant
* style(resource manager): merge the style end delete the unuse s3 config
* fix(resource manager): extend s3 to the storage of ds
UPDATE the rename resources when s3
* fix(resource manager): extend s3 to the storage of ds
1.fix the code style of QuartzImpl
* fix(resource manager): extend s3 to the storage of ds
1.impoort restore_type to CommonUtils
* fix(resource manager): update the work thread
* fix(resource manager): update the baseTaskProcessor
* fix(resource manager): upgrade dolphinscheduler-standalone-server.xml
* fix(resource manager): add user Info to dolphinscheduler_h2.sql
* fix(resource manager): merge the resourceType to NONE
* style(upgrade the log level to info):
* fix(resource manager): sysnc the h2.sql
* fix(resource manager): update the merge the user tenant
* fix(resource manager): merge the resourcesServiceImpl
* fix(resource manager):
when the storage is s3 ,that the directory can't be renamed
* fix(resource manager): in s3 ,the directory cannot be renamed
* fix(resource manager): delete the deleteRenameDirectory in E2E
* fix(resource manager): check the style and recoverd the test
* fix(resource manager): delete the log.print(LoginUser)
* date convert of timezone
* remove @JsonFormat
* add unit test
* fix time preview in scheduler
* optimization & add env config
Co-authored-by: caishunfeng <534328519@qq.com>
* add mock getTaskLogName() for UT test pass
* make task-tis rename to task-pigeon
* rename module name 'tis' to pigeon
* rename module name 'tis' to pigeon,make sanity check pass
* remove execute console.log()
* Enhance using experience of DataX by introduce TIS [Feature-5992]
* Enhance using experience of DataX by introduce TIS [Feature-5992]
* fix stylecheck error
* make testCase pass
* make dev testCase pass
* add new java dependency Java-WebSocket
* modfiy TISParameters.java avoid to name confliction
* add InterruptedException checking
* reAdd async-http-client annotation
* in order to staisfy the coverage degree add test which has been removed
* make testCase pass
* make testCase pass
* add jacoco dependency
* make code duplications be more lower
* add Java-WebSocket dependency in root pom
* remove useless code comment
* make tis http apply path and post body configurable ,the params save in config.properites
* Remove the dangerous instance of double-checked locking
* [BUG-#5678][Registry]fix registry init node miss (#5686)
* [Improvement][UI] Update the update time after the user information is successfully modified (#5684)
* improve
edit the userinfo success, but the updatetime is not the latest.
* Improved shell task execution result log information, adding process.waitFor() and process.exitValue() information to the original log (#5691)
Co-authored-by: shenglm <shenglm840722@126.com>
* [Feature-#5565][Master Worker-Server] Global Param passed by sense dependencies (#5603)
* add globalParams new plan with varPool
* add unit test
* add python task varPoolParams
Co-authored-by: wangxj <wangxj31>
* Issue robot translation judgment changed to Chinese (#5694)
Co-authored-by: chenxingchun <438044805@qq.com>
* the update function should use post instead of get (#5703)
* enhance form verify (#5696)
* checkState only supports %s not {} (#5711)
* [Fix-5701]When deleting a user, the accessToken associated with the user should also be deleted (#5697)
* update
* fix the codestyle error
* fix the compile error
* support rollback
* [Fix-5699][UI] Fix update user error in user information (#5700)
* [Improvement] the automatically generated spi service name in alert-plugin is wrong (#5676)
* bug fix
the auto generated spi service can't be recongized
* include a new method
* [Improvement-5622][project management] Modify the title (#5723)
* [Fix-5714] When updating the existing alarm instance, the creation time should't be updated (#5715)
* add a new init method.
* [Fix#5758] There are some problems in the api documentation that need to be improved (#5759)
* add the necessary parameters
* openapi improve
* fix code style error
* [FIX-#5721][master-server] Global params parameter missing (#5757)
Co-authored-by: wangxj <wangxj31>
* [Fix-5738][UI] The cancel button in the pop-up dialog of `batch copy` and `batch move` doesn't work. (#5739)
* Update relatedItems.vue
* Update relatedItems.vue
* [Improvement#5741][Worker] Improve task process status log (#5776)
* [Improvement-5773][server] need to support two parameters related to task (#5774)
* add some new parameter for task
* restore official properties
* improve imports
* modify a variable's name
Co-authored-by: jiang hua <jiang.hua@zhaopin.com.cn>
* [FIX-5786][Improvement][Server] When the Worker turns down, the MasterServer cannot handle the Remove event correctly and throws NPE
* [Improvement][Worker] Task log may be lost #5775 (#5783)
* [Imporvement #5725][CheckStyle] upgrade checkstyle file (#5789)
* [Imporvement #5725][CheckStyle] upgrade checkstyle file
Upgrade checkstyle.xml to support checkstyle version 8.24+
* change ci checkstyle version
* [Fix-5795][Improvement][Server] The starttime field in the HttpTask log is not displayed as expected. (#5796)
* improve timestamp format
make the startime in the log of httptask to be easier to read.
* fix bad code smell and update the note.
* [Imporvement #5621][job instance] start-time and end-time (#5621) (#5797)
·the list of workflow instances is sorted by start time and end time
·This closes#5621
* fix (#5803)
Co-authored-by: shuangbofu <fusb@tuya.com>
* fix: Remove duplicate "registryClient.close" method calls (#5805)
Co-authored-by: wen-hemin <wenhemin@apache.com>
* [Improvement][SPI] support load single plugin (#5794)
change load operation of 'registry.plugin.dir'
* [Improvement][Api Module] refactor registry client, remove spring annotation (#5814)
* fix: refactor registry client, remove spring annotation
* fix UT
* fix UT
* fix checkstyle
* fix UT
* fix UT
* fix UT
* fix: Rename RegistryCenterUtils method name
Co-authored-by: wen-hemin <wenhemin@apache.com>
* [Fix-5699][UI] Fix update user error in user information introduced by #5700 (#5735)
* [Fix-5726] When we used the UI page, we found some problems such as parameter validation, parameter update shows success but actually work (#5727)
* enhance the validation in UI
* enchance form verifaction
* simplify disable condition
* fix: Remove unused class (#5833)
Co-authored-by: wen-hemin <wenhemin@apache.com>
* [fix-5737] [Bug][Datasource] datsource other param check error (#5835)
Co-authored-by: wanggang <wanggy01@servyou.com.cn>
* [Fix-5719][K8s] Fix Ingress tls: got map expected array On TLS enabled On Kubernetes
[Fix-5719][K8s] Fix Ingress tls: got map expected array On TLS enabled On Kubernetes
* [Fix-5825][BUG][WEB] the resource tree in the process definition of latest dev branch can't display correctly (#5826)
* resoures-shows-error
* fix codestyle error
* add license header for new js
* fix codesmell
* [Improvement-5852][server] Support two parameters related to task for the rest of type of tasks. (#5867)
* provide two system parameters to support the rest of type of tasks
* provide two system parameters to support the rest of type of tasks
* improve test conversion
* [Improvement][Fix-5769][UI]When we try to delete the existing dag, the console in web browser would shows exception (#5770)
* fix bug
* cache the this variable
* Avoid self name
* fix code style compile error
* [Fix-5781][UT] Fix test coverage in sonar (#5817)
* build(UT): make jacoco running in offline-instrumentation
issue: #5781
* build(UT): remove the jacoco agent dependency in microbench
issue: #5781
* [Fix-5808][Server] When we try to transfer data using datax between different types of data sources, the worker will exit with ClassCastException (#5809)
* bug fix
* fix bug
* simplify the code format
* add a new parameter to make it easier to understand.
* [Fix-5830][Improvement][UI] Improve the selection style in dag edit dialog (#5829)
* improve the selection style
* update another file
* remove unnecessary css part.
* [Fix-5904][upgrade]fix dev branch upgrade mysql sql script error (#5821)
* fix dev branch upgrade mysql sql script error.
* Update naming convention.
* [Improvement][Api Module] refactor DataSourceParam and DependentParam, remove spring annotation (#5832)
* fix: refactor api utils class, remove spring annotation.
* fix: Optimization comments
Co-authored-by: wen-hemin <wenhemin@apache.com>
* correct the wrong annotion from zk queue implemented to java priority blocking queue (#5906)
Co-authored-by: ywang46 <ywang46@paypal.com>
Co-authored-by: Kirs <acm_master@163.com>
Co-authored-by: kyoty <echohlne@gmail.com>
Co-authored-by: ji04xiaogang <ji04xiaogang@163.com>
Co-authored-by: shenglm <shenglm840722@126.com>
Co-authored-by: wangxj3 <857234426@qq.com>
Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com>
Co-authored-by: chenxingchun <438044805@qq.com>
Co-authored-by: Shiwen Cheng <chengshiwen0103@gmail.com>
Co-authored-by: Jianchao Wang <akingchao@qq.com>
Co-authored-by: Tanvi Moharir <74228962+tanvimoharir@users.noreply.github.com>
Co-authored-by: Hua Jiang <jianghuachinacom@163.com>
Co-authored-by: jiang hua <jiang.hua@zhaopin.com.cn>
Co-authored-by: Wenjun Ruan <861923274@qq.com>
Co-authored-by: Tandoy <56899730+Tandoy@users.noreply.github.com>
Co-authored-by: 傅双波 <786183073@qq.com>
Co-authored-by: shuangbofu <fusb@tuya.com>
Co-authored-by: wen-hemin <39549317+wen-hemin@users.noreply.github.com>
Co-authored-by: wen-hemin <wenhemin@apache.com>
Co-authored-by: geosmart <geosmart@hotmail.com>
Co-authored-by: wanggang <wanggy01@servyou.com.cn>
Co-authored-by: AzureCN <colorazure@163.com>
Co-authored-by: 深刻 <tsund@qq.com>
Co-authored-by: zhuangchong <37063904+zhuangchong@users.noreply.github.com>
Co-authored-by: Yao WANG <Yao.MR.CN@gmail.com>
Co-authored-by: ywang46 <ywang46@paypal.com>
Co-authored-by: JinyLeeChina <297062848@qq.com>
* [BUG-#5678][Registry]fix registry init node miss (#5686)
* [Improvement][UI] Update the update time after the user information is successfully modified (#5684)
* improve
edit the userinfo success, but the updatetime is not the latest.
* Improved shell task execution result log information, adding process.waitFor() and process.exitValue() information to the original log (#5691)
Co-authored-by: shenglm <shenglm840722@126.com>
* [Feature-#5565][Master Worker-Server] Global Param passed by sense dependencies (#5603)
* add globalParams new plan with varPool
* add unit test
* add python task varPoolParams
Co-authored-by: wangxj <wangxj31>
* Issue robot translation judgment changed to Chinese (#5694)
Co-authored-by: chenxingchun <438044805@qq.com>
* the update function should use post instead of get (#5703)
* enhance form verify (#5696)
* checkState only supports %s not {} (#5711)
* [Fix-5701]When deleting a user, the accessToken associated with the user should also be deleted (#5697)
* update
* fix the codestyle error
* fix the compile error
* support rollback
* [Fix-5699][UI] Fix update user error in user information (#5700)
* [Improvement] the automatically generated spi service name in alert-plugin is wrong (#5676)
* bug fix
the auto generated spi service can't be recongized
* include a new method
* [Improvement-5622][project management] Modify the title (#5723)
* [Fix-5714] When updating the existing alarm instance, the creation time should't be updated (#5715)
* add a new init method.
* [Fix#5758] There are some problems in the api documentation that need to be improved (#5759)
* add the necessary parameters
* openapi improve
* fix code style error
* [FIX-#5721][master-server] Global params parameter missing (#5757)
Co-authored-by: wangxj <wangxj31>
* [Fix-5738][UI] The cancel button in the pop-up dialog of `batch copy` and `batch move` doesn't work. (#5739)
* Update relatedItems.vue
* Update relatedItems.vue
* [Improvement#5741][Worker] Improve task process status log (#5776)
* [Improvement-5773][server] need to support two parameters related to task (#5774)
* add some new parameter for task
* restore official properties
* improve imports
* modify a variable's name
Co-authored-by: jiang hua <jiang.hua@zhaopin.com.cn>
* [FIX-5786][Improvement][Server] When the Worker turns down, the MasterServer cannot handle the Remove event correctly and throws NPE
* [Improvement][Worker] Task log may be lost #5775 (#5783)
* [Imporvement #5725][CheckStyle] upgrade checkstyle file (#5789)
* [Imporvement #5725][CheckStyle] upgrade checkstyle file
Upgrade checkstyle.xml to support checkstyle version 8.24+
* change ci checkstyle version
* [Fix-5795][Improvement][Server] The starttime field in the HttpTask log is not displayed as expected. (#5796)
* improve timestamp format
make the startime in the log of httptask to be easier to read.
* fix bad code smell and update the note.
* [Imporvement #5621][job instance] start-time and end-time (#5621) (#5797)
·the list of workflow instances is sorted by start time and end time
·This closes#5621
* fix (#5803)
Co-authored-by: shuangbofu <fusb@tuya.com>
* fix: Remove duplicate "registryClient.close" method calls (#5805)
Co-authored-by: wen-hemin <wenhemin@apache.com>
* [Improvement][SPI] support load single plugin (#5794)
change load operation of 'registry.plugin.dir'
* [Improvement][Api Module] refactor registry client, remove spring annotation (#5814)
* fix: refactor registry client, remove spring annotation
* fix UT
* fix UT
* fix checkstyle
* fix UT
* fix UT
* fix UT
* fix: Rename RegistryCenterUtils method name
Co-authored-by: wen-hemin <wenhemin@apache.com>
* [Fix-5699][UI] Fix update user error in user information introduced by #5700 (#5735)
* [Fix-5726] When we used the UI page, we found some problems such as parameter validation, parameter update shows success but actually work (#5727)
* enhance the validation in UI
* enchance form verifaction
* simplify disable condition
* fix conflicts
Co-authored-by: Kirs <acm_master@163.com>
Co-authored-by: kyoty <echohlne@gmail.com>
Co-authored-by: ji04xiaogang <ji04xiaogang@163.com>
Co-authored-by: shenglm <shenglm840722@126.com>
Co-authored-by: wangxj3 <857234426@qq.com>
Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com>
Co-authored-by: chenxingchun <438044805@qq.com>
Co-authored-by: Shiwen Cheng <chengshiwen0103@gmail.com>
Co-authored-by: Jianchao Wang <akingchao@qq.com>
Co-authored-by: Tanvi Moharir <74228962+tanvimoharir@users.noreply.github.com>
Co-authored-by: Hua Jiang <jianghuachinacom@163.com>
Co-authored-by: jiang hua <jiang.hua@zhaopin.com.cn>
Co-authored-by: Wenjun Ruan <861923274@qq.com>
Co-authored-by: Tandoy <56899730+Tandoy@users.noreply.github.com>
Co-authored-by: 傅双波 <786183073@qq.com>
Co-authored-by: shuangbofu <fusb@tuya.com>
Co-authored-by: wen-hemin <39549317+wen-hemin@users.noreply.github.com>
Co-authored-by: wen-hemin <wenhemin@apache.com>
Co-authored-by: JinyLeeChina <297062848@qq.com>
* [BUG-#5678][Registry]fix registry init node miss (#5686)
* [Improvement][UI] Update the update time after the user information is successfully modified (#5684)
* improve
edit the userinfo success, but the updatetime is not the latest.
* Improved shell task execution result log information, adding process.waitFor() and process.exitValue() information to the original log (#5691)
Co-authored-by: shenglm <shenglm840722@126.com>
* [Feature-#5565][Master Worker-Server] Global Param passed by sense dependencies (#5603)
* add globalParams new plan with varPool
* add unit test
* add python task varPoolParams
Co-authored-by: wangxj <wangxj31>
* Issue robot translation judgment changed to Chinese (#5694)
Co-authored-by: chenxingchun <438044805@qq.com>
* the update function should use post instead of get (#5703)
* enhance form verify (#5696)
* checkState only supports %s not {} (#5711)
* [Fix-5701]When deleting a user, the accessToken associated with the user should also be deleted (#5697)
* update
* fix the codestyle error
* fix the compile error
* support rollback
Co-authored-by: Kirs <acm_master@163.com>
Co-authored-by: kyoty <echohlne@gmail.com>
Co-authored-by: ji04xiaogang <ji04xiaogang@163.com>
Co-authored-by: shenglm <shenglm840722@126.com>
Co-authored-by: wangxj3 <857234426@qq.com>
Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com>
Co-authored-by: chenxingchun <438044805@qq.com>
Co-authored-by: JinyLeeChina <297062848@qq.com>
[Feature#3961]Registry SPI
All the logical structure of the registry must be converted into a tree structure within the system, so some plug-ins must be converted internally, such as ETCD
The registry supports distributed locks.
todo: The specific information about the registration center of the API module needs to be adjusted.
[Feature#3961]Registry SPI
All the logical structure of the registry must be converted into a tree structure within the system, so some plug-ins must be converted internally, such as ETCD
The registry supports distributed locks.
todo: The specific information about the registration center of the API module needs to be adjusted.
* [Improvement][Docker] Support more configurations and improve image for python
* [Improvement][K8s] Support more configurations, more service accesses, and reduce the duplication configurations
* [Improvement][Python] Improve the compatibility of python home
* [Improvement][Install] Fix install config and install.sh
* [Improvement][Install] Fix workflow ut
* [Improvement][Docker] Optimize docker-swarm/check script
* [Improvement][DB] Update default username and password of database
* Update comments in master/worker.properties
* Specify version for all images
* [Improvement][Docker] Optimize PS1 and WORKDIR
* [Improvement][SQL] Reduce heap size to 64m in create-dolphinscheduler.sh and upgrade-dolphinscheduler.sh
* [Fix-5431][K8s] Fix master and worker cannot get the right address with custom DNS
* Create SparkparametersTest.java
Add a test case
* Update and rename SparkparametersTest.java to SparkParametersTest.java
* Update pom.xml
add SparkParametersTest.java in pom file
* reduce the number of data sources created twice by master server.
* update ProcessAlertManagerTest test class to increase coverage.
* add DolphinSchedulerPluginLoaderTest test ignore.
Co-authored-by: zhuang chong
Jiajie Zhong
caishunfeng
gaojun2048
WangJPLeo
nobolity
Kerwin
Wenjun Ruan
ronyang1985
zixi0825
springmonster
kezhenxu94
JinYong Li
mask
aCodingAddict
wind
calvinit
百岁
HanayoZz
geosmart
kyoty
wangxj3
Squidyu
Kirs
Shiwen Cheng
QuakeWang
ruanwenjun