cf91544b99
Refactor logic task ( #13948 )
2 years ago
db6c061c8b
[Feature][plugin] Add databend datasource in datasource plugin ( #13866 )
...
Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
2 years ago
ec29220977
[improve-#13966] mask datax log ( #13969 )
...
Co-authored-by: fuchanghai <‘2875334588@qq.com’>
2 years ago
c0126b7f30
[Feature][plugin] Support Kyuubi datasource ( #13642 )
...
---------
Co-authored-by: weitong <weitong@cisco.com>
Co-authored-by: Rick Cheng <38122586+rickchengx@users.noreply.github.com>
2 years ago
61a689aa5a
Use percentage to represent memory/cpu usage ( #13896 )
2 years ago
7ee66f2d02
[Feature-13429][Remote Logging] Add support for writing task logs to Google Cloud Storage ( #13777 )
2 years ago
41a8ba9aab
Support alert server HA ( #13865 )
...
* Support alert server cluster
* Remove WORKER_ALERT_LISTEN_HOST in statefulset-dolphinscheduler-worker.yaml
2 years ago
69fc217ab2
Fix trim might cause NPE in PropertyUtils ( #13853 )
2 years ago
9df89cf0cf
Use bootstrap user as the default tenant ( #13825 )
2 years ago
85605bbef9
[Improvement-13653][Remote Logging] Support api-server to get task log from remote target ( #13654 )
2 years ago
1ac76e1cbb
[Improvement]update spelling mistake for Constants ( #13738 )
...
* [Improvement][Document]Update architecture and process-start-flow pic
* [Improvement]update spelling mistake for Constants
---------
Co-authored-by: xuhaihui <xuhaihui@cmss.chinamobile.com>
2 years ago
2dceb6b8ca
[Feature-13428][Resource Center] Add support for Google Cloud Storage in the resource center ( #13562 )
2 years ago
aa79c01ce9
[Feature-13419][Remote Logging] Add support for writing task logs to S3 ( #13649 )
...
* [Feature-13419][Remote Logging] Add support for writing task logs to S3
* Update dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/log/remote/S3RemoteLogHandler.java
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
* use try with resource
* remove unnecessary log
---------
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2 years ago
35d8e8ac17
[Feature][DataSource] add oceanbase datasource ( #13675 )
...
Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
2 years ago
7bf3e3cdd6
[improve-#13201] update pid during running ( #13206 )
2 years ago
047fa2f65e
[Feature-13511] Submit Spark task directly on Kubernetes ( #13550 )
2 years ago
701d67c831
[improve-#13045] after a submit failure, stop the processInstance to avoid an endless loop ( #13051 )
...
* [improve-#13045] add max submit number of workflow
* [fix-13405] remove max times
* Update dolphinscheduler-master/src/main/resources/application.yaml
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
---------
Co-authored-by: fuchanghai <‘2875334588@qq.com’>
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
2 years ago
2bd65fb2df
[Feature][Remote Logging] Add support for writing task logs to OSS ( #13332 )
2 years ago
e368bde7ee
Publish Helm Chart snapshot to ghcr.io ( #13546 )
2 years ago
8d12dc0702
[Improvement-13491] Use lombok @Slf4j annotation to generate logger ( #13509 )
2 years ago
b11431db95
Use single thread to refresh kerberos ( #13456 )
2 years ago
12dd60fa46
Fix task wake up failed will block the event ( #13466 )
2 years ago
78e5569afd
[feat] Add type local turn on resource center by default ( #13303 )
2 years ago
78ba71bdbd
[Feature][task]add azure datafactory task type ( #13417 )
...
* init data factory
* add task-type-config.yaml
* add data factory ui
* submit doc
* submit license
---------
Co-authored-by: devosend <devosend@gmail.com>
2 years ago
713fdbc444
Split the upgrade code ( #13436 )
...
1. Split the upgrade code into different version
2. Log the dml/ddl sql in origin friendly format
3. Fix ddl of 3.0.0 and 3.1.0
4. Fix search resource regular expression injection
2 years ago
8be32d4145
[Feature][Api] When use api to run a process we want get processInstanceId ( #13184 )
...
* add sql
* add mapper
* add dao
* add excutor
Co-authored-by: qianl4 <qianl4@cicso.com>
2 years ago
3a6e10397f
[feature][api-datasource]add redshift auth mode to get connection by access key ( #13379 )
...
* submit redshift auth mode
* add front end
* add doc and fix frontend
Co-authored-by: devosend <devosend@gmail.com>
2 years ago
6634ff98af
Add source env list in common.properties ( #13317 )
2 years ago
bcf03ad4d1
[Feature-9467] add DAMENG DataSource ( #12860 )
...
* [Feature-9467][DataSource] add DAMENG （达梦） data source
* [Feature-9467] add DM datasource doc (Issue #9467 )
* [Feature-9467] add DM IDataBaseOptionKeys
* [Feature-9467] add DM DataSource: DmDataSourceChannelFactoryTest remove testGetDmConnection method
* [Feature-9467] modification package dm to dameng;
* [Feature-9467] add Dameng DmJdbcDriver18 License
* [Feature-9467] add DM DataSource: remove DruidDataSourceClient
Co-authored-by: lenian <yuenian@mananacare.com>
2 years ago
ccad56e88e
[Improvement][Master] Validate same content of input file when using task cache ( #13298 )
...
* support file content checksum
* fix inject null storageOperate bug
2 years ago
fd3afd84ad
Remove alert when delete workflow instance ( #13281 )
2 years ago
14ec4a2398
Remove dao module in worker ( #13242 )
2 years ago
f0dd4f081f
[Feature][task-plugins-12820] azure azure-sql datasource plugins ( #13209 )
...
* submit azure sql code
* change mssql-jdbc version
* merge changes to delete other
* merge changes to delete other
* add dependencies
2 years ago
66e20271ad
[Feature][Master] Add task caching mechanism to improve the running speed of repetitive tasks ( #13194 )
...
* Supports task instance cache operation
* add task plugin cache
* use SHA-256 to generate key
* Update dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql
Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
* Update dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_postgresql.sql
Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
* Optimizing database Scripts
* Optimize clear cache operation
Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
2 years ago
e4b9b67255
Allow execute task in workflow instance ( #13103 )
2 years ago
70ccffeee2
Format task parameter as pretty json ( #13173 )
2 years ago
6ef74073cc
[Improve-13001]migrate commons-collections -> commons-collections4 ( #13002 )
2 years ago
1064680ded
[feat][API] New restful API for workflow state ( #13031 )
2 years ago
38b876733c
[Feature-10498] Mask the password in the log of sqoop task ( #11589 )
2 years ago
b40c63f02d
[Feature][DataSource]Add trino datasource plugin ( #12911 )
2 years ago
43d69ae157
Upgrade clickhouse jdbc driver ( #12639 )
2 years ago
ff59acd02f
Splite cpu/memory info in OSUtils#isOverload ( #12663 )
2 years ago
9e0c9af1a5
Fix the waiting strategy cannot recovery if the serverstate is already in running ( #12651 )
2 years ago
b71d4983dd
[Feature-6586][Server]add some ds process definition demo when init ( #11759 )
...
* [Feature-6586][Server]add some ds process definition demo when init
1.add some ds process definition demo when init, to display what task type can run and make user easy to
use ds.
2.need configure the JVM parameters (-Ddemo=true) to turn on the StandaloneServer service
3.need modify the tenant information in it
2 years ago
08335b1032
[Improvement][Task] Improved way to collect yarn job's appIds ( #12197 )
...
* Provide aop way as an optional way to collect yarn job's applicationId, and import new module `dolphinscheduler-aop` to place the aop code.
* Add user property `appId.collect` for user to decide how to collect applicationId.
* Add new environment configuration for each type of yarn tasks to support aop in `dolphinscheduler_env.sh`
* Update docs to declare how to use aop way.
* Update `LogUtils` to support fetch applicationId in different ways based on the user property.
Co-authored-by: gabrywu <gabrywu@apache.com>
2 years ago
a0d37fc7ae
Set tenantDir permission ( #12486 )
2 years ago
dedff70f90
[Improvement] Merge spi.utils into common.utils ( #12502 )
...
* improve JSONUtil
* merge spi.utils into common.utils
* split constants && remove some copy utils method
* remove StringUtils
2 years ago
f0a34cddec
[improvement-#12416] Use CollectionUtils.isEmpty judge the collection isEmpty to avoid NPE ( #12449 )
2 years ago
d46e4659d9
[Improvement][Style] Fix remaining style errors and wildcard imports ( #12412 )
...
* Fix remaining style errors and wildcard imports
* Upgrade spotless to the latest stable version, exclude .github folder from spotless check and fix dead link
* Fix spotless matching scope of markdown files
2 years ago
b357781df2
[Migrate][Test] Migrate junit4 -> junit5 test cases in common, service, and spi modules ( #12363 )
2 years ago
Wenjun Ruan
Jeremy
fuchanghai
He Zhao
Rick Cheng
xuhhui
He Wang
Aaron Wang
kezhenxu94
seedscoder
Jay Chung
Tq
qianli2022
lenian
JieguangZhou
Yann Ann
insist777
simsicon
amao
caishunfeng
Eric Gao