606b56403e
[Fix-13381][plugin] fix error while parsing hive load sql ( #13378 )
...
* [fix][task plugin][task sql] fix parsing error while using variables sometimes parsing exception happened if task sql contains a variable followed by \" or \' .
for example:
input sql: load inpath '/tmp/test_table/dt=${dt}'
into table test_table partition(dt=${dt})
after replace: preparing : load inpath '/tmp/test_table/dt=?
into table test_table partition(dt=?)
2 years ago
8315279620
[Feature-#14148][Task] Added status check interval and dependent failure policy parameters for dependent task nodes ( #14150 )
...
* Added a dependent task failure policy in the dependent task node
2 years ago
67f50e3904
[fix-#13940] add status of ``` stop``` in class of TaskExecutionStatus ( #14016 )
2 years ago
c3809fbecf
[Improvement-13868][K8S Task] Support the custom labels for k8s task ( #14067 )
...
* [Improvement-13868] Support the custom labels for k8s task
* fix front-end type mismatched
2 years ago
3eb4c0ed1d
[Improvement-14054][Worker] Kill multiple yarn apps at the same time ( #14055 )
...
* [Improvement-14054][Worker] Kill multiple yarn apps at the same time
* Update dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/am/YarnApplicationManager.java
Co-authored-by: Aaron Wang <wangweirao16@gmail.com>
2 years ago
84edea7593
[Improvement-14052][Log] Remove the useless filter in getAppIdsFromLogFile ( #14053 )
2 years ago
3bad68a942
[Bug-13929][K8S Task] Set command contains single quote is invalid ( #13930 )
...
* [Fix-13929][K8S Task] Set command contains single quote is invalid
* fix UT
2 years ago
cf91544b99
Refactor logic task ( #13948 )
2 years ago
db6c061c8b
[Feature][plugin] Add databend datasource in datasource plugin ( #13866 )
...
Co-authored-by: Jay Chung <zhongjiajie955@gmail.com>
2 years ago
45586f71d5
[BUG][Resource Center] Task can not select main JAR ( #13536 )
2 years ago
119f08d2ba
[Bug-13882][Worker] Upgrade kubernetes-client version to 6.0.0 ( #13891 )
2 years ago
d772f2492b
fix: when sudo enable is false, execute command will failed ( #13860 )
2 years ago
d6fcf60534
[Improvement-13751][Worker] Support real-time pod log collection ( #13752 )
...
* Support real-time pod log collection
* datax task test have to mock ShellCommandExecutor with whole dependencies
* replace \n with EMPTY_STRING
* modify error log msg
* Update dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/AbstractCommandExecutor.java
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
* throw exception in pod log collection thread
---------
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2 years ago
9df89cf0cf
Use bootstrap user as the default tenant ( #13825 )
2 years ago
85605bbef9
[Improvement-13653][Remote Logging] Support api-server to get task log from remote target ( #13654 )
2 years ago
d1b6e6f02c
Take over task instance in master failover ( #13798 )
2 years ago
b98566e070
[Bug-13782] match ')' in setValue regular expression ( #13787 )
2 years ago
2dceb6b8ca
[Feature-13428][Resource Center] Add support for Google Cloud Storage in the resource center ( #13562 )
2 years ago
aef5524ee7
Clearer task runnable. ( #13689 )
...
* Clean unused method in AbstractTask
* Kill task by process.destroy
* wait 5/s after destroy process
2 years ago
c9066e8de9
Use MDC to filter task instance log ( #13673 )
...
* Use MDC to collect task instance log
* Use MDCAutoClosableContext to remove the MDC key
2 years ago
fd720880c7
[Improvement] Add missing placeholder ( #13663 )
...
* Add missing placeholder
* Update dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/am/KubernetesApplicationManager.java
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
---------
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
2 years ago
ca9df868f1
Fix task instance log order might be wrong due to async print ( #13660 )
2 years ago
7586710d49
[fix-13625] Modify systemd-run from MemoryMax to MemoryLimit for more generality ( #13627 )
...
Co-authored-by: xiecao <haolong.zhl@alibaba-inc.com>
2 years ago
4351a25f2a
Add execute function to handle the workflow instance operation ( #13610 )
2 years ago
7bf3e3cdd6
[improve-#13201] update pid during running ( #13206 )
2 years ago
047fa2f65e
[Feature-13511] Submit Spark task directly on Kubernetes ( #13550 )
2 years ago
2bd65fb2df
[Feature][Remote Logging] Add support for writing task logs to OSS ( #13332 )
2 years ago
16b193454b
[Improvement][Task Plugin] Support Task Plugin output response as a parameter ( #13529 )
...
* support http output response as param
* add UT
* fix sqoop UT
2 years ago
8d12dc0702
[Improvement-13491] Use lombok @Slf4j annotation to generate logger ( #13509 )
2 years ago
5a9cb7ee64
Fix task log might delay until task finished due to batch flush ( #13502 )
2 years ago
78ba71bdbd
[Feature][task]add azure datafactory task type ( #13417 )
...
* init data factory
* add task-type-config.yaml
* add data factory ui
* submit doc
* submit license
---------
Co-authored-by: devosend <devosend@gmail.com>
2 years ago
ba0a253f09
fix k8s task run fail ( #13348 )
2 years ago
6634ff98af
Add source env list in common.properties ( #13317 )
2 years ago
14ec4a2398
Remove dao module in worker ( #13242 )
2 years ago
042ec74a24
[refactor] cache compiled regex pattern ( #13208 )
...
* [refactor] cache compiled regex pattern
* [refactor] cache compiled regex pattern
* [refactor] cache compiled regex pattern
2 years ago
70ccffeee2
Format task parameter as pretty json ( #13173 )
2 years ago
560d232cbd
[Bug][Worker] Fix duplicated key bug when have out parameters of same name ( #12930 )
2 years ago
32a8241ef2
Don't source env file in task, this will override tenant's environment ( #13028 )
2 years ago
d480c271ec
[Bug-12994] [Worker] Fix kill process does not take effect ( #12995 )
2 years ago
92a9b7e243
Use System#lineSeparator to replace /n ( #12889 )
2 years ago
229c554912
[feature][task] Add Kubeflow task plugin for MLOps scenario ( #12843 )
2 years ago
43d69ae157
Upgrade clickhouse jdbc driver ( #12639 )
2 years ago
c4fb555883
Modify shell env to bash ( #12742 )
2 years ago
eaef68856b
Change command file permission to 755 ( #12678 )
...
(cherry picked from commit 95484d930762712f4ab68ece27ca3b65a2073ab2)
2 years ago
a47b4d0672
[Feature][Task] Transfer files between tasks ( #12552 )
...
* add data transfer between tasks
* add delete DATA_TRANSFER API
* convert Result to DeleteDataTransferResponse
* add api UT
* fix final
* fix doc
2 years ago
b71d4983dd
[Feature-6586][Server]add some ds process definition demo when init ( #11759 )
...
* [Feature-6586][Server]add some ds process definition demo when init
1.add some ds process definition demo when init, to display what task type can run and make user easy to
use ds.
2.need configure the JVM parameters (-Ddemo=true) to turn on the StandaloneServer service
3.need modify the tenant information in it
2 years ago
3c31ddfd9f
[Improvement-12536][k8s] Support the command for the container in k8s task plugin ( #12538 )
2 years ago
08335b1032
[Improvement][Task] Improved way to collect yarn job's appIds ( #12197 )
...
* Provide aop way as an optional way to collect yarn job's applicationId, and import new module `dolphinscheduler-aop` to place the aop code.
* Add user property `appId.collect` for user to decide how to collect applicationId.
* Add new environment configuration for each type of yarn tasks to support aop in `dolphinscheduler_env.sh`
* Update docs to declare how to use aop way.
* Update `LogUtils` to support fetch applicationId in different ways based on the user property.
Co-authored-by: gabrywu <gabrywu@apache.com>
2 years ago
dedff70f90
[Improvement] Merge spi.utils into common.utils ( #12502 )
...
* improve JSONUtil
* merge spi.utils into common.utils
* split constants && remove some copy utils method
* remove StringUtils
2 years ago
f0a34cddec
[improvement-#12416] Use CollectionUtils.isEmpty judge the collection isEmpty to avoid NPE ( #12449 )
2 years ago
Alex Ting
Kerwin
fuchanghai
Aaron Wang
Rick Cheng
Wenjun Ruan
Jeremy
huangxinan1990
QuantumXiecao
JieguangZhou
seedscoder
Tq
caishunfeng
youzipi
simsicon
amao