e5d06c40a5
[Feature-10182][task-datax] ( #11387 )
...
In the process of realizing datax synchronization, use custom json for better connection to enable kerberos-authenticated clusters, such as using plug-ins such as hdfs and hbase
2 years ago
f6ef628030
Support reading application properties from kubernetes config map ( #11402 )
2 years ago
896fef6c98
[Feature][Style] Enable spotless to manage imports ( #11458 )
2 years ago
b79bad19e5
[Fix-11051][Task]Fix the process exitValue exception when the process times out and th… ( #11099 )
2 years ago
5fa3e7b1ed
Add pause to task instance status ( #11390 )
2 years ago
6345a13cdc
[Fix-10392][SqlTask]: Fix list param error when use sql task ( #11285 )
...
* FIX: fix list param in.
* FIX: code bug
* FIX: fix one value
* FIX: header
* FIX: ci
* FIX: fix ci
* MOD: for review
* MOD: for review
2 years ago
0464123c2b
[Feature-11223] support stream task ( #11350 )
...
* add task execute type
* update task definition list paging
* update task instance list paging
* stream task start
* [Feature][UI] Some changes to execute task.
* Set the connection edge to dashed line.
* Add FLINK_STREAM task.
* add stream task
* flink savepoint and cancel
* fix query bug
* add stream task definition
* add task instance for stream task
* delete stream task definition state
* update api for stream task definition edit
* modify search for stream task instance
* add language
* delete task type search for stream task definition
* change task type search for stream task instance
* add jump button
* add savepoint
* add down log for stream task instance
* ui test
* stream task start
* run DAG
* [Fix][UI] Fix the stream task edgs not to be dashed when filling back.
* [Feature][UI] Remove some fields for FLINK_STREAM.
* add start modal
* add dryRun column for stream task instance
* fix duration
* fix pon
* fix build error
* Add success tip
* add auto sync for stream task instance
* remove forgien key for task instance
* license header
* UT fix
* modify locales
* recover common config
* fix UT
* add doc
Co-authored-by: Amy <amywang0104@163.com>
Co-authored-by: devosend <devosend@gmail.com>
2 years ago
b3919d30e7
[Improvement-11394][Task Plugin] Remove Passed Parameter --queue in Seatunnel Task Plugin ( #11395 )
2 years ago
03582f2e70
[Fix-11376][task plugin]Seatunnel Task cannot save successfully when … ( #11377 )
2 years ago
8774415197
Split ExecutionStatus to WorkflowExecutionStatus and TaskExecutionStatus ( #11340 )
2 years ago
6a02870926
[Feature][style] Add spotless maven plugin for automatic style fix. ( #11272 )
...
* [Feature][style] Add spotless maven plugin for automatic style fix (#10963 )
* Fix spotless ratchet configuration
* Remove license-check and decrease line length threshold value
* Update related docs
* Remove checkstyle and add pre-commit hook
* Test updated pre-commit hook
* Replace checkstyle with spotless in CI
* Remove reviewdog
2 years ago
9a53c6ac34
fix work oom when task logs's size is very large ( #11224 )
2 years ago
bfff3a7c5d
fix error ( #11206 )
2 years ago
513f336015
[Feature] Add new task type chunjun ( #10937 )
2 years ago
818648df7d
[Bug-11071] fix local params ( #11184 )
...
* fix local params
* add CI case
2 years ago
052ceaacd0
Add is success in loop task status ( #11169 )
2 years ago
cdfe115247
[Feature-10837][Task Plugin] Sqoop Task Add --split-by Parameter ( #10960 )
...
* [Feature-10837][Task Plugin] Sqoop Task Add --split-by Parameter
* checkstyle
2 years ago
c7789bf049
Add base loop task execotor and http template parser ( #11137 )
...
* Add base loop task execotor and http template parser
* Add JsonPathUtils
2 years ago
af7b359333
[Feature][Task Plugin] Add SageMaker Pipeline task plugin for MLOps scenario ( #10826 ) ( #10935 )
...
* add sagemaker pipeline task plugin
[DOC] add Sagemaker task plugin document
change license and fix dependencies
* [fix] Optimize the code
* Update dolphinscheduler-task-plugin/dolphinscheduler-task-sagemaker/src/main/java/org/apache/dolphinscheduler/plugin/task/sagemaker/SagemakerParameters.java
Optimization parameter judgment
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
* [fix] nips
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2 years ago
f689220290
[Feature][Task Plugin] Increase zeppelin task stability in production ( #11010 )
...
* [Feature][Task Plugin] Increase zeppelin task stability in production (#10584 )
* Add front-end and update docs for the production mode of zeppelin task
* Fix minor front-end bug of zeppelin task plugin
* Refactor ZeppelinParameters with lombok
* Fix formatting
* Replace @Data with @Getter , @Setter , @ToString to avoid decrease in test coverage
2 years ago
35f32cd16e
[Improvement][Task]Improve seatunnel task command log ( #11027 )
2 years ago
2f0d907bcb
[Improvement][Task Plugin]Refactor JupyterParameters.java with lombok ( #11045 )
...
* [Improvement][Task Plugin]Refactor JupyterParameters.java with lombok (#11040 )
2 years ago
5e9c7dad23
Add dolphinscheduler-bom to manage the dependency version ( #11025 )
2 years ago
11a354927d
[fix] [MLflow Task plugin] Sometimes errors are reported using preset algorithms ( #10986 ) ( #10987 )
...
* Optimize the download method of preset project
* fix checkstyle
2 years ago
7196b125d9
[Feature][Datax Task]Datax task plugin support custom parameters ( #10950 )
...
* datax task plugin support custom parameters
2 years ago
a38fa34d43
[Feature][Task Plugin] Enable users to switch endpoints in zeppelin tasks ( #10925 )
...
* [Feature][Task Plugin] Enable users to switch endpoints in zeppelin tasks (#9814 )
2 years ago
427c58546c
[fix][plugin]: fix spark sql use custom parma. ( #10821 )
...
* [fix][plugin]: fix spark sql use custom parma.
* [fix][plugin] roll back part of the code
Co-authored-by: huaangcg <huangcg@getui.com>
2 years ago
02b2de2a1e
[fix] mlflow project success status ( #10897 )
...
close : #10902
2 years ago
499e5b1307
[bug][task-plugins]fix switch class name and add param check to the switch task plugin ( #10894 )
...
* fix switch class name and add param check to the switch task plugin
2 years ago
8eaf5a2309
[Feature-10219][EMR] EMR supports use <add-Steps> to add steps to an existing cluster ( #10657 )
...
* Add the ProgramType parameter to distinguish task types
* EmrAddStepsTask supports Add-Steps
* UI supports Add-Steps
* EmrTask modify the name of the class to EmrJobFlowTask
* add ERM Task abstract base class AbstractEmrTask
* add testcase for EmrAddStepsTask
* Modifying help Documents
2 years ago
71f0168510
[Feature] Enable users to create python env from requirements.txt ( #10658 )
2 years ago
bd1be6cf36
[Task] Added Apache SeaTunnel 2.x task node ( #10480 )
2 years ago
7d79a2165e
[Optimization]Optimize some details of MLFlow task plugin #10740 ( #10739 )
...
* Optimize some details of MLFlow task plugin
* Update dolphinscheduler-task-plugin/dolphinscheduler-task-mlflow/src/main/java/org/apache/dolphinscheduler/plugin/task/mlflow/MlflowParameters.java
* fix some nips
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2 years ago
5b0347e8da
[Task] Add Dinky task to better support the development and execution of FlinkSQL ( #10640 )
2 years ago
4a3c3e7797
[feature][task-flink] Support Flink application mode ( #9577 )
2 years ago
8f621ff98b
[Optimization] Calculate global parameter and local parameter at master. ( #10704 )
...
* Global parameter and local parameter calculation external expansion.
* k8s task ut fix.
* TimePlaceholderUtils import DateUtils fix
* follow the review comments to fix.
* follow the review comments to fix.
* e2e rerun
2 years ago
7e7df1d8ff
[Bug] [Task] SQL task no exception stack trace in task log ( #10656 )
2 years ago
66624c5c86
[Bug] [Master] Worker failover will cause task cannot be failover ( #10631 )
...
* fix worker failover may lose event
2 years ago
db595b3eff
Optimize master log, use MDC to inject workflow instance id and task instance id in log ( #10516 )
...
* Optimize master log, add workflow instance id and task instance id in log
* Use MDC to set the workflow info in log4j
* Add workflowInstanceId and taskInstanceId in MDC
2 years ago
36e20cdfc8
[Feature][Task Plugin] Add DVC task plugin for MLops scenario ( #10372 ) ( #10407 )
2 years ago
64cee03fed
[Feature] Enable users to switch and install conda env in jupyter task ( #10337 )
2 years ago
4be3b877c3
[feature][task] Enable zeppelin schedule a whole zeppelin note ( #10434 )
2 years ago
2c227ab384
[Feature-10363][server] S3 Resource center supports bucket customization ( #10433 )
...
* [Feature-10363][server] S3 Resource center supports bucket customization
* Modify the configuration. Sort out HDFS and S3 configurations to distinguish them
* Modify the configuration comment to clarify the configuration meaning
* Modify official documents to add missing configuration descriptions and clarify configuration meanings and usage
* Modify the configuration file in e2E to the correct definition, dolphinscheduler-e2e/dolphinscheduler-e2e-case/src/test/resources/docker/file-manage/common.properties
* Modified code to support bucket customization
* Check that the bucket on personal permission, Otherwise, the startup fails
* S3 increase support resource. Storage. Upload. Base. The base path path
* Change S3Utils implementation style to be consistent with HadoopUtils
* [Feature-10363][server] S3 Resource center supports bucket customization
* Modify code review comments
2 years ago
ad2646ff1f
Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe ( #10479 )
...
* Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe
2 years ago
3258438f6e
[Feature] [MLOps] support mlflow deploy with docker compose ( #10217 )
...
* [Feature] [MLOps] support mlflow deploy with docker compose
fix doc
Update docs/docs/en/guide/task/mlflow.md
fix doc
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
revert cancel modification
fix ENV name and docker compose command
* fix doc image link
* fix testModelsDeployDockerCompose
* add docker compose container health check and fix mlflow bug
* update docker compose healthcheck timeout
2 years ago
bbba20d39a
change to common lang3 and fix checkstyle ( #10380 )
2 years ago
b1fb17119b
[Feature-6758][Task] Add limit resource usage for tasks base on cgroup ( #10373 )
2 years ago
da3c25dc67
[improve] Using create or replace function in sql task ( #10170 )
3 years ago
0bf4043d5f
Move datasource params to datasouce plugin module ( #10353 )
...
* Move datasource params to datasouce plugin module
3 years ago
d80cf21456
Clean up unused dependencies and packaging issues ( #9944 )
3 years ago
lugela
kezhenxu94
Eric Gao
Kerwin
Wenjun Ruan
Stalary
caishunfeng
Assert
microeastcowboy
xuhhui
JieguangZhou
zhuxt2015
chuxing
Tq
ZhaoGuodong
aiwenmo
Paul Zhang
WangJPLeo
xiangzihao
He Zhao