af7b359333
[Feature][Task Plugin] Add SageMaker Pipeline task plugin for MLOps scenario ( #10826 ) ( #10935 )
...
* add sagemaker pipeline task plugin
[DOC] add Sagemaker task plugin document
change license and fix dependencies
* [fix] Optimize the code
* Update dolphinscheduler-task-plugin/dolphinscheduler-task-sagemaker/src/main/java/org/apache/dolphinscheduler/plugin/task/sagemaker/SagemakerParameters.java
Optimization parameter judgment
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
* [fix] nips
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
2 years ago
f689220290
[Feature][Task Plugin] Increase zeppelin task stability in production ( #11010 )
...
* [Feature][Task Plugin] Increase zeppelin task stability in production (#10584 )
* Add front-end and update docs for the production mode of zeppelin task
* Fix minor front-end bug of zeppelin task plugin
* Refactor ZeppelinParameters with lombok
* Fix formatting
* Replace @Data with @Getter , @Setter , @ToString to avoid decrease in test coverage
2 years ago
35f32cd16e
[Improvement][Task]Improve seatunnel task command log ( #11027 )
2 years ago
2f0d907bcb
[Improvement][Task Plugin]Refactor JupyterParameters.java with lombok ( #11045 )
...
* [Improvement][Task Plugin]Refactor JupyterParameters.java with lombok (#11040 )
2 years ago
5e9c7dad23
Add dolphinscheduler-bom to manage the dependency version ( #11025 )
2 years ago
11a354927d
[fix] [MLflow Task plugin] Sometimes errors are reported using preset algorithms ( #10986 ) ( #10987 )
...
* Optimize the download method of preset project
* fix checkstyle
2 years ago
7196b125d9
[Feature][Datax Task]Datax task plugin support custom parameters ( #10950 )
...
* datax task plugin support custom parameters
2 years ago
a38fa34d43
[Feature][Task Plugin] Enable users to switch endpoints in zeppelin tasks ( #10925 )
...
* [Feature][Task Plugin] Enable users to switch endpoints in zeppelin tasks (#9814 )
2 years ago
427c58546c
[fix][plugin]: fix spark sql use custom parma. ( #10821 )
...
* [fix][plugin]: fix spark sql use custom parma.
* [fix][plugin] roll back part of the code
Co-authored-by: huaangcg <huangcg@getui.com>
2 years ago
02b2de2a1e
[fix] mlflow project success status ( #10897 )
...
close : #10902
2 years ago
499e5b1307
[bug][task-plugins]fix switch class name and add param check to the switch task plugin ( #10894 )
...
* fix switch class name and add param check to the switch task plugin
2 years ago
8eaf5a2309
[Feature-10219][EMR] EMR supports use <add-Steps> to add steps to an existing cluster ( #10657 )
...
* Add the ProgramType parameter to distinguish task types
* EmrAddStepsTask supports Add-Steps
* UI supports Add-Steps
* EmrTask modify the name of the class to EmrJobFlowTask
* add ERM Task abstract base class AbstractEmrTask
* add testcase for EmrAddStepsTask
* Modifying help Documents
2 years ago
71f0168510
[Feature] Enable users to create python env from requirements.txt ( #10658 )
2 years ago
bd1be6cf36
[Task] Added Apache SeaTunnel 2.x task node ( #10480 )
2 years ago
7d79a2165e
[Optimization]Optimize some details of MLFlow task plugin #10740 ( #10739 )
...
* Optimize some details of MLFlow task plugin
* Update dolphinscheduler-task-plugin/dolphinscheduler-task-mlflow/src/main/java/org/apache/dolphinscheduler/plugin/task/mlflow/MlflowParameters.java
* fix some nips
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2 years ago
5b0347e8da
[Task] Add Dinky task to better support the development and execution of FlinkSQL ( #10640 )
2 years ago
4a3c3e7797
[feature][task-flink] Support Flink application mode ( #9577 )
2 years ago
8f621ff98b
[Optimization] Calculate global parameter and local parameter at master. ( #10704 )
...
* Global parameter and local parameter calculation external expansion.
* k8s task ut fix.
* TimePlaceholderUtils import DateUtils fix
* follow the review comments to fix.
* follow the review comments to fix.
* e2e rerun
2 years ago
7e7df1d8ff
[Bug] [Task] SQL task no exception stack trace in task log ( #10656 )
2 years ago
66624c5c86
[Bug] [Master] Worker failover will cause task cannot be failover ( #10631 )
...
* fix worker failover may lose event
2 years ago
db595b3eff
Optimize master log, use MDC to inject workflow instance id and task instance id in log ( #10516 )
...
* Optimize master log, add workflow instance id and task instance id in log
* Use MDC to set the workflow info in log4j
* Add workflowInstanceId and taskInstanceId in MDC
2 years ago
36e20cdfc8
[Feature][Task Plugin] Add DVC task plugin for MLops scenario ( #10372 ) ( #10407 )
2 years ago
64cee03fed
[Feature] Enable users to switch and install conda env in jupyter task ( #10337 )
2 years ago
4be3b877c3
[feature][task] Enable zeppelin schedule a whole zeppelin note ( #10434 )
2 years ago
2c227ab384
[Feature-10363][server] S3 Resource center supports bucket customization ( #10433 )
...
* [Feature-10363][server] S3 Resource center supports bucket customization
* Modify the configuration. Sort out HDFS and S3 configurations to distinguish them
* Modify the configuration comment to clarify the configuration meaning
* Modify official documents to add missing configuration descriptions and clarify configuration meanings and usage
* Modify the configuration file in e2E to the correct definition, dolphinscheduler-e2e/dolphinscheduler-e2e-case/src/test/resources/docker/file-manage/common.properties
* Modified code to support bucket customization
* Check that the bucket on personal permission, Otherwise, the startup fails
* S3 increase support resource. Storage. Upload. Base. The base path path
* Change S3Utils implementation style to be consistent with HadoopUtils
* [Feature-10363][server] S3 Resource center supports bucket customization
* Modify code review comments
2 years ago
ad2646ff1f
Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe ( #10479 )
...
* Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe
3 years ago
3258438f6e
[Feature] [MLOps] support mlflow deploy with docker compose ( #10217 )
...
* [Feature] [MLOps] support mlflow deploy with docker compose
fix doc
Update docs/docs/en/guide/task/mlflow.md
fix doc
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
revert cancel modification
fix ENV name and docker compose command
* fix doc image link
* fix testModelsDeployDockerCompose
* add docker compose container health check and fix mlflow bug
* update docker compose healthcheck timeout
3 years ago
bbba20d39a
change to common lang3 and fix checkstyle ( #10380 )
3 years ago
b1fb17119b
[Feature-6758][Task] Add limit resource usage for tasks base on cgroup ( #10373 )
3 years ago
da3c25dc67
[improve] Using create or replace function in sql task ( #10170 )
3 years ago
0bf4043d5f
Move datasource params to datasouce plugin module ( #10353 )
...
* Move datasource params to datasouce plugin module
3 years ago
d80cf21456
Clean up unused dependencies and packaging issues ( #9944 )
3 years ago
a8a5da367a
[Feature][Task Plugin] Add support for dynamic form for zeppelin task plugin ( #9977 ) ( #10269 )
3 years ago
c5961aed25
[feat][task] Add OpenMLDB task plugin ( #10198 )
3 years ago
54ab2651bd
[Feature][MLops] Add MLflow Projects (custom project) component ( #10175 )
...
* Add MLflow Projects (custom project) component
* fix something
* update document
* update document
* Supplementary test cases
3 years ago
e11373b963
[Feature][MLops] Support MLflow Models to deploy model service (MLflow models serve and Docker) ( #10150 )
...
* Add backend of MLflow Moldes component
* Add front end of MLflow Moldes component
* [DOC] update mlflow document
* revert unnecessary
* fix doc image url
* Update docs/docs/en/guide/task/mlflow.md
Discard the lr abbreviation
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
3 years ago
9ab3bebc90
[Feature][Task Plugin] Add mlflow task plugin for MLops scenario. ( #10036 )
...
* [Feature][Task Plugin] Add mlflow task plugin (#9725 )
* [Feature][Task Plugin] Add mlflow task plugin UI (#9725 )
* [Feature][Task Plugin] fix license header (#9725 )
* [Feature][Task Plugin] fix license header (#9725 )
* [Feature][Task Plugin] revert unnecessary
* [Feature][Task Plugin] add auto ml to mlflow task plugin
* [DOC] add mlflow document
* [DOC] fix mlflow docs imgs
* [DOC] fix dead link localhost:5000
* [DOC] fix dead link localhost:5000
* [DOC] remove dead link localhost:5000
* Update docs/docs/en/guide/task/mlflow.md
* Update docs/docs/zh/guide/task/mlflow.md
* [DOC] format ui code
* [DOC] remove dead link localhost:5000
* [Feature][Task Plugin] revert unnecessary
* fix some nits
* Move the personal repository to the public repository.
* Run the command directly instead of saving the file
* fix paramsMap initialize
* revert unnecessary
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
3 years ago
5bb1eb04fc
[Feature][plugin] Add k8s task in task plugin ( #9425 )
...
* [Feature][plugin] Add k8s task in task plugin
* [Feature][plugin] fix dos and code problems
* [Feature][plugin] refactor some code based on sonar
* [Feature][UI] front-end for k8s task plugin
* [Feature][plugin] delete some front files
* [Feature][plugin] update document
* Update docs/docs/zh/guide/task/k8s.md
* Update docs/docs/en/guide/task/k8s.md
* Update docs/docs/en/guide/task/k8s.md
* Update docs/docs/en/guide/task/k8s.md
* [Feature][UI] front-end change from review comments
* [Feature][UI] replace get namespace list api
* [Feature][plugin] change file name
* Add kubernetes to zh task list
Co-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: William Tong <weitong@cisco.com>
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
3 years ago
d643e1c1cf
[Fix-10039] Flink run command when perfecting Python jobs ( #10042 )
...
* [fix] flink task
* [fix] flink task
3 years ago
359cbe2a1e
[dataquality] Fix task commnd null bug ( #9974 )
3 years ago
df04c4aef0
[fix-9991][worker]fix statement is closed before resultSet.getMetaData() ( #10014 )
3 years ago
80369363c9
[feat][task plugin] Add Jupyter task plugin ( #9872 )
...
Co-authored-by: Amy0104 <97265214+Amy0104@users.noreply.github.com>
3 years ago
2d36449444
[Feature- 9837][plugin/ui] support FlinkSQL Task ( #9840 )
...
* flink_sql
* '[refactor]flinksql'
* '[refactor]flinksql'
* '[refactor]flinksql'
* '[refactor]flinksql'
* '[refactor]flinksql'
* '[refactor]flinksql'
* '[refactor]flinksql'
* '[refactor]flinksql'
* '[refactor]flinksql'
* '[refactor]flinksql'
* '[refactor]flinksql'
* '[refactor]flinksql'
* [refactor] flinksql
* '[refactor]flinksql'
* [refactor] dolphinscheduler flinksql
* [refactor] flink sql
* [refactor] flink sql
* [refactor] flink sql
* [refactor] flink sql
* [refactor] flink sql
* [refactor] flink sql
* [refactor] flink sql
* [refactor] flink sql
* [refactor] flink sql
* [refactor] flink sql
* [refactor] flink sql
* [refactor] flink sql
* [refactor] flink sql
* [refactor] flink sql
* [refactor] flink sql
* [refactor] flink sql
* [refactor] flink sql
* [refactor] flink sql
* [refactor] flink sql
* [refactor] flink sql
* Update docs/docs/en/guide/task/flink.md
* Update docs/docs/zh/guide/task/flink.md
* prettier front-end code
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
Co-authored-by: devosend <devosend@gmail.com>
3 years ago
0e7c98d465
[Feature-9177][Task] The sql task supports configuring segmentation notation to provide execution of multiple statements ( #9917 )
...
* Support sql segmentation to execute multiple functions.
3 years ago
ed425d2d2d
[Feature-9772][plugin/ui] support SparkSQL Task ( #9790 )
...
* [refactor] Dolphinscheduler sparkSQL
* [refactor] Dolphinscheduler SparkSQL
* [refactor] Dolphinscheduler SparkSQL
* [refactor] Dolphinscheduler SparkSQL
* [refactor] Dolphinscheduler SparkSQL
* [refactor] Dolphinscheduler SparkSQL
* [refactor] Dolphinscheduler SparkSQL
* [refactor] Refactor ui code and add sparksql test cases
* [refactor] refactor dolphinscheduler SparkSQL
* [refactor] refactor dolphinscheduler plugin-sparkSQL
* [refactor] refactor dolphinscheduler plugin-SparkSQL
* [refactor] dolphinscheduler plugin-SparkTaskTest
* [refactor] dolphinscheduler plugin-SparkTask
* [refactor] dolphinscheduler plugin-Spark
* [refactor] dolphinscheduler plugin-SparkTask-SparkSQL
* [refactor] dolphinscheduler plugin-spark-SparkTask
* [refactor] dolphinscheduler plugin-spark-SparkTask redefine code
3 years ago
206b7c1c51
[Fix-9352][datasource-api]data source client uses a cache with an expiration time ( #9353 )
...
* datasource client cache with expiration time
* remove unused import
* fixed unit test
* fix
* fix
* fix
* fix
* fix
* fix
* fix
3 years ago
36f01155b5
[Improvement][server] varPool support syntax #{setValue(key=value)} ( #9586 )
3 years ago
c5b7e5adff
[Bug] [API-9558]fix homepage task instance count method to use submit time to recount ( #9559 )
...
* fix homepage task instance count method to use submit time to recount
* fix homepage task instance count method to use submit time to recount
* fix homepage task instance count method to use submit time to recount
* fix homepage task instance count method JUNIT
* fix homepage task instance count method JUNIT
* fix homepage task instance count method JUNIT
3 years ago
a5bbf7852d
[Feature][Task-Plugin]Add zeppelin task-plugin to support Apache Zeppelin ( #9327 )
3 years ago
7f41a96fc1
[Fix-9525] [Worker] Environment did not work as expected ( #9527 )
...
* fix #9525
* change to ${PYTHON_HOME}
* remove import
* fix ut error
3 years ago
JieguangZhou
Eric Gao
xuhhui
Wenjun Ruan
zhuxt2015
chuxing
Tq
ZhaoGuodong
Kerwin
aiwenmo
Paul Zhang
WangJPLeo
xiangzihao
He Zhao
kezhenxu94
HuangWei
Dannila
zixi0825
sq-q
exmy