ebcffb04aa
fix hdfs defaultFs not working ( #11823 )
2 years ago
444c7e6eb7
[Improvement] Replace commons-lang 2 function invocations with commons-lang3 ( #11810 )
...
* [Improvement] Replace commons-lang 2 function invocations with commons-lang3
2 years ago
b3cc8a55b9
[feature][API]feature: add task type list and enable user to add task type to fav ( #11727 )
...
* feature: add task type list and enable user to add task type to fav
* make DependentExecute.java variable protected
* add standalone server fileSet
2 years ago
67e7f88d8b
Refactor heart beat task, use json to serialize/deserialize ( #11702 )
...
* Refactor heart beat task, use json to serialize/deserialize
2 years ago
03e1e6fe45
[Bug] [Worker] Optimize the getAppId method to avoid worker OOM when kill task ( #11701 )
...
* Fix kill job may cause worker oom
2 years ago
c110f4acae
Refresh server startup time when recovery from waiting ( #11595 )
2 years ago
13e4720aa2
Upgrade spring boot and swagger ( #11574 )
2 years ago
3b72c6efe7
[Improvement]Add a stop function when you are ready to do pause operation ( #11543 )
...
* add can stop for execute state machine
* add execute type check for pause operation
2 years ago
7ff34c3947
[Feature-7024] Add waiting strategy to support master/worker can recover from registry lost ( #11368 )
...
* Add waiting strategy to support master/worker can recover from registry lost
* throw exception when zookeeper registry start failed due to interrupted
2 years ago
6e2bed4d6b
[fix-11378] fix API-Server release response message typo ( #11400 )
...
* [fix-11295] remove duplicate classes
* [fix-11378] fix response message error
Co-authored-by: fuchanghai <changhai.fu@marketingforce.com>
2 years ago
0464123c2b
[Feature-11223] support stream task ( #11350 )
...
* add task execute type
* update task definition list paging
* update task instance list paging
* stream task start
* [Feature][UI] Some changes to execute task.
* Set the connection edge to dashed line.
* Add FLINK_STREAM task.
* add stream task
* flink savepoint and cancel
* fix query bug
* add stream task definition
* add task instance for stream task
* delete stream task definition state
* update api for stream task definition edit
* modify search for stream task instance
* add language
* delete task type search for stream task definition
* change task type search for stream task instance
* add jump button
* add savepoint
* add down log for stream task instance
* ui test
* stream task start
* run DAG
* [Fix][UI] Fix the stream task edgs not to be dashed when filling back.
* [Feature][UI] Remove some fields for FLINK_STREAM.
* add start modal
* add dryRun column for stream task instance
* fix duration
* fix pon
* fix build error
* Add success tip
* add auto sync for stream task instance
* remove forgien key for task instance
* license header
* UT fix
* modify locales
* recover common config
* fix UT
* add doc
Co-authored-by: Amy <amywang0104@163.com>
Co-authored-by: devosend <devosend@gmail.com>
2 years ago
8774415197
Split ExecutionStatus to WorkflowExecutionStatus and TaskExecutionStatus ( #11340 )
2 years ago
05932bcf55
[fix-11295] remove duplicate classes ( #11355 )
2 years ago
c34503e3f6
[fix] Process definition length too long when copy or import ( #10621 )
2 years ago
8e21c38c00
Write alert result into db ( #11221 )
2 years ago
efd68f4805
[Fix-10665] [S3] Fix s3 download method ( #10675 )
...
when overwrite param is true and dest file is exist, throw exception
make dirs for ancestor directories of the source file path if it is not exists
To describe the error more clearly, change "the file isn`t exists" to specific error message
Co-authored-by: sheldonliu <sheldonliu>
2 years ago
6f0a609793
[Improvement 11072][common] Added HTTPS request support in kerberos environment ( #11127 )
...
* [Improvement-11072][dolphinscheduler-common] Added HTTPS request support in kerberos environment
* [Improvement-11072][dolphinscheduler-common] supplement
Co-authored-by: liyangyang <liyangyang@bizseer.com>
2 years ago
f13e7a95ae
[Improvement-11013][dolphinscheduler-common] YarnHAAdminUtils#getActiveRMName function Add HTTPS Hadoop environment support ( #11017 )
...
* [Improvement-11013][dolphinscheduler-common] YarnHAAdminUtils#getActiveRMName function Add HTTPS Hadoop environment support
* [Improvement-11013][dolphinscheduler-common] add test code
* [Improvement-11013][dolphinscheduler-common] code format
* [Improvement-11013][dolphinscheduler-common] add hadooputils test code
* [Improvement-11013][dolphinscheduler-common] remove extra blank lines
* [Improvement-11013][dolphinscheduler-common] adjusts test code
* [Improvement-11013][dolphinscheduler-common] remove useless import
* [Improvement-11013][dolphinscheduler-common] add test code
* [Improvement-11013][dolphinscheduler-common] shortened string
* [Improvement-11013][dolphinscheduler-common] format test code style
Co-authored-by: liyangyang <liyangyang@bizseer.com>
2 years ago
28b40da1b1
[Improve] Hide Change Pwd Page When Login With LDAP ( #11041 )
...
* hide-change-pwd-page-when-ldap-login
* add unit test
2 years ago
5e9c7dad23
Add dolphinscheduler-bom to manage the dependency version ( #11025 )
2 years ago
2be1d4bf0a
Fix worker cannot shutdown due to resource close failed or heart beat check failed ( #10979 )
...
* Use try-with-resource to close resource, and add heart error threshold to avoid worker cannot close due to heart beat check failed
* Move heartbeat error threshold to applicaiton.yml
2 years ago
a38fa34d43
[Feature][Task Plugin] Enable users to switch endpoints in zeppelin tasks ( #10925 )
...
* [Feature][Task Plugin] Enable users to switch endpoints in zeppelin tasks (#9814 )
2 years ago
0db9bbd538
[fix] delete workflow or task used by other tasks ( #10873 )
...
currently, users can delete process definitions used
in other sub-process tasks or in other dependent tasks.
but this change will break the dependence of those task
and failed DAG, this patch add validation of those
delete behavior to avoid this error
2 years ago
59cd86157f
[Python] Supports creating or editing resources. ( #10823 )
2 years ago
f639a2eed4
[Fix-10854] Fix database restart may lost task instance status ( #10866 )
...
* Fix database update error doesn't rollback the task instance status
* Fix database error may cause workflow dead with running status
2 years ago
030fb89d6e
[Fixed-10833] [Bug] [Quartz] timezone display doesn't match the next_fire_time in ds 3.0.0-beta1 version ( #10865 )
...
* closed [10619] [Improvement][Master] batch remove TaskInstaceId and workflowInstanceId
* fixed # 10833 timezone display error
* checkstyle
2 years ago
3f69ec8f28
[Fix-10842] Fix master/worker failover will cause status incorrect ( #10839 )
...
* Fix master failover will not update task instance status
* Add some failover log
* Fix worker failover will rerun task more than once
* Fix workflowInstance failover may rerun already success taskInstance
2 years ago
1e48bbc5c6
[Feature-10629][Improvement] Support multi cluster environments - namespace use cluster ( #10650 )
...
* server code
* namespace ui
* fix dao test
* fix test
* fix
* docs and image update
Co-authored-by: qianl4 <qianl4@cicso.com>
2 years ago
426567348e
Remove quartz in service ( #10748 )
...
* Remove quartz in service
2 years ago
67d14fb7b3
[Fix-10785] Fix state event handle error will not retry ( #10786 )
...
* Fix state event handle error will not retry
* Use state event handler to deal with the event
2 years ago
d2fe16d252
[Optimization] Config module resource relation fix. ( #10718 )
...
* Config module resource relation fix.
* Boolean judge fix
* e2e rerun
* e2e rerun
* e2e rerun
* ArrayList fix
* change variable name.
* Add transaction processing on create data resource method
2 years ago
8f621ff98b
[Optimization] Calculate global parameter and local parameter at master. ( #10704 )
...
* Global parameter and local parameter calculation external expansion.
* k8s task ut fix.
* TimePlaceholderUtils import DateUtils fix
* follow the review comments to fix.
* follow the review comments to fix.
* e2e rerun
2 years ago
e0b0f986f7
[Fix-10641] [Common] Fix the hdfs NPE ( #10642 )
...
Fix the hdfs NPE (#10642 )
2 years ago
66624c5c86
[Bug] [Master] Worker failover will cause task cannot be failover ( #10631 )
...
* fix worker failover may lose event
2 years ago
b5184138fa
[Feature] Time function analysis extension. ( #10624 )
...
* Time function analysis extension.
* param add.
* clear useless logs and update method notes
* permission omission fix.
* extending time functions to optimize static methods
* e2e rerun.
2 years ago
db595b3eff
Optimize master log, use MDC to inject workflow instance id and task instance id in log ( #10516 )
...
* Optimize master log, add workflow instance id and task instance id in log
* Use MDC to set the workflow info in log4j
* Add workflowInstanceId and taskInstanceId in MDC
2 years ago
b2879c0e05
[Fix] fix the HttpUtilsTest' test case , it is always time out. ( #10294 )
2 years ago
9345c16f3f
[Improve]Remove the code logger.isDebugEnabled() #5918 ( #10542 )
2 years ago
117f78ec4b
[Fix-10413] Fix Master startup failure the server still hang ( #10500 )
...
* Fix Master startup failure the server still hang
2 years ago
2c227ab384
[Feature-10363][server] S3 Resource center supports bucket customization ( #10433 )
...
* [Feature-10363][server] S3 Resource center supports bucket customization
* Modify the configuration. Sort out HDFS and S3 configurations to distinguish them
* Modify the configuration comment to clarify the configuration meaning
* Modify official documents to add missing configuration descriptions and clarify configuration meanings and usage
* Modify the configuration file in e2E to the correct definition, dolphinscheduler-e2e/dolphinscheduler-e2e-case/src/test/resources/docker/file-manage/common.properties
* Modified code to support bucket customization
* Check that the bucket on personal permission, Otherwise, the startup fails
* S3 increase support resource. Storage. Upload. Base. The base path path
* Change S3Utils implementation style to be consistent with HadoopUtils
* [Feature-10363][server] S3 Resource center supports bucket customization
* Modify code review comments
2 years ago
062146eecd
[Fix] Fix the JSONUtils tool class time zone problem #10282 ( #10284 )
...
* [Fix] Fix the JSONUtils tool class time zone problem #10282
* [Fix] Fix the JSONUtils tool class time zone problem #10282
* Update JSONUtils.java
remove unnessnary log
* Update JSONUtilsTest.java
add timezone import
Co-authored-by: xiangzihao <460888207@qq.com>
2 years ago
ad2646ff1f
Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe ( #10479 )
...
* Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe
2 years ago
f3c647d45e
[Improvement][Api] Interface and entity optimization to improve code scalability ( #10466 ). ( #10468 )
...
* [Improvement][Api] Interface and entity optimization to improve code scalability (#10466 ).
* [Improvement][Api] Interface and entity optimization to improve code scalability (#10466 ).
* [Improvement][Api] Interface and entity optimization to improve code scalability (#10466 ).
* [Improvement][Api] Interface and entity optimization to improve code scalability (#10466 ).
* fix ut timezone.
Co-authored-by: houshitao <shitaohou@163.com>
2 years ago
d16e0ddb2f
[Bug-10394] [Update Possword] Change password shows error ( #10437 )
...
* Add a status code for password verification length to improve user interaction
* Modify the length and null value, verify them separately and return
* Modify the length and null value, verify them separately and return
* remove the outer not empty validate and Modify judgment conditions
* add the outer not empty validate and remove inter empty
Co-authored-by: syyang <730122+syyang99@user.noreply.gitee.com>
2 years ago
ff065d8e5b
[Feature][Improvement] Support multi cluster environments - k8s type ( #10096 )
...
* service code
* [Feature][UI] Add front-end for cluster manage
* fix e2e
* remove comment on cluster controller
* doc
* img
* setting e2e.yaml
* test
* rerun e2e
* fix bug from comment
* Update index.tsx
use Nspace instead of css.
* Update index.tsx
Remove the style.
* Delete index.module.scss
Remove the useless file.
Co-authored-by: qianl4 <qianl4@cicso.com>
Co-authored-by: William Tong <weitong@cisco.com>
Co-authored-by: Amy0104 <97265214+Amy0104@users.noreply.github.com>
2 years ago
f6fea06f10
[Improve] Enhance complement function transformation ( #10376 )
2 years ago
e21d7b1551
[Feature][metrics] Add master, worker metrics ( #10326 )
...
* Add mater metrics
* fix UT
* Add url to mysql profile
* Add worker metrics
* Update grafana config
* Add system metrics doc
* Add process failover counter
* Add metrics image
* Change jpg to png
* Add command insert metrics
* Fix UT
* Revert UT
2 years ago
3ecbee3885
Optimize MasterServer, add MasterRPCService ( #10371 )
...
* Optimize MasterServer, avoid NPE
2 years ago
b1fb17119b
[Feature-6758][Task] Add limit resource usage for tasks base on cgroup ( #10373 )
2 years ago
67c4f2b259
[Feature][Monitoring] Added disk available capacity monitoring ( #10287 )
...
* Added disk available capacity monitoring
* resolve conflict
Added disk available capacity monitoring
* resolve conflict
Added disk available capacity monitoring
* resolve conflict
Added disk available capacity monitoring
2 years ago
xiangzihao
Kengo Seki
Tq
Wenjun Ruan
kezhenxu94
juzimao
fuchanghai
caishunfeng
zhuxt2015
Sheldon
liyangyang
旺阳
Eric Gao
Jiajie Zhong
陈家名
pinkhello
qianli2022
WangJPLeo
wjf
guodong
hstdream
syyangs799
retime123