feffd972b5
[Feature][JsonSplit] Fix task logger path ( #5331 )
...
* update SnowFlake
* update processDefinite from processInstance
* update processDefinite from processInstance
* Fix task logger path
Co-authored-by: JinyLeeChina <297062848@qq.com>
3 years ago
49b234609a
update SnowFlake ( #5321 )
...
Co-authored-by: JinyLeeChina <297062848@qq.com>
3 years ago
513eb76919
[Improvement-5147][Service]This judgment always returns true ( #5156 )
...
* this expression which always evaluates "true"
* delete unused variables
* LogClientService add AutoCloseable
* reformat code
* reformat code
* reformat code
* add log in ProcessService
* reformat code
* Optimize the code
* Optimize the code
* [improvement]modify all methods that refer to LogClientService in the code.
* delete unused code
3 years ago
83519bdfc3
[Improvement][Server] Optimize log display ( #5311 )
3 years ago
f7a06a033d
[Feature][JsonSplit] Fix master/worker about shellTask ( #5300 )
...
* Fix quoted bug about processDefineId of processInstance
* Fix master/worker about shellTask
* codeStyle
* codeStyle
* update taskType
* codeStyle
* update timeoutNotifyStrategy
* codeStyle
Co-authored-by: JinyLeeChina <297062848@qq.com>
3 years ago
e39409a412
[Improvement-5275][Server] Fix wrong excludeFilters and move zk/ZKMasterClient and registry/ServerNodeManager into master ( #5278 )
...
* [Improvement-5275][Server] Move zk/ZKMasterClient and registry/ServerNodeManager into master
* [Improvement-5275][Server] Fix code smells
* [Improvement-5275][Server] Add ZKMasterClientTest
3 years ago
1c1f6663a6
[Improvement][WorkerGroup] Optimize and reduce database and zookeeper queries and replace worker address input with multiple-select ( #5182 )
...
* [Improvement][Server] Optimize and reduce database and ZooKeeper query
* [Improvement][WorkerGroup] Replace worker address input with multiple-select
* [Improvement][WorkerGroup] Fix unit test
* [Improvement][API&UI] Optimize long host to short host and long host display in k8s
3 years ago
cc7a4446f5
[Improvement][Task] DataX Node doesn't Support Clickhouse as datasource ( #5180 ) ( #5243 )
...
* [Improvement][Module Name] DataX Node doesn't Support Clickhouse as the datasource (#5180 )
* add ut
* add ut
* resolve code smell
3 years ago
645847c096
Fix quoted bug about processDefineId of processInstance ( #5263 )
...
Co-authored-by: JinyLeeChina <297062848@qq.com>
3 years ago
73f5a8e0a7
[Feature][JsonSplit] Fix master/worker server ( #5253 )
...
* Fix dependency failure
* Fix taskInstance
* task instance list page
* code review
* fix logger path
* Fix master/worker server
Co-authored-by: JinyLeeChina <297062848@qq.com>
3 years ago
31188ffca0
[Feature][JsonSplit] Fix taskInstance ( #5246 )
...
* Fix dependency failure
* Fix taskInstance
* task instance list page
* code review
* fix logger path
Co-authored-by: JinyLeeChina <297062848@qq.com>
3 years ago
4a1303b625
[Improvement][Worker] Kill task error( #5199 ) ( #5212 )
3 years ago
a8f2894b79
[Feature][JsonSplit] Fix master/processInstance bug ( #5206 )
...
* transform taskCode from long to string
* fix process bug
* code review
* code review
* code review
* Fix master/processInstance bug
Co-authored-by: JinyLeeChina <297062848@qq.com>
3 years ago
86ce8f133a
[JsonSplit-4417][master/worker/api]json split- remove json in task instance. ( #5160 )
...
* [json split] refactor json in task instance
* add task_pramas in task_instance
* feature #4417 refactor json split in task instance
* code style
* code style
* code style
* code style
* code style
* update
* update
* update unit test
* update
3 years ago
97fb08f0ce
[Improvement][LogServer] Thread pool parameter adjustment #5161 ( #5163 )
...
* [Improvement][LogServer] Thread pool parameter adjustment #5161
* change log request processor thread num 2 * cpuCores + 1
3 years ago
dd36da8d1f
[Feature][JsonSplit] fix process bug ( #5193 )
...
* transform taskCode from long to string
* fix process bug
* code review
* code review
* code review
Co-authored-by: JinyLeeChina <297062848@qq.com>
3 years ago
e53369318b
[Fix-4721][worker]The shell background starts the YARN task scenario, and the kill function is abnormal ( #4722 )
...
* fix the shell starts the yarn task in the background
* update StringUtils code style.
* solve code smell.
* add method comment in StringUtils class.
* update AlertGroupMapper code style.
* update AlertGroupMapperTest
* update sql script test.
3 years ago
a467fffd7b
[Feature][JsonSplit] fix taskId in json ( #5165 )
...
* modify checkDAGRing and ProcessService method
* merge
* modify dagRing
* modify process instance for project home page
* fix save process bug
* codeStyle
* Fix logical bug in saving process definition
* codeSytle
* Fix bug in interface of queryProcessDefinitionList
* codeSytle
* Fix api bug"
* fix taskId in processDefinitionJson
* fix json taskId
* codeStyle
Co-authored-by: JinyLeeChina <297062848@qq.com>
3 years ago
a6a1b94df4
[Feature-#5128]Support turning off sudo permissions ( #5129 )
...
* [Feature-#5128]Support turning off sudo permissions
* [Feature-#5128] without sudo permission do not create tenant user
3 years ago
9c04396218
[Bug-5132][server] stop the take immediately if an exception occurs in the taskExecuteThread ( #5133 )
...
* [DS-5132][fix] stop the take immediately if an exception occurs in the taskExecuteThread
- let softkill() method return true when the process is stopped,else return false;
3 years ago
573252f3e1
[Bug][Server] DB transaction failure #5150 ( #5152 )
3 years ago
87ed189ef5
[Improvement] print exception when task execute ( #5159 )
3 years ago
492b318bd3
[Fix][Server] Fix clear task execute path is related to master ( #5123 )
3 years ago
a8b47e1d4f
[Bug][Master] Typo in MasterConfig #5141 ( #5142 )
3 years ago
0d065c0712
[Feature-5062][*] Create/Edit/Delete Worker Group in Web UI ( #5113 )
...
* [Improvement][Server] Rename MASTER_PREFIX to MASTER_TYPE and WORKER_PREFIX to WORKER_TYPE
* [Feature][Api&Dao] Support api and dao of worker group
* [Feature][UI] Support to create, edit and delete worker group
* [Feature][API&UI] Support work groups from database and zookeeper to display together in worker group management
* [Feature][API&Dao] Support long ip list and check request parameters when saving worker group
* [Feature][*] Refactor ipList to addrList
* [Feature][Server] Support host manager to select worker host by worker group in database
* [Improvement][Server] Improve the config of master and worker
* [Improvement][*] Rename zkRegistered to systemDefault
* [Improvement][Test] Fix unit test and improve ui
* [Improvement][Server] Improve getServerMaps
Co-authored-by: dailidong <dailidong66@gmail.com>
3 years ago
ba039dc252
[Improvement][dao] Resource features SQL slow query optimization. ( #4947 )
...
* Resourse features SQL slow query optimization.
* update resource mapper queryResourcePaging method.
* split resource and resource_uesr sql statement.
* determine the size of the collection.
* update process service class code style.
* update ResourceUserMapper java class code style.
* add resourceUserMapper in the test config file.
* solve code smell
* split paging query join statement.
* update ResourceUserMapper class.
* remove sql join statement.
* update resource test class code style.
* update ResourcesServiceTest test class.
* rerun ut.
* update user service test class.
* solve query resource list method code smell.
3 years ago
8ac72e80e6
[Feature-5087][SqlTask] Add a switch to send mail and print head logs in SqlTask ( #5088 )
...
* [Feature-5087][SqlTask] Add the switch to send mail and print head logs in SqlTask
* [Feature-5087][SqlTask] Add SqlParametersTest
3 years ago
f94cfc620d
[Feature][Worker] Add a configuration item to set whether the tenant is automatically created on Worker ( #5007 )
3 years ago
29d42fd92d
[Feature-#3805][server-master] Task parameter transfer ( #5077 )
...
* fix out param format bug
Co-authored-by: wangxj <wangxj31>
3 years ago
5856a12855
[Improvement-4984][Worker] Refactor and Improve worker load balance ( #4996 )
...
* [Improvement][Remote] Improve unit tests
* [Improvement][Worker] Improve worker load balance
* [Improvement][Worker] Fix code smells
* [Improvement][Worker] Rename weight to hostWeight
3 years ago
1cd62b4a5e
[Improvement][Datax] Re-assign DATAX_HOME to /opt/soft/datax instead of /opt/soft/datax/bin/datax.py ( #5019 )
3 years ago
0ba120e8ab
Fix shell task ${setValue(key,value)} is not working ( #5067 )
...
Co-authored-by: dalekliuhan <dalekliuhan@didiglobal.com>
3 years ago
6bf5031f45
[FIX] fix MySQL datasource jdbc connect parameters ( #4851 )
...
* [FIX-CVE-2020-11974] fix MySQLDataSource Security
* remove useless code
* add ut
* A more reasonable way
* code style
* fix test error
* code style
* update exception msg
3 years ago
2d8c4ec86b
[Improvement][Worker-Task] Python Command ( #5036 )
...
* create the interpreter for python command
* Update README.md
Co-authored-by: zhouhong <zhouhong@zhangyue.com>
3 years ago
a3153b5e44
[Bug][Master] Fix First master fault tolerance when startup ( #5045 )
...
* [Fix-#5044] Fix First master fault tolerance when startup
* [Fix-#5044] Fix spell error
3 years ago
ae608e024a
[Fix-5037][Server] Fix that both the master and the worker is hanging after restarting and stopping ( #5038 )
...
* [Fix-5037][Server] Fix that both the master and the worker is hanging after restarting and stopping
* [Improvement][*] Replace commons.lang.StringUtils with common.utils.StringUtils
3 years ago
ea6b1de120
[Improvement][worker] Optimize KillYarnJob (4939) ( #4943 )
3 years ago
beb5adb985
[Feature][MR] Support mapreduce name ( #5029 )
3 years ago
28dfde7002
[Fix-4862][Server] Kill yarn application command won't be execute whe… ( #4936 )
...
* [Fix-4862][Server] Kill yarn application command won't be execute when kill error
3 years ago
fd4b59ad6c
[Improvement][Worker] Rename worker.weight to worker.host.weight ( #4997 ) ( #5014 )
3 years ago
e74932cfc2
[Feature-4960][Spark] Add name input for Spark and improve Spark & MR args ( #4959 )
...
* [Improvement][Spark] Improve spark args and add name input
* [Improvement][MR] Improve map reduce args
* [Improvement][Spark] Fix check style
* [Improvement][Spark] Fix check style for spark, flink and mr
3 years ago
89c9b51740
[Improvement]Add comments to ZKMasterClient.java( #5005 )
3 years ago
c3f43a1924
[Improvement-#4972] add kerberos init before yarn kill ( #4973 )
...
* [Improvement-#4972] add kerberos init before yarn kill
* [Improvement-#4972] fix code smell
* [Improvement-#4972] add ut
3 years ago
39a9c0c392
[Feature][jsonspilt]modify process instance for project home page ( #4991 )
...
* modify checkDAGRing and ProcessService method
* merge
* modify dagRing
* modify process instance for project home page
Co-authored-by: JinyLeeChina <297062848@qq.com>
3 years ago
86600d1328
AlertClientService null pointer exception in SqlTask. ( #4983 )
3 years ago
8c68e2c60b
[Fix][Worker] Fix worker cannot start and alertClientService NullPointerException ( #4956 )
...
* [Fix][Worker] Fix worker cannot start
* [Fix][Worker] Fix alertClientService NullPointerException
* [Fix][Worker] Improve ut coverage
* [Improvement][CI] Improve workflow ci
3 years ago
24fda3fbb1
[Feature][Flink] Support parallelism input ( #4937 )
...
* [Feature][Flink] Support parallelism input
* [Feature][Flink] Fix check style
* [Feature][Flink] Remove FLINK_DETACH
3 years ago
5bfe3fca67
[Fix][SqlTask] fix hive sqltask invalid connection parameters. ( #4917 )
...
* fix hive sqltask invalid connection parameters.
* add setConnParams test method.
* update base datasource test class code style.
* add BaseDataSourceTest in the pom xml.
* update hive conf list jdbc url stitching.
* update hive datasource test class.
3 years ago
ccd8aaebab
[Improvement][Server] Improve dependent task unit tests ( #4911 )
3 years ago
f6088f277e
Optimize code comments ( #4933 )
...
Co-authored-by: mengyun <mengyun@pinduoduo.com>
3 years ago
JinyLeeChina
luoyuan
Shiwen Cheng
wenjun
bao liang
zhuangchong
guohaozhang
xiaojingXU
wangxj3
mrproliu
Kirs
游侠
dddyszy