caishunfeng
dedff70f90
[Improvement] Merge spi.utils into common.utils ( #12502 )
...
* improve JSONUtil
* merge spi.utils into common.utils
* split constants && remove some copy utils method
* remove StringUtils
2 years ago
Eric Gao
d46e4659d9
[Improvement][Style] Fix remaining style errors and wildcard imports ( #12412 )
...
* Fix remaining style errors and wildcard imports
* Upgrade spotless to the latest stable version, exclude .github folder from spotless check and fix dead link
* Fix spotless matching scope of markdown files
2 years ago
Yann Ann
b357781df2
[Migrate][Test] Migrate junit4 -> junit5 test cases in common, service, and spi modules ( #12363 )
2 years ago
kezhenxu94
db1d319159
Reorganize some classes in common module, remove duplicate classes ( #12321 )
2 years ago
Eric Gao
9ab79e064c
[Improvement][Test] Fully remove the usage of powermock from the whole project ( #12244 )
...
* Fully remove the usage of powermock from the whole project
* Upgrade org.reflections to 0.10.12
2 years ago
Yann Ann
c87f6d89a4
[fix#12195] Change date to timestamp to support cross diff time zones ( #12239 )
...
* fix DS support cross time zone, use timestamp replace date
2 years ago
zhuxt2015
e6832220c3
[Improvement][SQL Task]use default sql segment separator ( #10869 )
2 years ago
Eric Gao
66958b91d7
[Feature][Resource Center] Add support for Alibaba Cloud OSS as storage of resource center ( #11708 )
...
* Add support for Alibaba Cloud OSS as storage of resource center
* Add dependency licenses
* Refactor code, add unit tests and related docs
2 years ago
ZhaoGuodong
86291bb5e6
[Feature-10273][Datasource] Add dataSource for AWS Athena ( #11157 )
2 years ago
Wenjun Ruan
b96d69701a
[Improvement-#11613] Add spi priority factory ( #11614 )
...
* Add spi priority factory
* Add doc
* Add override log
* Use lombok
* Add comment
2 years ago
Wenjun Ruan
71cf7e1baa
[Fix-10918] Close datasource when expire from guava cache ( #11120 )
...
* Close datasource when expire from guava cache
* Remove duplicate datasource in HiveDataSourceClient
2 years ago
Wenjun Ruan
5e9c7dad23
Add dolphinscheduler-bom to manage the dependency version ( #11025 )
2 years ago
Eric Gao
71f0168510
[Feature] Enable users to create python env from requirements.txt ( #10658 )
2 years ago
HomminLee
dafed572d1
[Fix][dolphinscheduler-spi] json parse support LocalDateTime ( #10440 )
...
* JSONUtils support LocalDateTime
* remove extraneous unit tests
* remove unused classes import.
Co-authored-by: HomminLee <lihongmin@ysstech.com>
2 years ago
kezhenxu94
d80cf21456
Clean up unused dependencies and packaging issues ( #9944 )
2 years ago
Tq
592e433b1e
[Feature][Datasource]replace DataSourceController API with string JSON and cast to DTO ( #10276 )
...
* replace DataSourceController API with string JSON and cast to DTO
* add DataSourceProcessorManager and DataSourceProcessorProvider to datasource processor functions
2 years ago
WangJPLeo
31cd1b5e61
Serial wait for subsequent execution ( #9847 )
2 years ago
Eric Gao
fd6b43bc81
[Dev] Switch version in pom.xml to dev-SNAPSHOT ( #9223 ) ( #9299 )
2 years ago
caishunfeng
263791a63e
[Bug-9137][Server] fix timezone init ( #9219 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
2 years ago
Kerwin
a1bef43981
[Feature-8579][Task] Task back-end plugin optimization, new plugins only need to modify the plugin's own module ( #8584 )
...
* Task plug-in optimization, task-related classes are migrated to the task plugin module
* remove tasktype enum class.
* fix code style.
* fix code style
* fix code style
* fix ProcessServiceTest.testUpdateResourceInfo test class
* fix code style
* fix test task plugin manager check parameters null
* fix test ut
* update aws license
* change TaskTimeoutStrategy class
3 years ago
ronyang1985
7b66b66e8f
[Feature-8527][Datasource] Add dataSource for Amazon Redshift ( #8564 )
...
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
3 years ago
ronyang1985
20ee9a37fe
[Feature-8485][task] Add EMR task plugin to support submit task to AWS EMR ( #8503 )
...
Co-authored-by: Wenjun Ruan <wenjun@apache.org>
Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
3 years ago
yimaixinchen
a72742b9de
remove unuse import jsonFormat ( #8548 )
3 years ago
Molin Wang
e174d7a40e
[Fix-8394][Server]fix when the data is supplemented business time still date+1 ( #8532 )
3 years ago
caishunfeng
84f2cfec39
[Feature][Server] Custom timezone - Add timezone handler unified ( #8477 )
...
* date convert of timezone
* remove @JsonFormat
* add unit test
* fix time preview in scheduler
* optimization & add env config
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
caishunfeng
f2b9796bc2
[Bug-8110][WorkerServer] kill all running task before worker stop ( #8490 )
...
* kill all running task before worker stop
* kill local process
* remove kill yarn job
* adjust the orders of close
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
Ivan0626
60ddede6a6
fixbug: varPool value is error ( #7724 )
3 years ago
cheney
703d790415
[Bug] [Parameter Context] The preparameter OUT is null ( #8315 )
...
* fix:param context prop value is null
3 years ago
zixi0825
4ba6d667fc
[Feature][DataQuality] Add Data quality Module #4283 ( #6718 )
...
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
* fix Constants bug
* remove the unused class
* add unit test
* fix code style error
* add unit test
* refactor data quality common entity
* fix code style error
* add unit test
* close e2e test
* fix code smell bug
* modify dataquality enum value to 14 in TaskType
* add data qualtiy task
* update
* add getDatasourceOptions interface
* fix checkstyle
* close e2e test
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* add unit test and alter dataquality task result
* fix checkstyle
* fix process service test error
* add unit test and fix code smells
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* change execute sql type name
* revert ui pom.xml
* fix data quality task error
* fix checkstyle error
* fix dq task src_connector_type ui select bug
* fix spark rw postgresql bug
* change mysql driver scope
* fix form-create json bug
* fix code smell
* fix DolphinException Bug
* fix ui validate rule and Alert title
* fix target connection param bug
* fix threshold validate change
* add rule input entry index
* change statistic_comparison_check logic
* remove check type change
* add DateExpressionReplaceUtil
* fix null point expetion
* fix null point expetion
* fix test error
* add more sql driver
* fix test error and remove DateExprReplaceUtil
* add get datasource tables and columns
* add get datasource tables and columns
* remove hive-jdbc in pom.xml
* fix code smells
* update sql
* change the pom.xml
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
* update sql
* [Feature][DataQuality] Add data quality task ui (#5054 )
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* fix dq task src_connector_type ui select bug
* fix threshold validate change
* remove check type change
* add get datasource tables and columns
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality module (#4830 )
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* update
* close e2e test
* fix spark rw postgresql bug
* change mysql driver scope
* add more sql driver
* remove hive-jdbc in pom.xml
* change the pom.xml
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality task backend (#4883 )
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* refactor data_quality_module
* add header license
* data quality module refactor
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix code smell
* fix check style
* fix unit test error
* task statistics value add unique code
* fix unit test error
* fix checkstyle error
* fix checkstyle
* fix security hotspot
* fix unit test error
* fix security hotspot
* fix check
* add data quality task error handling
* fix unit test error
* add unit test
* add unit test
* optimize data quality result alert
* fix unit test
* fix sql script error
* fix bug
* update sql script
* fix checkstyle
* add license
* fix checkstyle
* fix checkstyle
* fix unit test
* add jacoco dependencies
* fix unit test
* fix unit test
* add jacoco dependencies
* add unit test
* add unit test
* add license
* fix checkstyle
* fix pom
* fix checkstyle
* fix checkstyle
* merge dev
* fix ui error
* fix pom error
* fix pom error
* fix test error
* fix test error
* mssql-jdbc exclude azure-keyvault
* fix test error
* merge dev and add unit test
* add notes
* rollback the CollectionUtils
* fix
* update sql
* fix
* fix
* fix query rule page error
* change dq.jar path
* fix sql error
* fix ui error
* fix(dq): jar path&task enum description
* add notes on DataQualityApplication
* fix dq result jump error
* fix(ui): page condition
* feat(ui): add show error output path
* change version
* remove all chinese word in sql
* merge
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
3 years ago
Kerwin
103ccb01da
change version to 2.0.4-SNAPSHOT ( #8188 )
3 years ago
mask
898727709f
[fix-7671][plugin] Supports whether SQL is placed in the same session for configuration(fix-7671) ( #7675 )
...
-Supports whether SQL is placed in the same session for configuration
3 years ago
wind
e4cdd82b7c
[Improvement][TaskLog] Unified task log ( #7831 )
...
* task logger
* update
* log format
* remove master config task-logger
* udpate log level
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
mask
8e62dd8246
[Fix-7291][datasource] Compatible with different versions of MySQL data source loading ( #7291 ) ( #7305 )
3 years ago
kezhenxu94
67cc260d52
Split the components into individual package, refactor configurations to use Spring properties ( #7160 )
...
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
3 years ago
Kerwin
802fc498b5
[Fix-#6605][Common] Fix the situation where the obtained string property is empty. ( #6661 )
...
* Fix the situation where the obtained string property is empty.
* recovery test
* update code style.
* megre dev.
3 years ago
yihong
861aaaf971
[FIX-#6505][Dao] upgrade the MySQL driver package for building MySQL jdbcUrl ( #6708 )
...
* replace com.mysql.jdbc.Driver everywhere with com.mysql.cj.jdbc.Driver
3 years ago
kezhenxu94
4114cb07f6
Add actuator and enable metrics, add Grafana configs ( #6840 )
3 years ago
kezhenxu94
11844415b0
Refactor datasource and task plugins and simplify its usage ( #6767 )
3 years ago
mask
0aea5ae11a
[feature-6071][plugin] add datasource plugin. ( #6071 ) ( #6361 )
...
* [feature-6071][plugin] add datasource plugin. (#6071 )
* remove the dependency of datasource in common and task-plugin module.
* the new datasource-api module provides data source connection acquisition.
* add sqlserver data source plug-in.
* add ds license
* fix check style
* update spi pom
* frx conflict resolution
* fix check style
* fix datasource ut
* fix code check style
* fix code check style
* fix datasource ut
* add pg hive mysql oracle clickhouse db2 datasource
* fix datasource package
* update datasource pom
* update datasource pom
* update datasource pom
* update datasource pom
* update pom
* update pom dep
* update pom dep
* fix license
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* add oracle package
* modify comments
* conflict resolution
* rename log comment
* rename log comment
3 years ago
kezhenxu94
05bae0b226
Refactor alert plugins and simplify its usage ( #6723 )
3 years ago
kezhenxu94
5400abc74f
Refactor registry plugin and simplify its usage ( #6712 )
3 years ago
kezhenxu94
5e923971bf
Migrate http to https in maven address ( #6698 )
3 years ago
wind
72daba58cd
[DS-6694][Master] Reduce the refresh resource interval of LowerWeight ( #6695 )
...
* [DS-6694][Master] Reduce the refresh resource interval of LowerWeight
* add server node update event handle
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
LiuBodong
71047db39e
[FIX][Standalone Server] check if maven settings.xml not exists ( #6691 )
3 years ago
LiuBodong
61a8726ccc
[Fix][Standalone Server] fix plugin load failed when local maven repository path is not default value ( #6625 ) ( #6626 )
...
* [Fix][Standalone Server] fix plugin load failed when local maven repository path is not default value
3 years ago
jon-qj
86c85114b2
make alert instance page display more friendly ( #6635 )
...
* make alert instance page display more friendly
* Update EmailAlertChannelTest.java
3 years ago
mask
9b1984c244
[fix-6570][task] add sql task udf resource load ( #6570 ) ( #6571 )
...
* [fix-6570][task] add sql task udf resource load (#6570 )
-add sql task udf resource load
-Fix UdfFuncRequest serialization
* fix checkstyle
* add defaultFS in sqlTaskContext
* fix checkstyle
3 years ago
Hua Jiang
a80fca70f7
[Fix-6497][Server] Fix the missing code about the feature of supporting multiple environments. ( #6509 )
3 years ago
kyoty
7dbea9218d
bug fix ( #6381 )
...
The newly introduced spi can't work in windows os.
3 years ago
Kirs
67508d8d1b
update project version to 2.0.0-SNAPSHOT ( #6383 )
3 years ago