Ivan0626
60ddede6a6
fixbug: varPool value is error ( #7724 )
3 years ago
ShuiMuNianHuaLP
806333a11c
fix sudo.enable=false Is invalid ( #8388 )
3 years ago
zixi0825
4ba6d667fc
[Feature][DataQuality] Add Data quality Module #4283 ( #6718 )
...
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
* fix Constants bug
* remove the unused class
* add unit test
* fix code style error
* add unit test
* refactor data quality common entity
* fix code style error
* add unit test
* close e2e test
* fix code smell bug
* modify dataquality enum value to 14 in TaskType
* add data qualtiy task
* update
* add getDatasourceOptions interface
* fix checkstyle
* close e2e test
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* add unit test and alter dataquality task result
* fix checkstyle
* fix process service test error
* add unit test and fix code smells
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* change execute sql type name
* revert ui pom.xml
* fix data quality task error
* fix checkstyle error
* fix dq task src_connector_type ui select bug
* fix spark rw postgresql bug
* change mysql driver scope
* fix form-create json bug
* fix code smell
* fix DolphinException Bug
* fix ui validate rule and Alert title
* fix target connection param bug
* fix threshold validate change
* add rule input entry index
* change statistic_comparison_check logic
* remove check type change
* add DateExpressionReplaceUtil
* fix null point expetion
* fix null point expetion
* fix test error
* add more sql driver
* fix test error and remove DateExprReplaceUtil
* add get datasource tables and columns
* add get datasource tables and columns
* remove hive-jdbc in pom.xml
* fix code smells
* update sql
* change the pom.xml
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
* update sql
* [Feature][DataQuality] Add data quality task ui (#5054 )
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* fix dq task src_connector_type ui select bug
* fix threshold validate change
* remove check type change
* add get datasource tables and columns
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality module (#4830 )
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* update
* close e2e test
* fix spark rw postgresql bug
* change mysql driver scope
* add more sql driver
* remove hive-jdbc in pom.xml
* change the pom.xml
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality task backend (#4883 )
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* refactor data_quality_module
* add header license
* data quality module refactor
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix code smell
* fix check style
* fix unit test error
* task statistics value add unique code
* fix unit test error
* fix checkstyle error
* fix checkstyle
* fix security hotspot
* fix unit test error
* fix security hotspot
* fix check
* add data quality task error handling
* fix unit test error
* add unit test
* add unit test
* optimize data quality result alert
* fix unit test
* fix sql script error
* fix bug
* update sql script
* fix checkstyle
* add license
* fix checkstyle
* fix checkstyle
* fix unit test
* add jacoco dependencies
* fix unit test
* fix unit test
* add jacoco dependencies
* add unit test
* add unit test
* add license
* fix checkstyle
* fix pom
* fix checkstyle
* fix checkstyle
* merge dev
* fix ui error
* fix pom error
* fix pom error
* fix test error
* fix test error
* mssql-jdbc exclude azure-keyvault
* fix test error
* merge dev and add unit test
* add notes
* rollback the CollectionUtils
* fix
* update sql
* fix
* fix
* fix query rule page error
* change dq.jar path
* fix sql error
* fix ui error
* fix(dq): jar path&task enum description
* add notes on DataQualityApplication
* fix dq result jump error
* fix(ui): page condition
* feat(ui): add show error output path
* change version
* remove all chinese word in sql
* merge
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
3 years ago
Wenjun Ruan
a1cc2fd11a
[task] Python task support PYTHON_HOME ( #8196 )
3 years ago
Kerwin
103ccb01da
change version to 2.0.4-SNAPSHOT ( #8188 )
3 years ago
Shiwen Cheng
0057cf3fe7
[Improvement][task-plugin] Remove unreasonable log ( #8115 )
3 years ago
seagle
f1a0215978
[bug] Fix datax task type error parse clickhouse reader sql ( #8091 )
3 years ago
wind
e4cdd82b7c
[Improvement][TaskLog] Unified task log ( #7831 )
...
* task logger
* update
* log format
* remove master config task-logger
* udpate log level
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
Kerwin
5172862a0a
Add seatunnel task plugin ( #7131 )
3 years ago
J·Y
a4168ae24e
[Improvement] The spark version is associated with the spark command ( #7544 )
...
* [Improvement] The spark version is associated with the spark command
3 years ago
xiangzihao
c58dbefaa5
[Fix-7459][Sqoop] running error in parallel ( #7752 )
...
* fix bug_7459
* fix bug_7459
* fix bug_7459
* fix bug_7459
* fix bug_7459
Co-authored-by: SbloodyS <sbloodys@qq.com>
3 years ago
wangxj3
cc77963522
[FIX-#6727][worker-server] Fix procedure params bug ( #7680 )
...
* fix Procedure param error
* code style
Co-authored-by: wangxj <wangxj31>
3 years ago
J·Y
6ae46f2c1b
[Improvement] Merge redundant codes ( #7550 )
3 years ago
Zonglei Dong
7e0010de35
[Refactor] Rename class file, unified word spelling. ( #7261 )
3 years ago
kezhenxu94
67cc260d52
Split the components into individual package, refactor configurations to use Spring properties ( #7160 )
...
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
3 years ago
Zonglei Dong
0b9792db76
[Refactor]Rename class file, DatasourceUtil to DataSourceUtils. ( #7181 )
...
* Rename class file, DatasourceUtil to DataSourceUtils.
3 years ago
张俊杰
463fc76bb7
[Fix-6809][dolphinscheduler-task-sql] Fix sql task can't send alert ( #7089 )
3 years ago
kezhenxu94
2ed3c0c3a7
Recover unit tests that have not been run for some time ( #7054 )
3 years ago
kezhenxu94
4114cb07f6
Add actuator and enable metrics, add Grafana configs ( #6840 )
3 years ago
kezhenxu94
11844415b0
Refactor datasource and task plugins and simplify its usage ( #6767 )
3 years ago
mask
0aea5ae11a
[feature-6071][plugin] add datasource plugin. ( #6071 ) ( #6361 )
...
* [feature-6071][plugin] add datasource plugin. (#6071 )
* remove the dependency of datasource in common and task-plugin module.
* the new datasource-api module provides data source connection acquisition.
* add sqlserver data source plug-in.
* add ds license
* fix check style
* update spi pom
* frx conflict resolution
* fix check style
* fix datasource ut
* fix code check style
* fix code check style
* fix datasource ut
* add pg hive mysql oracle clickhouse db2 datasource
* fix datasource package
* update datasource pom
* update datasource pom
* update datasource pom
* update datasource pom
* update pom
* update pom dep
* update pom dep
* fix license
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* fix ut
* add oracle package
* modify comments
* conflict resolution
* rename log comment
* rename log comment
3 years ago
mask
9b1984c244
[fix-6570][task] add sql task udf resource load ( #6570 ) ( #6571 )
...
* [fix-6570][task] add sql task udf resource load (#6570 )
-add sql task udf resource load
-Fix UdfFuncRequest serialization
* fix checkstyle
* add defaultFS in sqlTaskContext
* fix checkstyle
3 years ago
Kirs
09657d8576
[ #6337 ]When the SQL result reaches the limit value, increase the log result prompt ( #6528 )
3 years ago
Hua Jiang
8200bc1452
[Fix-6529] Improve the python task to solve the issue about getting the environment variables. ( #6531 )
...
* modify config
* improve python task
* improve python task
3 years ago
Kirs
bddf3e6f2c
[FIX-#6514]Task Class Load Err ( #6513 )
...
* [FIX-#6412]Task Class Load Err
* Task Class Load Err
3 years ago
Hua Jiang
a80fca70f7
[Fix-6497][Server] Fix the missing code about the feature of supporting multiple environments. ( #6509 )
3 years ago
百岁
d284e44d52
Rename tis to pigeon from backend layer ( #6437 )
...
* add mock getTaskLogName() for UT test pass
* make task-tis rename to task-pigeon
* rename module name 'tis' to pigeon
* rename module name 'tis' to pigeon,make sanity check pass
* remove execute console.log()
3 years ago
Kirs
1843e986f5
[BUG][Task-API]load codec class fail ( #6411 )
3 years ago
Kirs
67508d8d1b
update project version to 2.0.0-SNAPSHOT ( #6383 )
3 years ago
mask
a62a7a7216
[Fix][Task] sql limit param no default value ( #6337 ) ( #6387 )
...
* add default query sql limit
3 years ago
百岁
545173faa3
add mock getTaskLogName() for UT test pass ( #6369 )
3 years ago
百岁
34352c1bf8
Enhance using experience of DataX by introduce TIS ( #6229 )
...
* Enhance using experience of DataX by introduce TIS [Feature-5992]
* Enhance using experience of DataX by introduce TIS [Feature-5992]
* fix stylecheck error
* make testCase pass
* make dev testCase pass
* add new java dependency Java-WebSocket
* modfiy TISParameters.java avoid to name confliction
* add InterruptedException checking
* reAdd async-http-client annotation
* in order to staisfy the coverage degree add test which has been removed
* make testCase pass
* make testCase pass
* add jacoco dependency
* make code duplications be more lower
* add Java-WebSocket dependency in root pom
* remove useless code comment
* make tis http apply path and post body configurable ,the params save in config.properites
* Remove the dangerous instance of double-checked locking
3 years ago
Kirs
be158328fb
[Bug #6306 ][Worker] task log path generated error. ( #6338 )
3 years ago
caishunfeng
61c6461994
[BUG-6331][WorkerServer] fix preparedStatement auto close by try with resource ( #6334 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
Kirs
163a0696a4
[Bug #6306 ][Worker] task log path generated error. ( #6311 )
...
* [Bug #6306 ][Worker] task log path generated error.
* [Bug #6306 ][Worker] task log path generated error.
3 years ago
Kirs
9115062838
[Plugin][Task]Fix Task log may be lost ( #6294 )
3 years ago
Kirs
552fd17878
[Plugin][Task]Fix Yarn Task Kill Not valid ( #6293 )
3 years ago
Kirs
7e8febed4d
[Plugin][Task]Fix Sql Task NPE ( #6290 )
3 years ago
Kirs
fab4eae923
[SPI][Task]fix Task parse Err ( #6244 )
...
* [SPI][Task]fix Task parse Err
3 years ago
Kirs
fced3892ee
[Fix][Task] Wrong complement date ( #6009 ) ( #6186 )
...
* [Fix][Task] Wrong complement date (#6009 )
3 years ago
Kirs
b90152fc2e
[SPI][Task]add SQL Task ( #6237 )
...
Add Sql Task And Fix Task Parse Err
3 years ago
mask
259ec909d1
[Improvement-6198][task] remove Sparktaskchannelfactory duplicate declaration ( #6199 )
...
* remove Sparktaskchannelfactory duplicate declaration
3 years ago
Kirs
58b694a85c
[BUG][Plugin] Fix #6167 ( #6177 )
3 years ago
Martin Huang
0715be34d5
[Fix-6166][task-plugin]Fix Python Task Execution Error ( #6167 )
...
* fix python task execution error
* delete AbstractTask#setCommand
3 years ago
kezhenxu94
ea2a9dbcb2
Clean up third-party files that are not mentioned in LICENSE ( #6165 )
3 years ago
CalvinKirs
5b98ea0104
fix code style
3 years ago
CalvinKirs
d6d3980241
fix code style
3 years ago
CalvinKirs
ac8fafa343
fix code style
3 years ago
CalvinKirs
2dd5279388
Merge remote-tracking branch 'upstream/dev' into task-end
...
# Conflicts:
# .asf.yaml
# dolphinscheduler-alert/src/main/java/org/apache/dolphinscheduler/alert/utils/FuncUtils.java
# dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ExecutorController.java
# dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/SchedulerController.java
# dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java
# dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ProcessInstanceServiceImpl.java
# dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/controller/AlertGroupControllerTest.java
# dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/service/ExecutorService2Test.java
# dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/Constants.java
# dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/enums/StateEventType.java
# dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/Command.java
# dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/ErrorCommand.java
# dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/ProcessInstance.java
# dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/TaskDefinition.java
# dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ScheduleMapper.xml
# dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/TaskDefinitionLogMapper.xml
# dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/TaskDefinitionMapper.xml
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/ConditionsTaskExecThread.java
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/DependentTaskExecThread.java
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteThread.java
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/task/CommonTaskProcessor.java
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/task/ConditionTaskProcessor.java
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/task/DependentTaskProcessor.java
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/WorkerServer.java
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/ConditionsTaskExecThread.java
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/DependentTaskExecThread.java
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/MasterBaseTaskExecThread.java
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/MasterTaskExecThread.java
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/SubProcessTaskExecThread.java
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/TaskExecuteThread.java
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/WorkerManagerThread.java
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/task/python/PythonTask.java
# dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/task/sql/SqlTask.java
# dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/master/WorkflowExecuteThreadTest.java
# dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/worker/processor/TaskKillProcessorTest.java
# dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/worker/runner/TaskExecuteThreadTest.java
# dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/worker/runner/WorkerManagerThreadTest.java
# dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/worker/task/PythonCommandExecutorTest.java
# dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/worker/task/http/HttpTaskTest.java
# dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/worker/task/sql/SqlTaskTest.java
# dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/process/ProcessService.java
# dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/quartz/cron/CronUtils.java
# dolphinscheduler-service/src/test/java/org/apache/dolphinscheduler/service/process/ProcessServiceTest.java
# dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/DolphinSchedulerPlugin.java
# dolphinscheduler-standalone-server/pom.xml
# dolphinscheduler-standalone-server/src/main/java/org/apache/dolphinscheduler/server/StandaloneServer.java
# dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/ShellCommandExecutor.java
# dolphinscheduler-task-plugin/dolphinscheduler-task-procedure/src/main/java/org/apache/dolphinscheduler/plugin/task/procedure/ProcedureTaskChannelFactory.java
# dolphinscheduler-task-plugin/dolphinscheduler-task-python/src/main/java/org/apache/dolphinscheduler/plugin/task/python/PythonCommandExecutor.java
# dolphinscheduler-task-plugin/dolphinscheduler-task-shell/src/main/java/org/apache/dolphinscheduler/plugin/task/shell/ShellTask.java
# dolphinscheduler-task-plugin/dolphinscheduler-task-spark/src/main/java/org/apache/dolphinscheduler/plugin/task/spark/SparkTask.java
# dolphinscheduler-task-plugin/dolphinscheduler-task-sqoop/src/main/java/org/apache/dolphinscheduler/plugin/task/sqoop/SqoopTask.java
# dolphinscheduler-ui/package.json
# dolphinscheduler-ui/src/js/conf/home/pages/dag/_source/dag.scss
# dolphinscheduler-ui/src/js/conf/home/pages/dag/_source/dag.vue
# dolphinscheduler-ui/src/js/conf/home/pages/dag/_source/formModel/formModel.vue
# sql/dolphinscheduler_h2.sql
3 years ago
Kerwin
18047881d5
fix task plugin shell exec ( #6128 )
3 years ago