4ba6d667fc
[Feature][DataQuality] Add Data quality Module #4283 ( #6718 )
...
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
* fix Constants bug
* remove the unused class
* add unit test
* fix code style error
* add unit test
* refactor data quality common entity
* fix code style error
* add unit test
* close e2e test
* fix code smell bug
* modify dataquality enum value to 14 in TaskType
* add data qualtiy task
* update
* add getDatasourceOptions interface
* fix checkstyle
* close e2e test
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* add unit test and alter dataquality task result
* fix checkstyle
* fix process service test error
* add unit test and fix code smells
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* change execute sql type name
* revert ui pom.xml
* fix data quality task error
* fix checkstyle error
* fix dq task src_connector_type ui select bug
* fix spark rw postgresql bug
* change mysql driver scope
* fix form-create json bug
* fix code smell
* fix DolphinException Bug
* fix ui validate rule and Alert title
* fix target connection param bug
* fix threshold validate change
* add rule input entry index
* change statistic_comparison_check logic
* remove check type change
* add DateExpressionReplaceUtil
* fix null point expetion
* fix null point expetion
* fix test error
* add more sql driver
* fix test error and remove DateExprReplaceUtil
* add get datasource tables and columns
* add get datasource tables and columns
* remove hive-jdbc in pom.xml
* fix code smells
* update sql
* change the pom.xml
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
* update sql
* [Feature][DataQuality] Add data quality task ui (#5054 )
* add data quality task ui
* update skywalking-eyes SHA
* fix style
* fix eslint error
* fix eslint error
* test e2e
* fix dq task src_connector_type ui select bug
* fix threshold validate change
* remove check type change
* add get datasource tables and columns
* optimize multi_table_accuracy ui
* fix v-show error
* fix code smells
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality module (#4830 )
* add data quality module
* add license
* add package configuration in dist pom
* fix license and jar import bug
* replace apache/skywalking-eyes@9bd5feb SHA
* refacotr jbdc-connector and writer
* modify parameter name in HiveConnector
* fix checkstyle error
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix checkstyle error in dolphinschesuler-dist
* fix duplicate code bug
* fix code style bug
* fix code smells
* update
* close e2e test
* fix spark rw postgresql bug
* change mysql driver scope
* add more sql driver
* remove hive-jdbc in pom.xml
* change the pom.xml
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* [Feature][DataQuality] Add data quality task backend (#4883 )
* add dq relevant enums and parameter
* replace apache/skywalking-eyes@9bd5feb SHA
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
* refactor data_quality_module
* add header license
* data quality module refactor
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix checkstyle error
* fix unit test error
* fix code smell
* fix check style
* fix unit test error
* task statistics value add unique code
* fix unit test error
* fix checkstyle error
* fix checkstyle
* fix security hotspot
* fix unit test error
* fix security hotspot
* fix check
* add data quality task error handling
* fix unit test error
* add unit test
* add unit test
* optimize data quality result alert
* fix unit test
* fix sql script error
* fix bug
* update sql script
* fix checkstyle
* add license
* fix checkstyle
* fix checkstyle
* fix unit test
* add jacoco dependencies
* fix unit test
* fix unit test
* add jacoco dependencies
* add unit test
* add unit test
* add license
* fix checkstyle
* fix pom
* fix checkstyle
* fix checkstyle
* merge dev
* fix ui error
* fix pom error
* fix pom error
* fix test error
* fix test error
* mssql-jdbc exclude azure-keyvault
* fix test error
* merge dev and add unit test
* add notes
* rollback the CollectionUtils
* fix
* update sql
* fix
* fix
* fix query rule page error
* change dq.jar path
* fix sql error
* fix ui error
* fix(dq): jar path&task enum description
* add notes on DataQualityApplication
* fix dq result jump error
* fix(ui): page condition
* feat(ui): add show error output path
* change version
* remove all chinese word in sql
* merge
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
3 years ago
aeb7d89c50
to improvement #8192 ( #8193 )
...
Co-authored-by: ouyangyewei <yewei.oyyw@alibaba-inc.com>
3 years ago
fb9f4a1339
[Feature-5822][api] Add audit log ( #6322 )
...
* rebase
* update audit log type name
* fix the quoted property error
* fix bug
* update audit global switch default value
* fix wrong ddl comment
* fix wrong code style
* update audit log search sql
* add licenses
* add unit test
* fix wrong code style
* fix bugs
* fix wrong code style
* add test case
* add test case
* add license
* Add unit test in pom.xml
* resolve code smell problem
* mysql and postgresql ddl update
* update audit log path and audit page query
* update audit configuration
* update audit log schema, add resource type and resource id
* update audit query and list page
* resolve audit log front page bugs and log query error
* update audit configuration
* resolve wrong configuration
* update AuditLogControllerTest
* update application.yaml
* resolve code smell
* update standalone configuration
* resolve bugs
Co-authored-by: yc322 <1393242780@qq.com>
3 years ago
103ccb01da
change version to 2.0.4-SNAPSHOT ( #8188 )
3 years ago
b5fa54b6be
[Bug-7865][MasterServer] retry logic optimization ( #8156 )
...
* submit task optimization
* cloneAndReset
* update
* tolerant task restart when init
* fix snoar check
* fix test
* delete unuse file
* taskInstance key
* code style
* skip complete when retry
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
5e975dad34
[Enhancement-8006] Improve the performance of SQL query ( #8007 )
...
Co-authored-by: kuanghc1 <kuanghc1@lenovo.com>
3 years ago
24a97fda75
[Bug] [dolphinscheduler-api]support workflow instance to definition ( #7930 )
...
* add task save and binds workflow
* add task update with upstream
* support workflow instance to definition
* fix ut
3 years ago
5ea06e7eb8
Update ProcessDefinitionMapper.xml ( #7931 )
...
Resolves process definition list issue（data duplication ）
3 years ago
361d68db26
[FIX-7732][fix] fix column 'is_directory' of table `t_ds_resources` type error in PG database ( #7898 )
...
Fix column 'is_directory' of table t_ds_resources type error in PG database
This closes #7732
3 years ago
acf9c84c50
[Feature][workflow list edit] add task update with upstream ( #7829 )
...
* add task save and binds workflow
* add task update with upstream
3 years ago
4c2f77ee9c
[DS-7016][feat] Auto create workflow while import sql script with specific hint ( #7214 )
...
* [DS-7016][feat] Auto create workflow while import sql script with specific hint
* datasource : datasource name
* name: task name
* upstream: pre task names
* [DS-7016][feat] Auto create workflow while import sql script with specific hint
* remove excess blank lines
* [DS-7016][feat] Auto create workflow while import sql script with specific hint
* datasource : datasource name
* name: task name
* upstream: pre task names
* [DS-7016][feat] Auto create workflow while import sql script with specific hint
* datasource : datasource name
* name: task name
* upstream: pre task names
* Code optimization
* for Zip Bomb Attack
Co-authored-by: eye <eye.gu@aloudata.com>
3 years ago
4662e6d929
[BUG-7779][SQL] fix syntax error in init sql of postgre ( #7782 )
...
fix syntax error of table `t_ds_task_definition` and `t_ds_task_definition_log` in postgresql with column `task_group_priority int(4)`
3 years ago
c97170853e
add task save and binds workflow ( #7772 )
3 years ago
3e34e69cfb
The sql has a syntax error, and because it is a function, the exception information will be hidden ( #7761 )
3 years ago
d7d13f7f51
[Feature][workflow list edit] add task list page api ( #7743 )
...
* fix 7597
* fix copy bug
* add task list page api
* fix total
* fix total
3 years ago
71ccb42697
fix a few bugs ( #7703 )
3 years ago
5ab0135d18
[Fix-7345] Support the whole operations for the task group and the task group queue. ( #7678 )
...
* add task group
* modify task group
* add task group queue
* add task group queue
* add task group queue
* add task group queue
* add task group priority
* fix a few issue with api
* add task group queue
* modify task group queue
* modify task group
* fix a little bit issue
* fix a little bug in the page of managing task group
3 years ago
8808c0a700
[Bug-7474][MasterServer] fix failover when node host is null ( #7475 )
...
* fix failover when node host is null
* add failover execute thread
* worker handle dead server
* fix task instance failover time check
* fix upgrade sql
* failover logic update
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
8b292132c8
[Feature-7569]Optimize Dependent node loading times ( #7626 )
...
* feature 7569
* feature_7569
Co-authored-by: SbloodyS <sbloodys@qq.com>
3 years ago
c7d7eec679
[Fix-7520][Master] Data too long for column 'task_params' at row 1 ( #7521 )
...
* fix bug_7520
* fix bug_7520
* fix bug_7520
* bug_7520
Co-authored-by: SbloodyS <sbloodys@qq.com>
3 years ago
7a888c544c
Try ( #7603 )
3 years ago
9e763ad0d1
[Feature-#6422] [api-server] task group queue ( #7491 )
...
* add task group
* modify task group
* pull dev
* add license header
* modify code style
* fix code style
* fix sql error
* fix error
* fix test
* fix test
* fix test
* fix test
* fix code style
* fix ut
* code style
* fix unit test
* test ut
* ut
* add unittest
* test ut
* modify back ut
* majorization code
* fix conflict
* fix ut
* add task group api
* reset file
* fix ut
* fix lost column
* fix ut
* fix ut
* fix ut
* fix ut
* delete duplicate code
* fix code style 、name
* fix ut
* fix mapper
Co-authored-by: wangxj <wangxj31>
3 years ago
a00c4f9709
finished issue#7527 ( #7561 )
...
Co-authored-by: 时光 <caizhedong@cai-inc.com>
3 years ago
cdd4e7bf03
[Improvement-#7529][tools] Init DB schema from the full sql file. ( #7530 )
3 years ago
bb140fbc3b
[DS-7227][refactor] optimize alert plugin instance list order ( #7506 )
...
[DS-7227][refactor] optimize alert plugin instance list order
query all alert plugin instance list order by update time desc
optimize `select *` sql in the AlterPluginInstanceMapper.xml
3 years ago
676a952bcb
[Fix-#6854][dao] fix "ResourceMapper.xml -> queryResourcePaging" sql bug ( #6907 )
...
* fix complement data retry bug; issues: [Bug] [Master] A bug on task retry mechanism #6613
* [DS-#5293][chore]upgrade cron-utils version 5.0.5 -> 9.1.3
* [DS-#5293]update the licensing information of that dependency
* [DS-#5293]update the licensing information of that dependency
* update the licensing information of that dependency
* [DS-#5293]update the licensing information of that dependency
* [DS-#6854][fix] fix "ResourceMapper.xml -> queryResourcePaging" sql bug
* [DS-#6854][fix] fix "ResourceMapper.xml -> queryResourcePaging" sql bug
Co-authored-by: yangqiyu <yangqiyu@deepexi.com>
3 years ago
7e0010de35
[Refactor] Rename class file, unified word spelling. ( #7261 )
3 years ago
04805818c2
Fix alert plugin instance filter ( #7172 ) ( #7210 )
...
* Fix alert plugin instance filter (#7172 )
alert plugin could not filter by given
pattern, cause api server do not handle
parameter searchVal, This patch add parameter
`searchVal` for alert plugin.
fix : #7209
* Add test
* Recover AlertPluginInstanceVO
3 years ago
99b8ec6492
[Bug-7319][MasterServer] fix taskNode NPE when switch else branch is empty ( #7320 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
2ff76c7e73
Fix some SQL issues and make docker-compose runnable ( #7329 )
3 years ago
ac67a11d44
fix a few issues in the sql file ( #7327 )
3 years ago
f1b85cf892
Fix the missing project information in the process instance timeout alarm ( #7324 )
3 years ago
0f7e38ed3c
[Bug-7292][ApiServer] fix cache error when standalone ( #7293 )
3 years ago
f03b0e8c7b
to feature-7301 ( #7308 )
...
Co-authored-by: ouyangyewei <yewei.oyyw@alibaba-inc.com>
3 years ago
3fadfca085
fix in the process alarm message, projectid is 0 ( #7298 )
3 years ago
67cc260d52
Split the components into individual package, refactor configurations to use Spring properties ( #7160 )
...
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
* Split the components into individual package
A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
3 years ago
4e6265df11
fix Property or field 'all' cannot be found ( #7246 )
3 years ago
29414a080f
Fix homepage user wrong statistics result for new user ( #7242 )
...
* Fix homepage user wrong statistics result for new user
For now, when we create a new user, it would see homepage
statistic info as admin user like, cause `projectCodeArray`
would not filter when we render SQL. So I add prefix conditions
to handle this situation.
Also, I do some minimal refactor here. including change function
name no quit fit
fix : #7182
* Fix UT
3 years ago
a80a7b3fcf
[Feature-6963][MasterServer] unified cache manager ( #7187 )
...
* unified cache
* reduce db select
* checkstyle
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
9d65bf1452
fix postgre upgrade ( #7230 )
...
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
4743e8f891
add two files ( #7223 )
3 years ago
705ba74f52
[Fix-7195] The file path of 2.0.1-schema used a wrong separator. ( #7200 )
3 years ago
3c8d1c6da9
to feature #7191 ( #7194 )
...
Co-authored-by: ouyangyewei <yewei.oyyw@alibaba-inc.com>
3 years ago
27ea43d5df
to feature #7110 ( #7111 )
...
Co-authored-by: ouyangyewei <yewei.oyyw@alibaba-inc.com>
3 years ago
af9ed35295
[Feature][MasterServer] add cache for workGroup and schedule #6987 ( #7161 )
...
* to #6987
* to #6987
* to #6987
* to #6987
* to #6987
* to #6987 : @Param -> @Name
* to #6987 : fix Sonar
* to #6987 : fix Sonar
Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com>
3 years ago
0e75bd8008
[BUG][SQL] fix the process list page load slowly. ( #7166 )
...
* fix #7601 move ddl to 2.0.1
* fix #7601 move ddl to 2.0.1
3 years ago
ed9fca6c97
[Bug] [dolphinscheduler-api] The pages of project management, workflow instance and task instance are accessed slowly #7061 ( #7139 )
...
1、I found that the SQL behind this interface is very slow. The main table is large, but there is no index to use. By add the index, the all mode in the query plan can be changed into a more efficient ref mode. If this optimization still fails to meet the requirements, please contact me and I will continue to optimize。
2、modify index start_time_index in init script and upgrade scripts.
3 years ago
ba2b2a67c2
[Feature-6988][MasterServer] add cache manager for workflow ( #7090 )
...
* add cache manager for workflow
* [DS-6988][MasterServer] add cache manager for workflow
* cache evict code optimization
* test
Co-authored-by: caishunfeng <534328519@qq.com>
3 years ago
2bf73603bb
[Feature][dolphinscheduler-api] access control of taskDefinition and taskInstance in project to #7081 ( #7082 )
...
* to #7081
* fix #7081
* to #7081
Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com>
3 years ago
8ebe060658
[Feature][dolphinscheduler-api] access control of processDefinition and processInstance in project #7073 ( #7080 )
...
* to #7073
* to #7073
* to #7073
Co-authored-by: honghuo.zw <honghuo.zw@alibaba-inc.com>
3 years ago
zixi0825
ouyangyewei
yc322
Kerwin
wind
springmonster
JinYong Li
mazhong
天仇
ououtt
uh001
calvin
xiangzihao
kezhenxu94
wangxj3
时光
lgcareer
KyoYang
Zonglei Dong
Jiajie Zhong
zwZjut
BaoLiang
GaoTianDuo