Currently, our core and side module dependent on each others. and will cause
cycle import in our codebase, especially in issue #10905, we try to refactor
to solve this problem.
This patch do the following change:
* Rename module `side` to `models`
* Move `core/base` and `core/sidebase` to dir `modules`
* Move `configuration` and `default_config.yaml` to the root of pydolphinscheduler
* when task with more than one upstreams, mapper
TaskDefinitionMapper method queryByName will return
more than one record, and failed the mybatis result
type, so we have to add `limit 1` to it to
* add multiple runs of example in integrate test
* Change from subprocess.Popen to subprocess.call_check
in integrating test which will raise an error when failed
This patch fix the error release state not work when it set to
offline and submit it to Java gateway, it error because we do
not pass the attribute to Java gateway function
`createOrUpdateProcessDefinition`
close: #9779
* [python] Add task decorator for python function
* Add decorator `@task`
* Add a tutorial about it
* Change tutorial doc and combine into traditional docs
* Add sphinx-inline-tab for better view
* revert not need change
* Correct python function indent
* Correct integration test
Add a new method to get config from environment variables
and for now, we have three ways to get config and the
priority is `env-var > custom-config-file > built-in-config-file`.
Environment config setting does not work in CLI, because it will
confuse users when they get config value is `var-env` but value
in the configuration file is `var-in-file`, they may not find
the way how to change it
* Add documentation
* Add it to UPDATING.md
close: #8344
Currently the size of our distribute package is up to
800MB, this patch is migrate python gateway server into
api server
The distribute package size before and after this patch is:
```sh
# before
796M apache-dolphinscheduler-2.0.4-SNAPSHOT-bin.tar.gz
# after
647M apache-dolphinscheduler-2.0.4-SNAPSHOT-bin.tar.gz
```
* [python] Fix change exists pd attribute user error
* Remove attribute from class ProcessDefinition, cause it should to
object User instead of object ProcessDefinition.
* Grant project to user if attribute user change for exists ProcessDefinition
close: #8751
* Add py.test conftest.py for package integration
* [python] Add integrated test to python gateway server
* Build java code and create standalone server image in GA
* Add component start docker in python
* Run example to make sure it work to it
close: #8035
* Fix build docker image working directory
* Fix working directory
* Add a new commit hook to pre-commit, to auto
remove unused imports
* Remove all unused pass statements
* Add autoflake to tox and out CI
close: #8592
* add `.pre-commit-config.yaml` in the pydolphinscheduler module and configured basic check rules like `isort`, `black`, `flake8`
* add some instructions in `DEVELOP.MD`
* Fix some mistake and add some docs on it
Co-authored-by: Jiajie Zhong <zhongjiajie955@hotmail.com>
* [python] Add tox for out of box test
* Add `tox` for easy and complete test
* Correct package data attribute
* Add more dirs in flake8 ignore
* Change GA from raw to tox
* Fix env setter
* Temp skip sanity
* Rm not exists job
* Add code test
* fix
* rm all need
* fix
* fix
* fix
* fix
* fix
* fix
* Migrate pip upgrede to tox setting
* Change install_commands to commands
* fix config of install command
* Add GA env to do that
* Fix env error
* Finial commit
* remove duplicate
* Change doc
* Change dependence of ci
* Change dependence of ci
* Change `yaml_parser.py` method `to_string` to magic method
`__str__` make it more pythonic.
* Add some tests to magic method `__str__` and `__repr__`
* Add LICENSE and NOTICE to Python API
* Add new license dir `python-api-license` to keep all
Python API's LICENSE files
* Change LICENSE in distribute package
* Add content about should add LICENSE when new dependence
adding
* [python] refactor yaml file parser
* Change yaml parser package to ruamel.yaml
* Refactor configuration.py module
* And file.py to write file locally
* Add more tests on it
close: #8593
* Fix UT error
* Remove pypyaml from tests
* Fix file error when param create is False
* Fix error logic
* And tests to avoid regression
* [python] Add config mechanism and cli subcommand config
* Add configuration.py mechanism for more easy change config
and move some configs to it. It mechanism including
configuration.py module and default_config.yaml file
* Add `config` for cli subcommand allow users initialize, get,
set configs
close: #8344
* Change setup.py format
This patch add python api package as independent
directory in final dist directory when user run
command `mvn -U install package -Prelease`.
There have `tar.gz` and `whl` in the python dist directory
```
<the-old-dist-package>
python
|--- python-independent-pkg.tar.gz
|--- python-independent-pkg.whl
```
close: #8343
* Give dedicated control to docker build/push goals
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
* Add basic cli mechanism with Click, for now just including
one single subcommand `version`
* Add general and easy test class in tests/testing/cli, and
test to version
* Add sphinx-click to general cli docs basic on click
For now, python API could only communicate python gateway server
in the same hosts, this patch makes it could work with different hosts,
and export java gateway setting to configure file
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
Co-authored-by: ruanwenjun <861923274@qq.com>
* [python] Fix database error handler about no datasource name
The original code show too many unrelated log when database name
do not exists. BTW, it would continue the code when error miss.
This patch is try to fix it.
close: #7616
* Use java exception instead of self define
* Enhance
* [python] Add checker pd attr param and fix switch example
* Correct submit self.param to java gateway
* Fix missing parameter for switch example
* Add mechanism checker before submit to java gateway
close: #7803
* Fix mock get task code and version
* Change judge statement and add comment
* [python] Task condition missing two downstream param
We add two downstream tasks to set task condition
success and failed node
close: #7763
* Add getter and setter property condition_resulth in base task
* [python] Enhance task datax example
* Add full example for `CustomDataX.json`
* Add comment about datasource need to exists.
close: #7800
* Add missing parameter setting
* Make sure all file with `.py` extension and all file
end with "_example"
* Make sure all examples have `__doc__`
* Make sure not have duplicate process definition name
* Make sure process definition same as filename
close: #7729
After #7505 merged. we could use condition task type
but our dependent set in the wrong direction, all
the condition operators should be upstream of the
current task instead of downstream
fix: #7649
Jiajie Zhong
JieguangZhou
陈家名
Eric Gao
天仇
Devosend