Browse Source
* [cherry-pick] Cherry pick python api form branch dev * Also change python gateway server code * Add .gitkeep to license ignore2.0.7-release
Jiajie Zhong
3 years ago
committed by
GitHub
70 changed files with 2948 additions and 257 deletions
@ -0,0 +1,120 @@
|
||||
<!-- |
||||
Licensed to the Apache Software Foundation (ASF) under one |
||||
or more contributor license agreements. See the NOTICE file |
||||
distributed with this work for additional information |
||||
regarding copyright ownership. The ASF licenses this file |
||||
to you under the Apache License, Version 2.0 (the |
||||
"License"); you may not use this file except in compliance |
||||
with the License. You may obtain a copy of the License at |
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0 |
||||
|
||||
Unless required by applicable law or agreed to in writing, |
||||
software distributed under the License is distributed on an |
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
KIND, either express or implied. See the License for the |
||||
specific language governing permissions and limitations |
||||
under the License. |
||||
--> |
||||
|
||||
# Develop |
||||
|
||||
pydolphinscheduler is python API for Apache DolphinScheduler, it just defines what workflow look like instead of |
||||
store or execute it. We here use [py4j][py4j] to dynamically access Java Virtual Machine. |
||||
|
||||
## Setup Develop Environment |
||||
|
||||
**PyDolphinScheduler** use GitHub to hold all source code, you should clone the code before you do same change. |
||||
|
||||
```shell |
||||
git clone git@github.com:apache/dolphinscheduler.git |
||||
``` |
||||
|
||||
Now, we should install all dependence to make sure we could run test or check code style locally |
||||
|
||||
```shell |
||||
cd dolphinscheduler/dolphinscheduler-python/pydolphinscheduler |
||||
pip install .[dev] |
||||
``` |
||||
|
||||
Next, we have to open pydolphinscheduler project in you editor. We recommend you use [pycharm][pycharm] |
||||
instead of [IntelliJ IDEA][idea] to open it. And you could just open directory |
||||
`dolphinscheduler-python/pydolphinscheduler` instead of `dolphinscheduler-python`. |
||||
|
||||
|
||||
## Brief Concept |
||||
|
||||
Apache DolphinScheduler is design to define workflow by UI, and pydolphinscheduler try to define it by code. When |
||||
define by code, user usually do not care user, tenant, or queue exists or not. All user care about is created |
||||
a new workflow by the code his/her definition. So we have some **side object** in `pydolphinscheduler/side` |
||||
directory, their only check object exists or not, and create them if not exists. |
||||
|
||||
### Process Definition |
||||
|
||||
pydolphinscheduler workflow object name, process definition is also same name as Java object(maybe would be change to |
||||
other word for more simple). |
||||
|
||||
### Tasks |
||||
|
||||
pydolphinscheduler tasks object, we use tasks to define exact job we want DolphinScheduler do for us. For now, |
||||
we only support `shell` task to execute shell task. [This link][all-task] list all tasks support in DolphinScheduler |
||||
and would be implemented in the further. |
||||
|
||||
## Code Style |
||||
|
||||
We use [isort][isort] to automatically keep Python imports alphabetically, and use [Black][black] for code |
||||
formatter and [Flake8][flake8] for pep8 checker. If you use [pycharm][pycharm]or [IntelliJ IDEA][idea], |
||||
maybe you could follow [Black-integration][black-editor] to configure them in your environment. |
||||
|
||||
Our Python API CI would automatically run code style checker and unittest when you submit pull request in |
||||
GitHub, you could also run static check locally. |
||||
|
||||
```shell |
||||
# We recommend you run isort and Black before Flake8, because Black could auto fix some code style issue |
||||
# but Flake8 just hint when code style not match pep8 |
||||
|
||||
# Run Isort |
||||
isort . |
||||
|
||||
# Run Black |
||||
black . |
||||
|
||||
# Run Flake8 |
||||
flake8 |
||||
``` |
||||
|
||||
## Testing |
||||
|
||||
pydolphinscheduler using [pytest][pytest] to test our codebase. GitHub Action will run our test when you create |
||||
pull request or commit to dev branch, with python version `3.6|3.7|3.8|3.9` and operating system `linux|macOS|windows`. |
||||
|
||||
To test locally, you could directly run pytest after set `PYTHONPATH` |
||||
|
||||
```shell |
||||
PYTHONPATH=src/ pytest |
||||
``` |
||||
|
||||
We try to keep pydolphinscheduler usable through unit test coverage. 90% test coverage is our target, but for |
||||
now, we require test coverage up to 85%, and each pull request leas than 85% would fail our CI step |
||||
`Tests coverage`. We use [coverage][coverage] to check our test coverage, and you could check it locally by |
||||
run command. |
||||
|
||||
```shell |
||||
coverage run && coverage report |
||||
``` |
||||
|
||||
It would not only run unit test but also show each file coverage which cover rate less than 100%, and `TOTAL` |
||||
line show you total coverage of you code. If your CI failed with coverage you could go and find some reason by |
||||
this command output. |
||||
|
||||
<!-- content --> |
||||
[py4j]: https://www.py4j.org/index.html |
||||
[pycharm]: https://www.jetbrains.com/pycharm |
||||
[idea]: https://www.jetbrains.com/idea/ |
||||
[all-task]: https://dolphinscheduler.apache.org/en-us/docs/dev/user_doc/guide/task/shell.html |
||||
[pytest]: https://docs.pytest.org/en/latest/ |
||||
[black]: https://black.readthedocs.io/en/stable/index.html |
||||
[flake8]: https://flake8.pycqa.org/en/latest/index.html |
||||
[black-editor]: https://black.readthedocs.io/en/stable/integrations/editors.html#pycharm-intellij-idea |
||||
[coverage]: https://coverage.readthedocs.io/en/stable/ |
||||
[isort]: https://pycqa.github.io/isort/index.html |
||||
@ -0,0 +1,35 @@
|
||||
<!-- |
||||
Licensed to the Apache Software Foundation (ASF) under one |
||||
or more contributor license agreements. See the NOTICE file |
||||
distributed with this work for additional information |
||||
regarding copyright ownership. The ASF licenses this file |
||||
to you under the Apache License, Version 2.0 (the |
||||
"License"); you may not use this file except in compliance |
||||
with the License. You may obtain a copy of the License at |
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0 |
||||
|
||||
Unless required by applicable law or agreed to in writing, |
||||
software distributed under the License is distributed on an |
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
KIND, either express or implied. See the License for the |
||||
specific language governing permissions and limitations |
||||
under the License. |
||||
--> |
||||
|
||||
# Release |
||||
|
||||
**PyDolphinScheduler** office release is in [ASF Distribution Directory](https://downloads.apache.org/dolphinscheduler/), |
||||
and it should be released together with [apache-dolphinscheduler](https://github.com/apache/dolphinscheduler). |
||||
|
||||
## To ASF Distribution Directory |
||||
|
||||
You could release to [ASF Distribution Directory](https://downloads.apache.org/dolphinscheduler/) according to |
||||
[release guide](https://dolphinscheduler.apache.org/en-us/community/release-prepare.html) in DolphinScheduler |
||||
website. |
||||
|
||||
## To PyPi |
||||
|
||||
[PyPI](https://pypi.org), Python Package Index, is a repository of software for the Python programming language. |
||||
User could install Python package from it. Release to PyPi make user easier to install and try PyDolphinScheduler, |
||||
There is an official way to package project from [PyPA](https://packaging.python.org/en/latest/tutorials/packaging-projects) |
||||
@ -1,34 +0,0 @@
|
||||
<!-- |
||||
Licensed to the Apache Software Foundation (ASF) under one |
||||
or more contributor license agreements. See the NOTICE file |
||||
distributed with this work for additional information |
||||
regarding copyright ownership. The ASF licenses this file |
||||
to you under the Apache License, Version 2.0 (the |
||||
"License"); you may not use this file except in compliance |
||||
with the License. You may obtain a copy of the License at |
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0 |
||||
|
||||
Unless required by applicable law or agreed to in writing, |
||||
software distributed under the License is distributed on an |
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
KIND, either express or implied. See the License for the |
||||
specific language governing permissions and limitations |
||||
under the License. |
||||
--> |
||||
|
||||
## Roadmap |
||||
|
||||
### v0.0.3 |
||||
|
||||
Add other features, tasks, parameters in DS, keep code coverage up to 90% |
||||
|
||||
### v0.0.2 |
||||
|
||||
Add docs about how to use and develop package, code coverage up to 90%, add CI/CD |
||||
for package |
||||
|
||||
### v0.0.1(current) |
||||
|
||||
Setup up POC, for defining DAG with python code, running DAG manually, |
||||
releasing to pypi |
||||
@ -0,0 +1,39 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one
|
||||
# or more contributor license agreements. See the NOTICE file
|
||||
# distributed with this work for additional information
|
||||
# regarding copyright ownership. The ASF licenses this file
|
||||
# to you under the Apache License, Version 2.0 (the
|
||||
# "License"); you may not use this file except in compliance
|
||||
# with the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing,
|
||||
# software distributed under the License is distributed on an
|
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
# KIND, either express or implied. See the License for the
|
||||
# specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
# Minimal makefile for Sphinx documentation
|
||||
#
|
||||
|
||||
# You can set these variables from the command line, and also
|
||||
# from the environment for the first two.
|
||||
|
||||
# Add opts `turn warnings into errors` strict sphinx-build behavior
|
||||
SPHINXOPTS ?= -W
|
||||
SPHINXBUILD ?= sphinx-build
|
||||
SOURCEDIR = source
|
||||
BUILDDIR = build
|
||||
|
||||
# Put it first so that "make" without argument is like "make help".
|
||||
help: |
||||
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
|
||||
|
||||
.PHONY: help Makefile |
||||
|
||||
# Catch-all target: route all unknown targets to Sphinx using the new
|
||||
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
|
||||
%: Makefile |
||||
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
|
||||
@ -0,0 +1,54 @@
|
||||
REM Licensed to the Apache Software Foundation (ASF) under one |
||||
REM or more contributor license agreements. See the NOTICE file |
||||
REM distributed with this work for additional information |
||||
REM regarding copyright ownership. The ASF licenses this file |
||||
REM to you under the Apache License, Version 2.0 (the |
||||
REM "License"); you may not use this file except in compliance |
||||
REM with the License. You may obtain a copy of the License at |
||||
REM |
||||
REM http://www.apache.org/licenses/LICENSE-2.0 |
||||
REM |
||||
REM Unless required by applicable law or agreed to in writing, |
||||
REM software distributed under the License is distributed on an |
||||
REM "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
REM KIND, either express or implied. See the License for the |
||||
REM specific language governing permissions and limitations |
||||
REM under the License. |
||||
|
||||
@ECHO OFF |
||||
|
||||
pushd %~dp0 |
||||
|
||||
REM Command file for Sphinx documentation |
||||
|
||||
if "%SPHINXBUILD%" == "" ( |
||||
set SPHINXBUILD=sphinx-build |
||||
) |
||||
set SOURCEDIR=source |
||||
set BUILDDIR=build |
||||
REM Add opts `turn warnings into errors` strict sphinx-build behavior |
||||
set SPHINXOPTS=-W |
||||
|
||||
if "%1" == "" goto help |
||||
|
||||
%SPHINXBUILD% >NUL 2>NUL |
||||
if errorlevel 9009 ( |
||||
echo. |
||||
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx |
||||
echo.installed, then set the SPHINXBUILD environment variable to point |
||||
echo.to the full path of the 'sphinx-build' executable. Alternatively you |
||||
echo.may add the Sphinx directory to PATH. |
||||
echo. |
||||
echo.If you don't have Sphinx installed, grab it from |
||||
echo.https://www.sphinx-doc.org/ |
||||
exit /b 1 |
||||
) |
||||
|
||||
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% |
||||
goto end |
||||
|
||||
:help |
||||
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% |
||||
|
||||
:end |
||||
popd |
||||
@ -0,0 +1,47 @@
|
||||
.. Licensed to the Apache Software Foundation (ASF) under one |
||||
or more contributor license agreements. See the NOTICE file |
||||
distributed with this work for additional information |
||||
regarding copyright ownership. The ASF licenses this file |
||||
to you under the Apache License, Version 2.0 (the |
||||
"License"); you may not use this file except in compliance |
||||
with the License. You may obtain a copy of the License at |
||||
|
||||
.. http://www.apache.org/licenses/LICENSE-2.0 |
||||
|
||||
.. Unless required by applicable law or agreed to in writing, |
||||
software distributed under the License is distributed on an |
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
KIND, either express or implied. See the License for the |
||||
specific language governing permissions and limitations |
||||
under the License. |
||||
|
||||
API |
||||
=== |
||||
|
||||
Core |
||||
---- |
||||
|
||||
.. automodule:: pydolphinscheduler.core |
||||
:inherited-members: |
||||
|
||||
Sides |
||||
----- |
||||
|
||||
.. automodule:: pydolphinscheduler.side |
||||
:inherited-members: |
||||
|
||||
Tasks |
||||
----- |
||||
|
||||
.. automodule:: pydolphinscheduler.tasks |
||||
:inherited-members: |
||||
|
||||
Constants |
||||
--------- |
||||
|
||||
.. automodule:: pydolphinscheduler.constants |
||||
|
||||
Exceptions |
||||
---------- |
||||
|
||||
.. automodule:: pydolphinscheduler.exceptions |
||||
@ -0,0 +1,151 @@
|
||||
.. Licensed to the Apache Software Foundation (ASF) under one |
||||
or more contributor license agreements. See the NOTICE file |
||||
distributed with this work for additional information |
||||
regarding copyright ownership. The ASF licenses this file |
||||
to you under the Apache License, Version 2.0 (the |
||||
"License"); you may not use this file except in compliance |
||||
with the License. You may obtain a copy of the License at |
||||
|
||||
.. http://www.apache.org/licenses/LICENSE-2.0 |
||||
|
||||
.. Unless required by applicable law or agreed to in writing, |
||||
software distributed under the License is distributed on an |
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
KIND, either express or implied. See the License for the |
||||
specific language governing permissions and limitations |
||||
under the License. |
||||
|
||||
Concepts |
||||
======== |
||||
|
||||
In this section, you would know the core concepts of *PyDolphinScheduler*. |
||||
|
||||
Process Definition |
||||
------------------ |
||||
|
||||
Process definition describe the whole things except `tasks`_ and `tasks dependence`_, which including |
||||
name, schedule interval, schedule start time and end time. You would know scheduler |
||||
|
||||
Process definition could be initialized in normal assign statement or in context manger. |
||||
|
||||
.. code-block:: python |
||||
|
||||
# Initialization with assign statement |
||||
pd = ProcessDefinition(name="my first process definition") |
||||
|
||||
# Or context manger |
||||
with ProcessDefinition(name="my first process definition") as pd: |
||||
pd.submit() |
||||
|
||||
Process definition is the main object communicate between *PyDolphinScheduler* and DolphinScheduler daemon. |
||||
After process definition and task is be declared, you could use `submit` and `run` notify server your definition. |
||||
|
||||
If you just want to submit your definition and create workflow, without run it, you should use attribute `submit`. |
||||
But if you want to run the workflow after you submit it, you could use attribute `run`. |
||||
|
||||
.. code-block:: python |
||||
|
||||
# Just submit definition, without run it |
||||
pd.submit() |
||||
|
||||
# Both submit and run definition |
||||
pd.run() |
||||
|
||||
Schedule |
||||
~~~~~~~~ |
||||
|
||||
We use parameter `schedule` determine the schedule interval of workflow, *PyDolphinScheduler* support seven |
||||
asterisks expression, and each of the meaning of position as below |
||||
|
||||
.. code-block:: text |
||||
|
||||
* * * * * * * |
||||
┬ ┬ ┬ ┬ ┬ ┬ ┬ |
||||
│ │ │ │ │ │ │ |
||||
│ │ │ │ │ │ └─── year |
||||
│ │ │ │ │ └───── day of week (0 - 7) (0 to 6 are Sunday to Saturday, or use names; 7 is Sunday, the same as 0) |
||||
│ │ │ │ └─────── month (1 - 12) |
||||
│ │ │ └───────── day of month (1 - 31) |
||||
│ │ └─────────── hour (0 - 23) |
||||
│ └───────────── min (0 - 59) |
||||
└─────────────── second (0 - 59) |
||||
|
||||
Here we add some example crontab: |
||||
|
||||
- `0 0 0 * * ? *`: Workflow execute every day at 00:00:00. |
||||
- `10 2 * * * ? *`: Workflow execute hourly day at ten pass two. |
||||
- `10,11 20 0 1,2 * ? *`: Workflow execute first and second day of month at 00:20:10 and 00:20:11. |
||||
|
||||
Tenant |
||||
~~~~~~ |
||||
|
||||
Tenant is the user who run task command in machine or in virtual machine. it could be assign by simple string. |
||||
|
||||
.. code-block:: python |
||||
|
||||
# |
||||
pd = ProcessDefinition(name="process definition tenant", tenant="tenant_exists") |
||||
|
||||
.. note:: |
||||
|
||||
Make should tenant exists in target machine, otherwise it will raise an error when you try to run command |
||||
|
||||
Tasks |
||||
----- |
||||
|
||||
Task is the minimum unit running actual job, and it is nodes of DAG, aka directed acyclic graph. You could define |
||||
what you want to in the task. It have some required parameter to make uniqueness and definition. |
||||
|
||||
Here we use :py:meth:`pydolphinscheduler.tasks.Shell` as example, parameter `name` and `command` is required and must be provider. Parameter |
||||
`name` set name to the task, and parameter `command` declare the command you wish to run in this task. |
||||
|
||||
.. code-block:: python |
||||
|
||||
# We named this task as "shell", and just run command `echo shell task` |
||||
shell_task = Shell(name="shell", command="echo shell task") |
||||
|
||||
If you want to see all type of tasks, you could see :doc:`tasks/index`. |
||||
|
||||
Tasks Dependence |
||||
~~~~~~~~~~~~~~~~ |
||||
|
||||
You could define many tasks in on single `Process Definition`_. If all those task is in parallel processing, |
||||
then you could leave them alone without adding any additional information. But if there have some tasks should |
||||
not be run unless pre task in workflow have be done, we should set task dependence to them. Set tasks dependence |
||||
have two mainly way and both of them is easy. You could use bitwise operator `>>` and `<<`, or task attribute |
||||
`set_downstream` and `set_upstream` to do it. |
||||
|
||||
.. code-block:: python |
||||
|
||||
# Set task1 as task2 upstream |
||||
task1 >> task2 |
||||
# You could use attribute `set_downstream` too, is same as `task1 >> task2` |
||||
task1.set_downstream(task2) |
||||
|
||||
# Set task1 as task2 downstream |
||||
task1 << task2 |
||||
# It is same as attribute `set_upstream` |
||||
task1.set_upstream(task2) |
||||
|
||||
# Beside, we could set dependence between task and sequence of tasks, |
||||
# we set `task1` is upstream to both `task2` and `task3`. It is useful |
||||
# for some tasks have same dependence. |
||||
task1 >> [task2, task3] |
||||
|
||||
Task With Process Definition |
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
||||
|
||||
In most of data orchestration cases, you should assigned attribute `process_definition` to task instance to |
||||
decide workflow of task. You could set `process_definition` in both normal assign or in context manger mode |
||||
|
||||
.. code-block:: python |
||||
|
||||
# Normal assign, have to explicit declaration and pass `ProcessDefinition` instance to task |
||||
pd = ProcessDefinition(name="my first process definition") |
||||
shell_task = Shell(name="shell", command="echo shell task", process_definition=pd) |
||||
|
||||
# Context manger, `ProcessDefinition` instance pd would implicit declaration to task |
||||
with ProcessDefinition(name="my first process definition") as pd: |
||||
shell_task = Shell(name="shell", command="echo shell task", |
||||
|
||||
With both `Process Definition`_, `Tasks`_ and `Tasks Dependence`_, we could build a workflow with multiple tasks. |
||||
@ -0,0 +1,88 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one |
||||
# or more contributor license agreements. See the NOTICE file |
||||
# distributed with this work for additional information |
||||
# regarding copyright ownership. The ASF licenses this file |
||||
# to you under the Apache License, Version 2.0 (the |
||||
# "License"); you may not use this file except in compliance |
||||
# with the License. You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, |
||||
# software distributed under the License is distributed on an |
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
# KIND, either express or implied. See the License for the |
||||
# specific language governing permissions and limitations |
||||
# under the License. |
||||
|
||||
# Configuration file for the Sphinx documentation builder. |
||||
# |
||||
# This file only contains a selection of the most common options. For a full |
||||
# list see the documentation: |
||||
# https://www.sphinx-doc.org/en/master/usage/configuration.html |
||||
|
||||
# -- Path setup -------------------------------------------------------------- |
||||
|
||||
# If extensions (or modules to document with autodoc) are in another directory, |
||||
# add these directories to sys.path here. If the directory is relative to the |
||||
# documentation root, use os.path.abspath to make it absolute, like shown here. |
||||
# |
||||
# import os |
||||
# import sys |
||||
# sys.path.insert(0, os.path.abspath('.')) |
||||
|
||||
|
||||
# -- Project information ----------------------------------------------------- |
||||
|
||||
project = "pydolphinscheduler" |
||||
copyright = "2022, apache" |
||||
author = "apache" |
||||
|
||||
# The full version, including alpha/beta/rc tags |
||||
release = "0.0.1" |
||||
|
||||
|
||||
# -- General configuration --------------------------------------------------- |
||||
|
||||
# Add any Sphinx extension module names here, as strings. They can be |
||||
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom |
||||
# ones. |
||||
extensions = [ |
||||
# Measures durations of Sphinx processing |
||||
"sphinx.ext.duration", |
||||
# Semi-automatic make docstrings to document |
||||
"sphinx.ext.autodoc", |
||||
"sphinx.ext.viewcode", |
||||
"sphinx.ext.autosectionlabel", |
||||
"sphinx_rtd_theme", |
||||
] |
||||
|
||||
# Add any paths that contain templates here, relative to this directory. |
||||
templates_path = ["_templates"] |
||||
|
||||
# List of patterns, relative to source directory, that match files and |
||||
# directories to ignore when looking for source files. |
||||
# This pattern also affects html_static_path and html_extra_path. |
||||
exclude_patterns = [] |
||||
|
||||
autodoc_default_options = { |
||||
"members": True, |
||||
"show-inheritance": True, |
||||
"private-members": True, |
||||
"undoc-members": True, |
||||
"member-order": "groupwise", |
||||
} |
||||
|
||||
autosectionlabel_prefix_document = True |
||||
|
||||
# -- Options for HTML output ------------------------------------------------- |
||||
|
||||
# The theme to use for HTML and HTML Help pages. See the documentation for |
||||
# a list of builtin themes. |
||||
# |
||||
html_theme = "sphinx_rtd_theme" |
||||
|
||||
# Add any paths that contain custom static files (such as style sheets) here, |
||||
# relative to this directory. They are copied after the builtin static files, |
||||
# so a file named "default.css" will overwrite the builtin "default.css". |
||||
html_static_path = ["_static"] |
||||
@ -0,0 +1,42 @@
|
||||
.. Licensed to the Apache Software Foundation (ASF) under one |
||||
or more contributor license agreements. See the NOTICE file |
||||
distributed with this work for additional information |
||||
regarding copyright ownership. The ASF licenses this file |
||||
to you under the Apache License, Version 2.0 (the |
||||
"License"); you may not use this file except in compliance |
||||
with the License. You may obtain a copy of the License at |
||||
|
||||
.. http://www.apache.org/licenses/LICENSE-2.0 |
||||
|
||||
.. Unless required by applicable law or agreed to in writing, |
||||
software distributed under the License is distributed on an |
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
KIND, either express or implied. See the License for the |
||||
specific language governing permissions and limitations |
||||
under the License. |
||||
|
||||
PyDolphinScheduler |
||||
================== |
||||
|
||||
**PyDolphinScheduler** is Python API for `Apache DolphinScheduler <https://dolphinscheduler.apache.org>`_, |
||||
which allow you definition your workflow by Python code, aka workflow-as-codes. |
||||
|
||||
I could go and find how to :ref:`install <start:getting started>` the project. Or if you want to see simply example |
||||
then go and see :doc:`tutorial` for more detail. |
||||
|
||||
|
||||
.. toctree:: |
||||
:maxdepth: 2 |
||||
|
||||
start |
||||
tutorial |
||||
concept |
||||
tasks/index |
||||
api |
||||
|
||||
Indices and tables |
||||
================== |
||||
|
||||
* :ref:`genindex` |
||||
* :ref:`modindex` |
||||
* :ref:`search` |
||||
@ -0,0 +1,113 @@
|
||||
.. Licensed to the Apache Software Foundation (ASF) under one |
||||
or more contributor license agreements. See the NOTICE file |
||||
distributed with this work for additional information |
||||
regarding copyright ownership. The ASF licenses this file |
||||
to you under the Apache License, Version 2.0 (the |
||||
"License"); you may not use this file except in compliance |
||||
with the License. You may obtain a copy of the License at |
||||
|
||||
.. http://www.apache.org/licenses/LICENSE-2.0 |
||||
|
||||
.. Unless required by applicable law or agreed to in writing, |
||||
software distributed under the License is distributed on an |
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
KIND, either express or implied. See the License for the |
||||
specific language governing permissions and limitations |
||||
under the License. |
||||
|
||||
Getting Started |
||||
=============== |
||||
|
||||
To get started with *PyDolphinScheduler* you must ensure python and pip |
||||
installed on your machine, if you're already set up, you can skip straight |
||||
to `Installing PyDolphinScheduler`_, otherwise please continue with |
||||
`Installing Python`_. |
||||
|
||||
Installing Python |
||||
----------------- |
||||
|
||||
How to install `python` and `pip` depends on what operating system |
||||
you're using. The python wiki provides up to date |
||||
`instructions for all platforms here`_. When you entering the website |
||||
and choice your operating system, you would be offered the choice and |
||||
select python version. *PyDolphinScheduler* recommend use version above |
||||
Python 3.6 and we highly recommend you install *Stable Releases* instead |
||||
of *Pre-releases*. |
||||
|
||||
After you have download and installed Python, you should open your terminal, |
||||
typing and running :code:`python --version` to check whether the installation |
||||
is correct or not. If all thing good, you could see the version in console |
||||
without error(here is a example after Python 3.8.7 installed) |
||||
|
||||
.. code-block:: bash |
||||
|
||||
$ python --version |
||||
Python 3.8.7 |
||||
|
||||
Installing PyDolphinScheduler |
||||
----------------------------- |
||||
|
||||
After Python is already installed on your machine following section |
||||
`installing Python`_, it easy to *PyDolphinScheduler* by pip. |
||||
|
||||
.. code-block:: bash |
||||
|
||||
$ pip install apache-dolphinscheduler |
||||
|
||||
The latest version of *PyDolphinScheduler* would be installed after you run above |
||||
command in your terminal. You could go and `start Python Gateway Server`_ to finish |
||||
the prepare, and then go to :doc:`tutorial` to make your hand dirty. But if you |
||||
want to install the unreleased version of *PyDolphinScheduler*, you could go and see |
||||
section `installing PyDolphinScheduler in dev`_ for more detail. |
||||
|
||||
Installing PyDolphinScheduler In Dev |
||||
------------------------------------ |
||||
|
||||
Because the project is developing and some of the features still not release. |
||||
If you want to try some thing unreleased you could install from the source code |
||||
which we hold in GitHub |
||||
|
||||
.. code-block:: bash |
||||
|
||||
# Clone Apache DolphinScheduler repository |
||||
$ git clone git@github.com:apache/dolphinscheduler.git |
||||
# Install PyDolphinScheduler in develop mode |
||||
$ cd dolphinscheduler-python/pydolphinscheduler && pip install -e . |
||||
|
||||
After you installed *PyDolphinScheduler*, please remember `start Python Gateway Server`_ |
||||
which waiting for *PyDolphinScheduler*'s workflow definition require. |
||||
|
||||
Start Python Gateway Server |
||||
--------------------------- |
||||
|
||||
Since **PyDolphinScheduler** is Python API for `Apache DolphinScheduler`_, it |
||||
could define workflow and tasks structure, but could not run it unless you |
||||
`install Apache DolphinScheduler`_ and start Python gateway server. We only |
||||
and some key steps here and you could go `install Apache DolphinScheduler`_ |
||||
for more detail |
||||
|
||||
.. code-block:: bash |
||||
|
||||
# Start pythonGatewayServer |
||||
$ ./bin/dolphinscheduler-daemon.sh start pythonGatewayServer |
||||
|
||||
To check whether the server is alive or not, you could run :code:`jps`. And |
||||
the server is health if keyword `PythonGatewayServer` in the console. |
||||
|
||||
.. code-block:: bash |
||||
|
||||
$ jps |
||||
.... |
||||
201472 PythonGatewayServer |
||||
.... |
||||
|
||||
What's More |
||||
----------- |
||||
|
||||
If you do not familiar with *PyDolphinScheduler*, you could go to :doc:`tutorial` |
||||
and see how it work. But if you already know the inside of *PyDolphinScheduler*, |
||||
maybe you could go and play with all :doc:`tasks/index` *PyDolphinScheduler* supports. |
||||
|
||||
.. _`instructions for all platforms here`: https://wiki.python.org/moin/BeginnersGuide/Download |
||||
.. _`Apache DolphinScheduler`: https://dolphinscheduler.apache.org |
||||
.. _`install Apache DolphinScheduler`: https://dolphinscheduler.apache.org/en-us/docs/latest/user_doc/guide/installation/standalone.html |
||||
@ -0,0 +1,33 @@
|
||||
.. Licensed to the Apache Software Foundation (ASF) under one |
||||
or more contributor license agreements. See the NOTICE file |
||||
distributed with this work for additional information |
||||
regarding copyright ownership. The ASF licenses this file |
||||
to you under the Apache License, Version 2.0 (the |
||||
"License"); you may not use this file except in compliance |
||||
with the License. You may obtain a copy of the License at |
||||
|
||||
.. http://www.apache.org/licenses/LICENSE-2.0 |
||||
|
||||
.. Unless required by applicable law or agreed to in writing, |
||||
software distributed under the License is distributed on an |
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
KIND, either express or implied. See the License for the |
||||
specific language governing permissions and limitations |
||||
under the License. |
||||
|
||||
Condition |
||||
========= |
||||
|
||||
A condition task type's example and dive into information of **PyDolphinScheduler**. |
||||
|
||||
Example |
||||
------- |
||||
|
||||
.. literalinclude:: ../../../src/pydolphinscheduler/examples/task_condition_example.py |
||||
:start-after: [start workflow_declare] |
||||
:end-before: [end workflow_declare] |
||||
|
||||
Dive Into |
||||
--------- |
||||
|
||||
.. automodule:: pydolphinscheduler.tasks.condition |
||||
@ -0,0 +1,33 @@
|
||||
.. Licensed to the Apache Software Foundation (ASF) under one |
||||
or more contributor license agreements. See the NOTICE file |
||||
distributed with this work for additional information |
||||
regarding copyright ownership. The ASF licenses this file |
||||
to you under the Apache License, Version 2.0 (the |
||||
"License"); you may not use this file except in compliance |
||||
with the License. You may obtain a copy of the License at |
||||
|
||||
.. http://www.apache.org/licenses/LICENSE-2.0 |
||||
|
||||
.. Unless required by applicable law or agreed to in writing, |
||||
software distributed under the License is distributed on an |
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
KIND, either express or implied. See the License for the |
||||
specific language governing permissions and limitations |
||||
under the License. |
||||
|
||||
Datax |
||||
===== |
||||
|
||||
A DataX task type's example and dive into information of **PyDolphinScheduler**. |
||||
|
||||
Example |
||||
------- |
||||
|
||||
.. literalinclude:: ../../../src/pydolphinscheduler/examples/task_datax_example.py |
||||
:start-after: [start workflow_declare] |
||||
:end-before: [end workflow_declare] |
||||
|
||||
Dive Into |
||||
--------- |
||||
|
||||
.. automodule:: pydolphinscheduler.tasks.datax |
||||
@ -0,0 +1,33 @@
|
||||
.. Licensed to the Apache Software Foundation (ASF) under one |
||||
or more contributor license agreements. See the NOTICE file |
||||
distributed with this work for additional information |
||||
regarding copyright ownership. The ASF licenses this file |
||||
to you under the Apache License, Version 2.0 (the |
||||
"License"); you may not use this file except in compliance |
||||
with the License. You may obtain a copy of the License at |
||||
|
||||
.. http://www.apache.org/licenses/LICENSE-2.0 |
||||
|
||||
.. Unless required by applicable law or agreed to in writing, |
||||
software distributed under the License is distributed on an |
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
KIND, either express or implied. See the License for the |
||||
specific language governing permissions and limitations |
||||
under the License. |
||||
|
||||
Dependent |
||||
========= |
||||
|
||||
A dependent task type's example and dive into information of **PyDolphinScheduler**. |
||||
|
||||
Example |
||||
------- |
||||
|
||||
.. literalinclude:: ../../../src/pydolphinscheduler/examples/task_dependent_example.py |
||||
:start-after: [start workflow_declare] |
||||
:end-before: [end workflow_declare] |
||||
|
||||
Dive Into |
||||
--------- |
||||
|
||||
.. automodule:: pydolphinscheduler.tasks.dependent |
||||
@ -0,0 +1,33 @@
|
||||
.. Licensed to the Apache Software Foundation (ASF) under one |
||||
or more contributor license agreements. See the NOTICE file |
||||
distributed with this work for additional information |
||||
regarding copyright ownership. The ASF licenses this file |
||||
to you under the Apache License, Version 2.0 (the |
||||
"License"); you may not use this file except in compliance |
||||
with the License. You may obtain a copy of the License at |
||||
|
||||
.. http://www.apache.org/licenses/LICENSE-2.0 |
||||
|
||||
.. Unless required by applicable law or agreed to in writing, |
||||
software distributed under the License is distributed on an |
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
KIND, either express or implied. See the License for the |
||||
specific language governing permissions and limitations |
||||
under the License. |
||||
|
||||
Flink |
||||
===== |
||||
|
||||
A flink task type's example and dive into information of **PyDolphinScheduler**. |
||||
|
||||
Example |
||||
------- |
||||
|
||||
.. literalinclude:: ../../../src/pydolphinscheduler/examples/task_flink_example.py |
||||
:start-after: [start workflow_declare] |
||||
:end-before: [end workflow_declare] |
||||
|
||||
Dive Into |
||||
--------- |
||||
|
||||
.. automodule:: pydolphinscheduler.tasks.flink |
||||
@ -0,0 +1,21 @@
|
||||
.. Licensed to the Apache Software Foundation (ASF) under one |
||||
or more contributor license agreements. See the NOTICE file |
||||
distributed with this work for additional information |
||||
regarding copyright ownership. The ASF licenses this file |
||||
to you under the Apache License, Version 2.0 (the |
||||
"License"); you may not use this file except in compliance |
||||
with the License. You may obtain a copy of the License at |
||||
|
||||
.. http://www.apache.org/licenses/LICENSE-2.0 |
||||
|
||||
.. Unless required by applicable law or agreed to in writing, |
||||
software distributed under the License is distributed on an |
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
KIND, either express or implied. See the License for the |
||||
specific language governing permissions and limitations |
||||
under the License. |
||||
|
||||
HTTP |
||||
==== |
||||
|
||||
.. automodule:: pydolphinscheduler.tasks.http |
||||
@ -0,0 +1,41 @@
|
||||
.. Licensed to the Apache Software Foundation (ASF) under one |
||||
or more contributor license agreements. See the NOTICE file |
||||
distributed with this work for additional information |
||||
regarding copyright ownership. The ASF licenses this file |
||||
to you under the Apache License, Version 2.0 (the |
||||
"License"); you may not use this file except in compliance |
||||
with the License. You may obtain a copy of the License at |
||||
|
||||
.. http://www.apache.org/licenses/LICENSE-2.0 |
||||
|
||||
.. Unless required by applicable law or agreed to in writing, |
||||
software distributed under the License is distributed on an |
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
KIND, either express or implied. See the License for the |
||||
specific language governing permissions and limitations |
||||
under the License. |
||||
|
||||
Tasks |
||||
===== |
||||
|
||||
In this section |
||||
|
||||
.. toctree:: |
||||
:maxdepth: 1 |
||||
|
||||
shell |
||||
sql |
||||
python |
||||
http |
||||
|
||||
switch |
||||
condition |
||||
dependent |
||||
|
||||
spark |
||||
flink |
||||
map_reduce |
||||
procedure |
||||
|
||||
datax |
||||
sub_process |
||||
@ -0,0 +1,34 @@
|
||||
.. Licensed to the Apache Software Foundation (ASF) under one |
||||
or more contributor license agreements. See the NOTICE file |
||||
distributed with this work for additional information |
||||
regarding copyright ownership. The ASF licenses this file |
||||
to you under the Apache License, Version 2.0 (the |
||||
"License"); you may not use this file except in compliance |
||||
with the License. You may obtain a copy of the License at |
||||
|
||||
.. http://www.apache.org/licenses/LICENSE-2.0 |
||||
|
||||
.. Unless required by applicable law or agreed to in writing, |
||||
software distributed under the License is distributed on an |
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
KIND, either express or implied. See the License for the |
||||
specific language governing permissions and limitations |
||||
under the License. |
||||
|
||||
Map Reduce |
||||
========== |
||||
|
||||
|
||||
A Map Reduce task type's example and dive into information of **PyDolphinScheduler**. |
||||
|
||||
Example |
||||
------- |
||||
|
||||
.. literalinclude:: ../../../src/pydolphinscheduler/examples/task_map_reduce_example.py |
||||
:start-after: [start workflow_declare] |
||||
:end-before: [end workflow_declare] |
||||
|
||||
Dive Into |
||||
--------- |
||||
|
||||
.. automodule:: pydolphinscheduler.tasks.map_reduce |
||||
@ -0,0 +1,21 @@
|
||||
.. Licensed to the Apache Software Foundation (ASF) under one |
||||
or more contributor license agreements. See the NOTICE file |
||||
distributed with this work for additional information |
||||
regarding copyright ownership. The ASF licenses this file |
||||
to you under the Apache License, Version 2.0 (the |
||||
"License"); you may not use this file except in compliance |
||||
with the License. You may obtain a copy of the License at |
||||
|
||||
.. http://www.apache.org/licenses/LICENSE-2.0 |
||||
|
||||
.. Unless required by applicable law or agreed to in writing, |
||||
software distributed under the License is distributed on an |
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
KIND, either express or implied. See the License for the |
||||
specific language governing permissions and limitations |
||||
under the License. |
||||
|
||||
Procedure |
||||
========= |
||||
|
||||
.. automodule:: pydolphinscheduler.tasks.procedure |
||||
@ -0,0 +1,21 @@
|
||||
.. Licensed to the Apache Software Foundation (ASF) under one |
||||
or more contributor license agreements. See the NOTICE file |
||||
distributed with this work for additional information |
||||
regarding copyright ownership. The ASF licenses this file |
||||
to you under the Apache License, Version 2.0 (the |
||||
"License"); you may not use this file except in compliance |
||||
with the License. You may obtain a copy of the License at |
||||
|
||||
.. http://www.apache.org/licenses/LICENSE-2.0 |
||||
|
||||
.. Unless required by applicable law or agreed to in writing, |
||||
software distributed under the License is distributed on an |
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
KIND, either express or implied. See the License for the |
||||
specific language governing permissions and limitations |
||||
under the License. |
||||
|
||||
Python |
||||
====== |
||||
|
||||
.. automodule:: pydolphinscheduler.tasks.python |
||||
@ -0,0 +1,33 @@
|
||||
.. Licensed to the Apache Software Foundation (ASF) under one |
||||
or more contributor license agreements. See the NOTICE file |
||||
distributed with this work for additional information |
||||
regarding copyright ownership. The ASF licenses this file |
||||
to you under the Apache License, Version 2.0 (the |
||||
"License"); you may not use this file except in compliance |
||||
with the License. You may obtain a copy of the License at |
||||
|
||||
.. http://www.apache.org/licenses/LICENSE-2.0 |
||||
|
||||
.. Unless required by applicable law or agreed to in writing, |
||||
software distributed under the License is distributed on an |
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
KIND, either express or implied. See the License for the |
||||
specific language governing permissions and limitations |
||||
under the License. |
||||
|
||||
Shell |
||||
===== |
||||
|
||||
A shell task type's example and dive into information of **PyDolphinScheduler**. |
||||
|
||||
Example |
||||
------- |
||||
|
||||
.. literalinclude:: ../../../src/pydolphinscheduler/examples/tutorial.py |
||||
:start-after: [start workflow_declare] |
||||
:end-before: [end task_relation_declare] |
||||
|
||||
Dive Into |
||||
--------- |
||||
|
||||
.. automodule:: pydolphinscheduler.tasks.shell |
||||
@ -0,0 +1,33 @@
|
||||
.. Licensed to the Apache Software Foundation (ASF) under one |
||||
or more contributor license agreements. See the NOTICE file |
||||
distributed with this work for additional information |
||||
regarding copyright ownership. The ASF licenses this file |
||||
to you under the Apache License, Version 2.0 (the |
||||
"License"); you may not use this file except in compliance |
||||
with the License. You may obtain a copy of the License at |
||||
|
||||
.. http://www.apache.org/licenses/LICENSE-2.0 |
||||
|
||||
.. Unless required by applicable law or agreed to in writing, |
||||
software distributed under the License is distributed on an |
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
KIND, either express or implied. See the License for the |
||||
specific language governing permissions and limitations |
||||
under the License. |
||||
|
||||
Spark |
||||
===== |
||||
|
||||
A spark task type's example and dive into information of **PyDolphinScheduler**. |
||||
|
||||
Example |
||||
------- |
||||
|
||||
.. literalinclude:: ../../../src/pydolphinscheduler/examples/task_spark_example.py |
||||
:start-after: [start workflow_declare] |
||||
:end-before: [end workflow_declare] |
||||
|
||||
Dive Into |
||||
--------- |
||||
|
||||
.. automodule:: pydolphinscheduler.tasks.spark |
||||
@ -0,0 +1,21 @@
|
||||
.. Licensed to the Apache Software Foundation (ASF) under one |
||||
or more contributor license agreements. See the NOTICE file |
||||
distributed with this work for additional information |
||||
regarding copyright ownership. The ASF licenses this file |
||||
to you under the Apache License, Version 2.0 (the |
||||
"License"); you may not use this file except in compliance |
||||
with the License. You may obtain a copy of the License at |
||||
|
||||
.. http://www.apache.org/licenses/LICENSE-2.0 |
||||
|
||||
.. Unless required by applicable law or agreed to in writing, |
||||
software distributed under the License is distributed on an |
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
KIND, either express or implied. See the License for the |
||||
specific language governing permissions and limitations |
||||
under the License. |
||||
|
||||
SQL |
||||
=== |
||||
|
||||
.. automodule:: pydolphinscheduler.tasks.sql |
||||
@ -0,0 +1,21 @@
|
||||
.. Licensed to the Apache Software Foundation (ASF) under one |
||||
or more contributor license agreements. See the NOTICE file |
||||
distributed with this work for additional information |
||||
regarding copyright ownership. The ASF licenses this file |
||||
to you under the Apache License, Version 2.0 (the |
||||
"License"); you may not use this file except in compliance |
||||
with the License. You may obtain a copy of the License at |
||||
|
||||
.. http://www.apache.org/licenses/LICENSE-2.0 |
||||
|
||||
.. Unless required by applicable law or agreed to in writing, |
||||
software distributed under the License is distributed on an |
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
KIND, either express or implied. See the License for the |
||||
specific language governing permissions and limitations |
||||
under the License. |
||||
|
||||
Sub Process |
||||
=========== |
||||
|
||||
.. automodule:: pydolphinscheduler.tasks.sub_process |
||||
@ -0,0 +1,33 @@
|
||||
.. Licensed to the Apache Software Foundation (ASF) under one |
||||
or more contributor license agreements. See the NOTICE file |
||||
distributed with this work for additional information |
||||
regarding copyright ownership. The ASF licenses this file |
||||
to you under the Apache License, Version 2.0 (the |
||||
"License"); you may not use this file except in compliance |
||||
with the License. You may obtain a copy of the License at |
||||
|
||||
.. http://www.apache.org/licenses/LICENSE-2.0 |
||||
|
||||
.. Unless required by applicable law or agreed to in writing, |
||||
software distributed under the License is distributed on an |
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
KIND, either express or implied. See the License for the |
||||
specific language governing permissions and limitations |
||||
under the License. |
||||
|
||||
Switch |
||||
====== |
||||
|
||||
A switch task type's example and dive into information of **PyDolphinScheduler**. |
||||
|
||||
Example |
||||
------- |
||||
|
||||
.. literalinclude:: ../../../src/pydolphinscheduler/examples/task_switch_example.py |
||||
:start-after: [start workflow_declare] |
||||
:end-before: [end workflow_declare] |
||||
|
||||
Dive Into |
||||
--------- |
||||
|
||||
.. automodule:: pydolphinscheduler.tasks.switch |
||||
@ -0,0 +1,150 @@
|
||||
.. Licensed to the Apache Software Foundation (ASF) under one |
||||
or more contributor license agreements. See the NOTICE file |
||||
distributed with this work for additional information |
||||
regarding copyright ownership. The ASF licenses this file |
||||
to you under the Apache License, Version 2.0 (the |
||||
"License"); you may not use this file except in compliance |
||||
with the License. You may obtain a copy of the License at |
||||
|
||||
.. http://www.apache.org/licenses/LICENSE-2.0 |
||||
|
||||
.. Unless required by applicable law or agreed to in writing, |
||||
software distributed under the License is distributed on an |
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
KIND, either express or implied. See the License for the |
||||
specific language governing permissions and limitations |
||||
under the License. |
||||
|
||||
Tutorial |
||||
======== |
||||
|
||||
This tutorial show you the basic concept of *PyDolphinScheduler* and tell all |
||||
things you should know before you submit or run your first workflow. If you |
||||
still not install *PyDolphinScheduler* and start Apache DolphinScheduler, you |
||||
could go and see :ref:`how to getting start PyDolphinScheduler <start:getting started>` |
||||
|
||||
Overview of Tutorial |
||||
-------------------- |
||||
|
||||
Here have an overview of our tutorial, and it look a little complex but do not |
||||
worry about that because we explain this example below as detailed as possible. |
||||
|
||||
.. literalinclude:: ../../src/pydolphinscheduler/examples/tutorial.py |
||||
:start-after: [start tutorial] |
||||
:end-before: [end tutorial] |
||||
|
||||
Import Necessary Module |
||||
----------------------- |
||||
|
||||
First of all, we should importing necessary module which we would use later just |
||||
like other Python package. We just create a minimum demo here, so we just import |
||||
:class:`pydolphinscheduler.core.process_definition` and |
||||
:class:`pydolphinscheduler.tasks.shell`. |
||||
|
||||
.. literalinclude:: ../../src/pydolphinscheduler/examples/tutorial.py |
||||
:start-after: [start package_import] |
||||
:end-before: [end package_import] |
||||
|
||||
If you want to use other task type you could click and |
||||
:doc:`see all tasks we support <tasks/index>` |
||||
|
||||
Process Definition Declaration |
||||
------------------------------ |
||||
|
||||
We should instantiate object after we import them from `import necessary module`_. |
||||
Here we declare basic arguments for process definition(aka, workflow). We define |
||||
the name of process definition, using `Python context manager`_ and it |
||||
**the only required argument** for object process definition. Beside that we also |
||||
declare three arguments named `schedule`, `start_time` which setting workflow schedule |
||||
interval and schedule start_time, and argument `tenant` which changing workflow's |
||||
task running user in the worker, :ref:`section tenant <concept:tenant>` in *PyDolphinScheduler* |
||||
:doc:`concept` page have more detail information. |
||||
|
||||
.. literalinclude:: ../../src/pydolphinscheduler/examples/tutorial.py |
||||
:start-after: [start workflow_declare] |
||||
:end-before: [end workflow_declare] |
||||
|
||||
We could find more detail about process definition in |
||||
:ref:`concept about process definition <concept:process definition>` if you interested in it. |
||||
For all arguments of object process definition, you could find in the |
||||
:class:`pydolphinscheduler.core.process_definition` api documentation. |
||||
|
||||
Task Declaration |
||||
---------------- |
||||
|
||||
Here we declare four tasks, and bot of them are simple task of |
||||
:class:`pydolphinscheduler.tasks.shell` which running `echo` command in terminal. |
||||
Beside the argument `command`, we also need setting argument `name` for each task *(not |
||||
only shell task, `name` is required for each type of task)*. |
||||
|
||||
.. literalinclude:: ../../src/pydolphinscheduler/examples/tutorial.py |
||||
:dedent: 0 |
||||
:start-after: [start task_declare] |
||||
:end-before: [end task_declare] |
||||
|
||||
Beside shell task, *PyDolphinScheduler* support multiple tasks and you could |
||||
find in :doc:`tasks/index`. |
||||
|
||||
Setting Task Dependence |
||||
----------------------- |
||||
|
||||
After we declare both process definition and task, we have one workflow with |
||||
four tasks, both all tasks is independent so that they would run in parallel. |
||||
We should reorder the sort and the dependence of tasks. It useful when we need |
||||
run prepare task before we run actual task or we need tasks running is specific |
||||
rule. We both support attribute `set_downstream` and `set_upstream`, or bitwise |
||||
operators `>>` and `<<`. |
||||
|
||||
In this example, we set task `task_parent` is the upstream task of task |
||||
`task_child_one` and `task_child_two`, and task `task_union` is the downstream |
||||
task of both these two task. |
||||
|
||||
.. literalinclude:: ../../src/pydolphinscheduler/examples/tutorial.py |
||||
:dedent: 0 |
||||
:start-after: [start task_relation_declare] |
||||
:end-before: [end task_relation_declare] |
||||
|
||||
Please notice that we could grouping some tasks and set dependence if they have |
||||
same downstream or upstream. We declare task `task_child_one` and `task_child_two` |
||||
as a group here, named as `task_group` and set task `task_parent` as upstream of |
||||
both of them. You could see more detail in :ref:`concept:Tasks Dependence` section in concept |
||||
documentation. |
||||
|
||||
Submit Or Run Workflow |
||||
---------------------- |
||||
|
||||
Now we finish our workflow definition, with task and task dependence, but all |
||||
these things are in local, we should let Apache DolphinScheduler daemon know what we |
||||
define our workflow. So the last thing we have to do here is submit our workflow to |
||||
Apache DolphinScheduler daemon. |
||||
|
||||
We here in the example using `ProcessDefinition` attribute `run` to submit workflow |
||||
to the daemon, and set the schedule time we just declare in `process definition declaration`_. |
||||
|
||||
Now, we could run the Python code like other Python script, for the basic usage run |
||||
:code:`python tutorial.py` to trigger and run it. |
||||
|
||||
.. literalinclude:: ../../src/pydolphinscheduler/examples/tutorial.py |
||||
:dedent: 0 |
||||
:start-after: [start submit_or_run] |
||||
:end-before: [end submit_or_run] |
||||
|
||||
If you not start your Apache DolphinScheduler server, you could find the way in |
||||
:ref:`start:start Python gateway server` and it would have more detail about related server |
||||
start. Beside attribute `run`, we have attribute `submit` for object `ProcessDefinition` |
||||
and it just submit workflow to the daemon but not setting the schedule information. For |
||||
more detail you could see :ref:`concept:process definition`. |
||||
|
||||
DAG Graph After Tutorial Run |
||||
---------------------------- |
||||
|
||||
After we run the tutorial code, you could login Apache DolphinScheduler web UI, |
||||
go and see the `DolphinScheduler project page`_. they is a new process definition be |
||||
created and named "Tutorial". It create by *PyDolphinScheduler* and the DAG graph as below |
||||
|
||||
.. literalinclude:: ../../src/pydolphinscheduler/examples/tutorial.py |
||||
:language: text |
||||
:lines: 24-28 |
||||
|
||||
.. _`DolphinScheduler project page`: https://dolphinscheduler.apache.org/en-us/docs/latest/user_doc/guide/project.html |
||||
.. _`Python context manager`: https://docs.python.org/3/library/stdtypes.html#context-manager-types |
||||
@ -0,0 +1,95 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one |
||||
# or more contributor license agreements. See the NOTICE file |
||||
# distributed with this work for additional information |
||||
# regarding copyright ownership. The ASF licenses this file |
||||
# to you under the Apache License, Version 2.0 (the |
||||
# "License"); you may not use this file except in compliance |
||||
# with the License. You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, |
||||
# software distributed under the License is distributed on an |
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
# KIND, either express or implied. See the License for the |
||||
# specific language governing permissions and limitations |
||||
# under the License. |
||||
|
||||
"""Module engine.""" |
||||
|
||||
from typing import Dict, Optional |
||||
|
||||
from py4j.protocol import Py4JJavaError |
||||
|
||||
from pydolphinscheduler.core.task import Task |
||||
from pydolphinscheduler.exceptions import PyDSParamException |
||||
from pydolphinscheduler.java_gateway import launch_gateway |
||||
|
||||
|
||||
class ProgramType(str): |
||||
"""Type of program engine runs, for now it just contain `JAVA`, `SCALA` and `PYTHON`.""" |
||||
|
||||
JAVA = "JAVA" |
||||
SCALA = "SCALA" |
||||
PYTHON = "PYTHON" |
||||
|
||||
|
||||
class Engine(Task): |
||||
"""Task engine object, declare behavior for engine task to dolphinscheduler. |
||||
|
||||
This is the parent class of spark, flink and mr tasks, |
||||
and is used to provide the programType, mainClass and mainJar task parameters for reuse. |
||||
""" |
||||
|
||||
def __init__( |
||||
self, |
||||
name: str, |
||||
task_type: str, |
||||
main_class: str, |
||||
main_package: str, |
||||
program_type: Optional[ProgramType] = ProgramType.SCALA, |
||||
*args, |
||||
**kwargs |
||||
): |
||||
super().__init__(name, task_type, *args, **kwargs) |
||||
self.main_class = main_class |
||||
self.main_package = main_package |
||||
self.program_type = program_type |
||||
self._resource = {} |
||||
|
||||
def get_resource_info(self, program_type, main_package): |
||||
"""Get resource info from java gateway, contains resource id, name.""" |
||||
if self._resource: |
||||
return self._resource |
||||
else: |
||||
gateway = launch_gateway() |
||||
try: |
||||
self._resource = gateway.entry_point.getResourcesFileInfo( |
||||
program_type, main_package |
||||
) |
||||
# Handler source do not exists error, for now we just terminate the process. |
||||
except Py4JJavaError as ex: |
||||
raise PyDSParamException(str(ex.java_exception)) |
||||
return self._resource |
||||
|
||||
def get_jar_id(self) -> int: |
||||
"""Get jar id from java gateway, a wrapper for :func:`get_resource_info`.""" |
||||
return self.get_resource_info(self.program_type, self.main_package).get("id") |
||||
|
||||
@property |
||||
def task_params(self, camel_attr: bool = True, custom_attr: set = None) -> Dict: |
||||
"""Override Task.task_params for engine children task. |
||||
|
||||
children task have some specials attribute for task_params, and is odd if we |
||||
directly set as python property, so we Override Task.task_params here. |
||||
""" |
||||
params = super().task_params |
||||
custom_params = { |
||||
"programType": self.program_type, |
||||
"mainClass": self.main_class, |
||||
"mainJar": { |
||||
"id": self.get_jar_id(), |
||||
}, |
||||
} |
||||
params.update(custom_params) |
||||
return params |
||||
@ -0,0 +1,33 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one |
||||
# or more contributor license agreements. See the NOTICE file |
||||
# distributed with this work for additional information |
||||
# regarding copyright ownership. The ASF licenses this file |
||||
# to you under the Apache License, Version 2.0 (the |
||||
# "License"); you may not use this file except in compliance |
||||
# with the License. You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, |
||||
# software distributed under the License is distributed on an |
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
# KIND, either express or implied. See the License for the |
||||
# specific language governing permissions and limitations |
||||
# under the License. |
||||
|
||||
# [start workflow_declare] |
||||
"""A example workflow for task flink.""" |
||||
|
||||
from pydolphinscheduler.core.process_definition import ProcessDefinition |
||||
from pydolphinscheduler.tasks.flink import DeployMode, Flink, ProgramType |
||||
|
||||
with ProcessDefinition(name="task_flink_example", tenant="tenant_exists") as pd: |
||||
task = Flink( |
||||
name="task_flink", |
||||
main_class="org.apache.flink.streaming.examples.wordcount.WordCount", |
||||
main_package="WordCount.jar", |
||||
program_type=ProgramType.JAVA, |
||||
deploy_mode=DeployMode.LOCAL, |
||||
) |
||||
pd.run() |
||||
# [end workflow_declare] |
||||
@ -0,0 +1,34 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one |
||||
# or more contributor license agreements. See the NOTICE file |
||||
# distributed with this work for additional information |
||||
# regarding copyright ownership. The ASF licenses this file |
||||
# to you under the Apache License, Version 2.0 (the |
||||
# "License"); you may not use this file except in compliance |
||||
# with the License. You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, |
||||
# software distributed under the License is distributed on an |
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
# KIND, either express or implied. See the License for the |
||||
# specific language governing permissions and limitations |
||||
# under the License. |
||||
|
||||
# [start workflow_declare] |
||||
"""A example workflow for task mr.""" |
||||
|
||||
from pydolphinscheduler.core.engine import ProgramType |
||||
from pydolphinscheduler.core.process_definition import ProcessDefinition |
||||
from pydolphinscheduler.tasks.map_reduce import MR |
||||
|
||||
with ProcessDefinition(name="task_map_reduce_example", tenant="tenant_exists") as pd: |
||||
task = MR( |
||||
name="task_mr", |
||||
main_class="wordcount", |
||||
main_package="hadoop-mapreduce-examples-3.3.1.jar", |
||||
program_type=ProgramType.JAVA, |
||||
main_args="/dolphinscheduler/tenant_exists/resources/file.txt /output/ds", |
||||
) |
||||
pd.run() |
||||
# [end workflow_declare] |
||||
@ -0,0 +1,33 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one |
||||
# or more contributor license agreements. See the NOTICE file |
||||
# distributed with this work for additional information |
||||
# regarding copyright ownership. The ASF licenses this file |
||||
# to you under the Apache License, Version 2.0 (the |
||||
# "License"); you may not use this file except in compliance |
||||
# with the License. You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, |
||||
# software distributed under the License is distributed on an |
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
# KIND, either express or implied. See the License for the |
||||
# specific language governing permissions and limitations |
||||
# under the License. |
||||
|
||||
# [start workflow_declare] |
||||
"""A example workflow for task spark.""" |
||||
|
||||
from pydolphinscheduler.core.process_definition import ProcessDefinition |
||||
from pydolphinscheduler.tasks.spark import DeployMode, ProgramType, Spark |
||||
|
||||
with ProcessDefinition(name="task_spark_example", tenant="tenant_exists") as pd: |
||||
task = Spark( |
||||
name="task_spark", |
||||
main_class="org.apache.spark.examples.SparkPi", |
||||
main_package="spark-examples_2.12-3.2.0.jar", |
||||
program_type=ProgramType.JAVA, |
||||
deploy_mode=DeployMode.LOCAL, |
||||
) |
||||
pd.run() |
||||
# [end workflow_declare] |
||||
@ -0,0 +1,93 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one |
||||
# or more contributor license agreements. See the NOTICE file |
||||
# distributed with this work for additional information |
||||
# regarding copyright ownership. The ASF licenses this file |
||||
# to you under the Apache License, Version 2.0 (the |
||||
# "License"); you may not use this file except in compliance |
||||
# with the License. You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, |
||||
# software distributed under the License is distributed on an |
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
# KIND, either express or implied. See the License for the |
||||
# specific language governing permissions and limitations |
||||
# under the License. |
||||
|
||||
"""Task Flink.""" |
||||
|
||||
from typing import Optional |
||||
|
||||
from pydolphinscheduler.constants import TaskType |
||||
from pydolphinscheduler.core.engine import Engine, ProgramType |
||||
|
||||
|
||||
class FlinkVersion(str): |
||||
"""Flink version, for now it just contain `HIGHT` and `LOW`.""" |
||||
|
||||
LOW_VERSION = "<1.10" |
||||
HIGHT_VERSION = ">=1.10" |
||||
|
||||
|
||||
class DeployMode(str): |
||||
"""Flink deploy mode, for now it just contain `LOCAL` and `CLUSTER`.""" |
||||
|
||||
LOCAL = "local" |
||||
CLUSTER = "cluster" |
||||
|
||||
|
||||
class Flink(Engine): |
||||
"""Task flink object, declare behavior for flink task to dolphinscheduler.""" |
||||
|
||||
_task_custom_attr = { |
||||
"deploy_mode", |
||||
"flink_version", |
||||
"slot", |
||||
"task_manager", |
||||
"job_manager_memory", |
||||
"task_manager_memory", |
||||
"app_name", |
||||
"parallelism", |
||||
"main_args", |
||||
"others", |
||||
} |
||||
|
||||
def __init__( |
||||
self, |
||||
name: str, |
||||
main_class: str, |
||||
main_package: str, |
||||
program_type: Optional[ProgramType] = ProgramType.SCALA, |
||||
deploy_mode: Optional[DeployMode] = DeployMode.CLUSTER, |
||||
flink_version: Optional[FlinkVersion] = FlinkVersion.LOW_VERSION, |
||||
app_name: Optional[str] = None, |
||||
job_manager_memory: Optional[str] = "1G", |
||||
task_manager_memory: Optional[str] = "2G", |
||||
slot: Optional[int] = 1, |
||||
task_manager: Optional[int] = 2, |
||||
parallelism: Optional[int] = 1, |
||||
main_args: Optional[str] = None, |
||||
others: Optional[str] = None, |
||||
*args, |
||||
**kwargs |
||||
): |
||||
super().__init__( |
||||
name, |
||||
TaskType.FLINK, |
||||
main_class, |
||||
main_package, |
||||
program_type, |
||||
*args, |
||||
**kwargs |
||||
) |
||||
self.deploy_mode = deploy_mode |
||||
self.flink_version = flink_version |
||||
self.app_name = app_name |
||||
self.job_manager_memory = job_manager_memory |
||||
self.task_manager_memory = task_manager_memory |
||||
self.slot = slot |
||||
self.task_manager = task_manager |
||||
self.parallelism = parallelism |
||||
self.main_args = main_args |
||||
self.others = others |
||||
@ -0,0 +1,52 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one |
||||
# or more contributor license agreements. See the NOTICE file |
||||
# distributed with this work for additional information |
||||
# regarding copyright ownership. The ASF licenses this file |
||||
# to you under the Apache License, Version 2.0 (the |
||||
# "License"); you may not use this file except in compliance |
||||
# with the License. You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, |
||||
# software distributed under the License is distributed on an |
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
# KIND, either express or implied. See the License for the |
||||
# specific language governing permissions and limitations |
||||
# under the License. |
||||
|
||||
"""Task MR.""" |
||||
|
||||
from typing import Optional |
||||
|
||||
from pydolphinscheduler.constants import TaskType |
||||
from pydolphinscheduler.core.engine import Engine, ProgramType |
||||
|
||||
|
||||
class MR(Engine): |
||||
"""Task mr object, declare behavior for mr task to dolphinscheduler.""" |
||||
|
||||
_task_custom_attr = { |
||||
"app_name", |
||||
"main_args", |
||||
"others", |
||||
} |
||||
|
||||
def __init__( |
||||
self, |
||||
name: str, |
||||
main_class: str, |
||||
main_package: str, |
||||
program_type: Optional[ProgramType] = ProgramType.SCALA, |
||||
app_name: Optional[str] = None, |
||||
main_args: Optional[str] = None, |
||||
others: Optional[str] = None, |
||||
*args, |
||||
**kwargs |
||||
): |
||||
super().__init__( |
||||
name, TaskType.MR, main_class, main_package, program_type, *args, **kwargs |
||||
) |
||||
self.app_name = app_name |
||||
self.main_args = main_args |
||||
self.others = others |
||||
@ -0,0 +1,94 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one |
||||
# or more contributor license agreements. See the NOTICE file |
||||
# distributed with this work for additional information |
||||
# regarding copyright ownership. The ASF licenses this file |
||||
# to you under the Apache License, Version 2.0 (the |
||||
# "License"); you may not use this file except in compliance |
||||
# with the License. You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, |
||||
# software distributed under the License is distributed on an |
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
# KIND, either express or implied. See the License for the |
||||
# specific language governing permissions and limitations |
||||
# under the License. |
||||
|
||||
"""Task Spark.""" |
||||
|
||||
from typing import Optional |
||||
|
||||
from pydolphinscheduler.constants import TaskType |
||||
from pydolphinscheduler.core.engine import Engine, ProgramType |
||||
|
||||
|
||||
class SparkVersion(str): |
||||
"""Spark version, for now it just contain `SPARK1` and `SPARK2`.""" |
||||
|
||||
SPARK1 = "SPARK1" |
||||
SPARK2 = "SPARK2" |
||||
|
||||
|
||||
class DeployMode(str): |
||||
"""SPARK deploy mode, for now it just contain `LOCAL`, `CLIENT` and `CLUSTER`.""" |
||||
|
||||
LOCAL = "local" |
||||
CLIENT = "client" |
||||
CLUSTER = "cluster" |
||||
|
||||
|
||||
class Spark(Engine): |
||||
"""Task spark object, declare behavior for spark task to dolphinscheduler.""" |
||||
|
||||
_task_custom_attr = { |
||||
"deploy_mode", |
||||
"spark_version", |
||||
"driver_cores", |
||||
"driver_memory", |
||||
"num_executors", |
||||
"executor_memory", |
||||
"executor_cores", |
||||
"app_name", |
||||
"main_args", |
||||
"others", |
||||
} |
||||
|
||||
def __init__( |
||||
self, |
||||
name: str, |
||||
main_class: str, |
||||
main_package: str, |
||||
program_type: Optional[ProgramType] = ProgramType.SCALA, |
||||
deploy_mode: Optional[DeployMode] = DeployMode.CLUSTER, |
||||
spark_version: Optional[SparkVersion] = SparkVersion.SPARK2, |
||||
app_name: Optional[str] = None, |
||||
driver_cores: Optional[int] = 1, |
||||
driver_memory: Optional[str] = "512M", |
||||
num_executors: Optional[int] = 2, |
||||
executor_memory: Optional[str] = "2G", |
||||
executor_cores: Optional[int] = 2, |
||||
main_args: Optional[str] = None, |
||||
others: Optional[str] = None, |
||||
*args, |
||||
**kwargs |
||||
): |
||||
super().__init__( |
||||
name, |
||||
TaskType.SPARK, |
||||
main_class, |
||||
main_package, |
||||
program_type, |
||||
*args, |
||||
**kwargs |
||||
) |
||||
self.deploy_mode = deploy_mode |
||||
self.spark_version = spark_version |
||||
self.app_name = app_name |
||||
self.driver_cores = driver_cores |
||||
self.driver_memory = driver_memory |
||||
self.num_executors = num_executors |
||||
self.executor_memory = executor_memory |
||||
self.executor_cores = executor_cores |
||||
self.main_args = main_args |
||||
self.others = others |
||||
@ -0,0 +1,147 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one |
||||
# or more contributor license agreements. See the NOTICE file |
||||
# distributed with this work for additional information |
||||
# regarding copyright ownership. The ASF licenses this file |
||||
# to you under the Apache License, Version 2.0 (the |
||||
# "License"); you may not use this file except in compliance |
||||
# with the License. You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, |
||||
# software distributed under the License is distributed on an |
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
# KIND, either express or implied. See the License for the |
||||
# specific language governing permissions and limitations |
||||
# under the License. |
||||
|
||||
"""Test Task Engine.""" |
||||
|
||||
|
||||
from unittest.mock import patch |
||||
|
||||
import pytest |
||||
|
||||
from pydolphinscheduler.core.engine import Engine, ProgramType |
||||
|
||||
TEST_ENGINE_TASK_TYPE = "ENGINE" |
||||
TEST_MAIN_CLASS = "org.apache.examples.mock.Mock" |
||||
TEST_MAIN_PACKAGE = "Mock.jar" |
||||
TEST_PROGRAM_TYPE = ProgramType.JAVA |
||||
|
||||
|
||||
@patch( |
||||
"pydolphinscheduler.core.task.Task.gen_code_and_version", |
||||
return_value=(123, 1), |
||||
) |
||||
@patch( |
||||
"pydolphinscheduler.core.engine.Engine.get_resource_info", |
||||
return_value=({"id": 1, "name": "mock_name"}), |
||||
) |
||||
def test_get_jar_detail(mock_resource, mock_code_version): |
||||
"""Test :func:`get_jar_id` can return expect value.""" |
||||
name = "test_get_jar_detail" |
||||
task = Engine( |
||||
name, |
||||
TEST_ENGINE_TASK_TYPE, |
||||
TEST_MAIN_CLASS, |
||||
TEST_MAIN_PACKAGE, |
||||
TEST_PROGRAM_TYPE, |
||||
) |
||||
assert 1 == task.get_jar_id() |
||||
|
||||
|
||||
@pytest.mark.parametrize( |
||||
"attr, expect", |
||||
[ |
||||
( |
||||
{ |
||||
"name": "test-task-params", |
||||
"task_type": "test-engine", |
||||
"main_class": "org.apache.examples.mock.Mock", |
||||
"main_package": "TestMock.jar", |
||||
"program_type": ProgramType.JAVA, |
||||
}, |
||||
{ |
||||
"mainClass": "org.apache.examples.mock.Mock", |
||||
"mainJar": { |
||||
"id": 1, |
||||
}, |
||||
"programType": ProgramType.JAVA, |
||||
"localParams": [], |
||||
"resourceList": [], |
||||
"dependence": {}, |
||||
"conditionResult": {"successNode": [""], "failedNode": [""]}, |
||||
"waitStartTimeout": {}, |
||||
}, |
||||
) |
||||
], |
||||
) |
||||
@patch( |
||||
"pydolphinscheduler.core.task.Task.gen_code_and_version", |
||||
return_value=(123, 1), |
||||
) |
||||
@patch( |
||||
"pydolphinscheduler.core.engine.Engine.get_resource_info", |
||||
return_value=({"id": 1, "name": "mock_name"}), |
||||
) |
||||
def test_property_task_params(mock_resource, mock_code_version, attr, expect): |
||||
"""Test task engine task property.""" |
||||
task = Engine(**attr) |
||||
assert expect == task.task_params |
||||
|
||||
|
||||
@pytest.mark.parametrize( |
||||
"attr, expect", |
||||
[ |
||||
( |
||||
{ |
||||
"name": "test-task-test_engine_get_define", |
||||
"task_type": "test-engine", |
||||
"main_class": "org.apache.examples.mock.Mock", |
||||
"main_package": "TestMock.jar", |
||||
"program_type": ProgramType.JAVA, |
||||
}, |
||||
{ |
||||
"code": 123, |
||||
"name": "test-task-test_engine_get_define", |
||||
"version": 1, |
||||
"description": None, |
||||
"delayTime": 0, |
||||
"taskType": "test-engine", |
||||
"taskParams": { |
||||
"mainClass": "org.apache.examples.mock.Mock", |
||||
"mainJar": { |
||||
"id": 1, |
||||
}, |
||||
"programType": ProgramType.JAVA, |
||||
"localParams": [], |
||||
"resourceList": [], |
||||
"dependence": {}, |
||||
"conditionResult": {"successNode": [""], "failedNode": [""]}, |
||||
"waitStartTimeout": {}, |
||||
}, |
||||
"flag": "YES", |
||||
"taskPriority": "MEDIUM", |
||||
"workerGroup": "default", |
||||
"failRetryTimes": 0, |
||||
"failRetryInterval": 1, |
||||
"timeoutFlag": "CLOSE", |
||||
"timeoutNotifyStrategy": None, |
||||
"timeout": 0, |
||||
}, |
||||
) |
||||
], |
||||
) |
||||
@patch( |
||||
"pydolphinscheduler.core.task.Task.gen_code_and_version", |
||||
return_value=(123, 1), |
||||
) |
||||
@patch( |
||||
"pydolphinscheduler.core.engine.Engine.get_resource_info", |
||||
return_value=({"id": 1, "name": "mock_name"}), |
||||
) |
||||
def test_engine_get_define(mock_resource, mock_code_version, attr, expect): |
||||
"""Test task engine function get_define.""" |
||||
task = Engine(**attr) |
||||
assert task.get_define() == expect |
||||
@ -0,0 +1,172 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one |
||||
# or more contributor license agreements. See the NOTICE file |
||||
# distributed with this work for additional information |
||||
# regarding copyright ownership. The ASF licenses this file |
||||
# to you under the Apache License, Version 2.0 (the |
||||
# "License"); you may not use this file except in compliance |
||||
# with the License. You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, |
||||
# software distributed under the License is distributed on an |
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
# KIND, either express or implied. See the License for the |
||||
# specific language governing permissions and limitations |
||||
# under the License. |
||||
|
||||
"""Test example.""" |
||||
|
||||
import ast |
||||
import importlib |
||||
from unittest.mock import patch |
||||
|
||||
import pytest |
||||
|
||||
from tests.testing.constants import task_without_example |
||||
from tests.testing.path import get_all_examples, get_tasks |
||||
from tests.testing.task import Task |
||||
|
||||
process_definition_name = set() |
||||
|
||||
|
||||
def import_module(script_name, script_path): |
||||
"""Import and run example module in examples directory.""" |
||||
spec = importlib.util.spec_from_file_location(script_name, script_path) |
||||
module = importlib.util.module_from_spec(spec) |
||||
spec.loader.exec_module(module) |
||||
return module |
||||
|
||||
|
||||
def test_task_without_example(): |
||||
"""Test task which without example. |
||||
|
||||
Avoiding add new type of tasks but without adding example describe how to use it. |
||||
""" |
||||
# We use example/tutorial.py as shell task example |
||||
ignore_name = {"__init__.py", "shell.py"} |
||||
all_tasks = {task.stem for task in get_tasks(ignore_name=ignore_name)} |
||||
|
||||
have_example_tasks = set() |
||||
start = "task_" |
||||
end = "_example" |
||||
for ex in get_all_examples(): |
||||
stem = ex.stem |
||||
if stem.startswith(start) and stem.endswith(end): |
||||
task_name = stem.replace(start, "").replace(end, "") |
||||
have_example_tasks.add(task_name) |
||||
|
||||
assert all_tasks.difference(have_example_tasks) == task_without_example |
||||
|
||||
|
||||
@pytest.fixture |
||||
def setup_and_teardown_for_stuff(): |
||||
"""Fixture of py.test handle setup and teardown.""" |
||||
yield |
||||
global process_definition_name |
||||
process_definition_name = set() |
||||
|
||||
|
||||
def submit_check_without_same_name(self): |
||||
"""Side effect for verifying process definition name and adding it to global variable.""" |
||||
if self.name in process_definition_name: |
||||
raise ValueError( |
||||
"Example process definition should not have same name, but get duplicate name: %s", |
||||
self.name, |
||||
) |
||||
submit_add_process_definition(self) |
||||
|
||||
|
||||
def submit_add_process_definition(self): |
||||
"""Side effect for adding process definition name to global variable.""" |
||||
process_definition_name.add(self.name) |
||||
|
||||
|
||||
def test_example_basic(): |
||||
"""Test example basic information. |
||||
|
||||
Which including: |
||||
* File extension name is `.py` |
||||
* All example except `tutorial.py` is end with keyword "_example" |
||||
* All example must have not empty `__doc__`. |
||||
""" |
||||
for ex in get_all_examples(): |
||||
# All files in example is python script |
||||
assert ( |
||||
ex.suffix == ".py" |
||||
), f"We expect all examples is python script, but get {ex.name}." |
||||
|
||||
# All except tutorial and __init__ is end with keyword "_example" |
||||
if ex.stem != "tutorial" and ex.stem != "__init__": |
||||
assert ex.stem.endswith( |
||||
"_example" |
||||
), f"We expect all examples script end with keyword '_example', but get {ex.stem}." |
||||
|
||||
# All files have __doc__ |
||||
tree = ast.parse(ex.read_text()) |
||||
example_doc = ast.get_docstring(tree, clean=False) |
||||
assert ( |
||||
example_doc is not None |
||||
), f"We expect all examples have __doc__, but {ex.name} do not." |
||||
|
||||
|
||||
@patch("pydolphinscheduler.core.process_definition.ProcessDefinition.start") |
||||
@patch( |
||||
"pydolphinscheduler.core.process_definition.ProcessDefinition.submit", |
||||
side_effect=submit_check_without_same_name, |
||||
autospec=True, |
||||
) |
||||
@patch( |
||||
"pydolphinscheduler.core.task.Task.gen_code_and_version", |
||||
# Example bulk_create_example.py would create workflow dynamic by :func:`get_one_task_by_name` |
||||
# and would raise error in :func:`get_one_task_by_name` if we return constant value |
||||
# using :arg:`return_value` |
||||
side_effect=Task("test_example", "test_example").gen_code_and_version, |
||||
) |
||||
def test_example_process_definition_without_same_name( |
||||
mock_code_version, mock_submit, mock_start |
||||
): |
||||
"""Test all examples file without same process definition's name. |
||||
|
||||
Our process definition would compete with others if we have same process definition name. It will make |
||||
different between actually workflow and our workflow-as-code file which make users feel strange. |
||||
""" |
||||
for ex in get_all_examples(): |
||||
# We use side_effect `submit_check_without_same_name` overwrite :func:`submit` |
||||
# and check whether it have duplicate name or not |
||||
import_module(ex.name, str(ex)) |
||||
assert True |
||||
|
||||
|
||||
@patch("pydolphinscheduler.core.process_definition.ProcessDefinition.start") |
||||
@patch( |
||||
"pydolphinscheduler.core.process_definition.ProcessDefinition.submit", |
||||
side_effect=submit_add_process_definition, |
||||
autospec=True, |
||||
) |
||||
@patch( |
||||
"pydolphinscheduler.core.task.Task.gen_code_and_version", |
||||
# Example bulk_create_example.py would create workflow dynamic by :func:`get_one_task_by_name` |
||||
# and would raise error in :func:`get_one_task_by_name` if we return constant value |
||||
# using :arg:`return_value` |
||||
side_effect=Task("test_example", "test_example").gen_code_and_version, |
||||
) |
||||
def test_file_name_in_process_definition(mock_code_version, mock_submit, mock_start): |
||||
"""Test example file name in example definition name. |
||||
|
||||
We should not directly assert equal, because some of the examples contain |
||||
more than one process definition. |
||||
""" |
||||
global process_definition_name |
||||
for ex in get_all_examples(): |
||||
# Skip __init__ file |
||||
if ex.stem == "__init__": |
||||
continue |
||||
# Skip bulk_create_example check, cause it contain multiple workflow and |
||||
# without one named bulk_create_example |
||||
if ex.stem == "bulk_create_example": |
||||
continue |
||||
process_definition_name = set() |
||||
assert ex.stem not in process_definition_name |
||||
import_module(ex.name, str(ex)) |
||||
assert ex.stem in process_definition_name |
||||
@ -0,0 +1,82 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one |
||||
# or more contributor license agreements. See the NOTICE file |
||||
# distributed with this work for additional information |
||||
# regarding copyright ownership. The ASF licenses this file |
||||
# to you under the Apache License, Version 2.0 (the |
||||
# "License"); you may not use this file except in compliance |
||||
# with the License. You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, |
||||
# software distributed under the License is distributed on an |
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
# KIND, either express or implied. See the License for the |
||||
# specific language governing permissions and limitations |
||||
# under the License. |
||||
|
||||
"""Test Task Flink.""" |
||||
|
||||
from unittest.mock import patch |
||||
|
||||
from pydolphinscheduler.tasks.flink import DeployMode, Flink, FlinkVersion, ProgramType |
||||
|
||||
|
||||
@patch( |
||||
"pydolphinscheduler.core.engine.Engine.get_resource_info", |
||||
return_value=({"id": 1, "name": "test"}), |
||||
) |
||||
def test_flink_get_define(mock_resource): |
||||
"""Test task flink function get_define.""" |
||||
code = 123 |
||||
version = 1 |
||||
name = "test_flink_get_define" |
||||
main_class = "org.apache.flink.test_main_class" |
||||
main_package = "test_main_package" |
||||
program_type = ProgramType.JAVA |
||||
deploy_mode = DeployMode.LOCAL |
||||
|
||||
expect = { |
||||
"code": code, |
||||
"name": name, |
||||
"version": 1, |
||||
"description": None, |
||||
"delayTime": 0, |
||||
"taskType": "FLINK", |
||||
"taskParams": { |
||||
"mainClass": main_class, |
||||
"mainJar": { |
||||
"id": 1, |
||||
}, |
||||
"programType": program_type, |
||||
"deployMode": deploy_mode, |
||||
"flinkVersion": FlinkVersion.LOW_VERSION, |
||||
"slot": 1, |
||||
"parallelism": 1, |
||||
"taskManager": 2, |
||||
"jobManagerMemory": "1G", |
||||
"taskManagerMemory": "2G", |
||||
"appName": None, |
||||
"mainArgs": None, |
||||
"others": None, |
||||
"localParams": [], |
||||
"resourceList": [], |
||||
"dependence": {}, |
||||
"conditionResult": {"successNode": [""], "failedNode": [""]}, |
||||
"waitStartTimeout": {}, |
||||
}, |
||||
"flag": "YES", |
||||
"taskPriority": "MEDIUM", |
||||
"workerGroup": "default", |
||||
"failRetryTimes": 0, |
||||
"failRetryInterval": 1, |
||||
"timeoutFlag": "CLOSE", |
||||
"timeoutNotifyStrategy": None, |
||||
"timeout": 0, |
||||
} |
||||
with patch( |
||||
"pydolphinscheduler.core.task.Task.gen_code_and_version", |
||||
return_value=(code, version), |
||||
): |
||||
task = Flink(name, main_class, main_package, program_type, deploy_mode) |
||||
assert task.get_define() == expect |
||||
@ -0,0 +1,75 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one |
||||
# or more contributor license agreements. See the NOTICE file |
||||
# distributed with this work for additional information |
||||
# regarding copyright ownership. The ASF licenses this file |
||||
# to you under the Apache License, Version 2.0 (the |
||||
# "License"); you may not use this file except in compliance |
||||
# with the License. You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, |
||||
# software distributed under the License is distributed on an |
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
# KIND, either express or implied. See the License for the |
||||
# specific language governing permissions and limitations |
||||
# under the License. |
||||
|
||||
"""Test Task MR.""" |
||||
|
||||
from unittest.mock import patch |
||||
|
||||
from pydolphinscheduler.tasks.map_reduce import MR, ProgramType |
||||
|
||||
|
||||
@patch( |
||||
"pydolphinscheduler.core.engine.Engine.get_resource_info", |
||||
return_value=({"id": 1, "name": "test"}), |
||||
) |
||||
def test_mr_get_define(mock_resource): |
||||
"""Test task mr function get_define.""" |
||||
code = 123 |
||||
version = 1 |
||||
name = "test_mr_get_define" |
||||
main_class = "org.apache.mr.test_main_class" |
||||
main_package = "test_main_package" |
||||
program_type = ProgramType.JAVA |
||||
main_args = "/dolphinscheduler/resources/file.txt /output/ds" |
||||
|
||||
expect = { |
||||
"code": code, |
||||
"name": name, |
||||
"version": 1, |
||||
"description": None, |
||||
"delayTime": 0, |
||||
"taskType": "MR", |
||||
"taskParams": { |
||||
"mainClass": main_class, |
||||
"mainJar": { |
||||
"id": 1, |
||||
}, |
||||
"programType": program_type, |
||||
"appName": None, |
||||
"mainArgs": main_args, |
||||
"others": None, |
||||
"localParams": [], |
||||
"resourceList": [], |
||||
"dependence": {}, |
||||
"conditionResult": {"successNode": [""], "failedNode": [""]}, |
||||
"waitStartTimeout": {}, |
||||
}, |
||||
"flag": "YES", |
||||
"taskPriority": "MEDIUM", |
||||
"workerGroup": "default", |
||||
"failRetryTimes": 0, |
||||
"failRetryInterval": 1, |
||||
"timeoutFlag": "CLOSE", |
||||
"timeoutNotifyStrategy": None, |
||||
"timeout": 0, |
||||
} |
||||
with patch( |
||||
"pydolphinscheduler.core.task.Task.gen_code_and_version", |
||||
return_value=(code, version), |
||||
): |
||||
task = MR(name, main_class, main_package, program_type, main_args=main_args) |
||||
assert task.get_define() == expect |
||||
@ -0,0 +1,82 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one |
||||
# or more contributor license agreements. See the NOTICE file |
||||
# distributed with this work for additional information |
||||
# regarding copyright ownership. The ASF licenses this file |
||||
# to you under the Apache License, Version 2.0 (the |
||||
# "License"); you may not use this file except in compliance |
||||
# with the License. You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, |
||||
# software distributed under the License is distributed on an |
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
# KIND, either express or implied. See the License for the |
||||
# specific language governing permissions and limitations |
||||
# under the License. |
||||
|
||||
"""Test Task Spark.""" |
||||
|
||||
from unittest.mock import patch |
||||
|
||||
from pydolphinscheduler.tasks.spark import DeployMode, ProgramType, Spark, SparkVersion |
||||
|
||||
|
||||
@patch( |
||||
"pydolphinscheduler.core.engine.Engine.get_resource_info", |
||||
return_value=({"id": 1, "name": "test"}), |
||||
) |
||||
def test_spark_get_define(mock_resource): |
||||
"""Test task spark function get_define.""" |
||||
code = 123 |
||||
version = 1 |
||||
name = "test_spark_get_define" |
||||
main_class = "org.apache.spark.test_main_class" |
||||
main_package = "test_main_package" |
||||
program_type = ProgramType.JAVA |
||||
deploy_mode = DeployMode.LOCAL |
||||
|
||||
expect = { |
||||
"code": code, |
||||
"name": name, |
||||
"version": 1, |
||||
"description": None, |
||||
"delayTime": 0, |
||||
"taskType": "SPARK", |
||||
"taskParams": { |
||||
"mainClass": main_class, |
||||
"mainJar": { |
||||
"id": 1, |
||||
}, |
||||
"programType": program_type, |
||||
"deployMode": deploy_mode, |
||||
"sparkVersion": SparkVersion.SPARK2, |
||||
"driverCores": 1, |
||||
"driverMemory": "512M", |
||||
"numExecutors": 2, |
||||
"executorMemory": "2G", |
||||
"executorCores": 2, |
||||
"appName": None, |
||||
"mainArgs": None, |
||||
"others": None, |
||||
"localParams": [], |
||||
"resourceList": [], |
||||
"dependence": {}, |
||||
"conditionResult": {"successNode": [""], "failedNode": [""]}, |
||||
"waitStartTimeout": {}, |
||||
}, |
||||
"flag": "YES", |
||||
"taskPriority": "MEDIUM", |
||||
"workerGroup": "default", |
||||
"failRetryTimes": 0, |
||||
"failRetryInterval": 1, |
||||
"timeoutFlag": "CLOSE", |
||||
"timeoutNotifyStrategy": None, |
||||
"timeout": 0, |
||||
} |
||||
with patch( |
||||
"pydolphinscheduler.core.task.Task.gen_code_and_version", |
||||
return_value=(code, version), |
||||
): |
||||
task = Spark(name, main_class, main_package, program_type, deploy_mode) |
||||
assert task.get_define() == expect |
||||
@ -0,0 +1,59 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one |
||||
# or more contributor license agreements. See the NOTICE file |
||||
# distributed with this work for additional information |
||||
# regarding copyright ownership. The ASF licenses this file |
||||
# to you under the Apache License, Version 2.0 (the |
||||
# "License"); you may not use this file except in compliance |
||||
# with the License. You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, |
||||
# software distributed under the License is distributed on an |
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
# KIND, either express or implied. See the License for the |
||||
# specific language governing permissions and limitations |
||||
# under the License. |
||||
|
||||
"""Test pydolphinscheduler docs.""" |
||||
|
||||
import re |
||||
|
||||
from tests.testing.constants import task_without_example |
||||
from tests.testing.path import get_doc_tasks, get_tasks |
||||
|
||||
ignore_code_file = {"__init__.py"} |
||||
ignore_doc_file = {"index.rst"} |
||||
|
||||
|
||||
def test_without_missing_task_rst(): |
||||
"""Test without missing any task document by compare filename. |
||||
|
||||
Avoiding add new type of tasks but without adding document about it. |
||||
""" |
||||
code_files = {p.stem for p in get_tasks(ignore_name=ignore_code_file)} |
||||
doc_files = {p.stem for p in get_doc_tasks(ignore_name=ignore_doc_file)} |
||||
assert code_files == doc_files |
||||
|
||||
|
||||
def test_task_without_example(): |
||||
"""Test task document which without example. |
||||
|
||||
Avoiding add new type of tasks but without adding example content describe how to use it. |
||||
""" |
||||
task_without_example_detected = set() |
||||
pattern = re.compile("Example\n-------") |
||||
|
||||
for doc in get_doc_tasks(ignore_name=ignore_doc_file): |
||||
search_result = pattern.search(doc.read_text()) |
||||
if not search_result: |
||||
task_without_example_detected.add(doc.stem) |
||||
assert task_without_example == task_without_example_detected |
||||
|
||||
|
||||
def test_doc_automodule_directive_name(): |
||||
"""Test task document with correct name in directive automodule.""" |
||||
pattern = re.compile(".. automodule:: (.*)") |
||||
for doc in get_doc_tasks(ignore_name=ignore_doc_file): |
||||
match_string = pattern.search(doc.read_text()).group(1) |
||||
assert f"pydolphinscheduler.tasks.{doc.stem}" == match_string |
||||
@ -0,0 +1,28 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one |
||||
# or more contributor license agreements. See the NOTICE file |
||||
# distributed with this work for additional information |
||||
# regarding copyright ownership. The ASF licenses this file |
||||
# to you under the Apache License, Version 2.0 (the |
||||
# "License"); you may not use this file except in compliance |
||||
# with the License. You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, |
||||
# software distributed under the License is distributed on an |
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
# KIND, either express or implied. See the License for the |
||||
# specific language governing permissions and limitations |
||||
# under the License. |
||||
|
||||
"""Constants variables for test module.""" |
||||
|
||||
# Record some task without example in directory `example`. Some of them maybe can not write example, |
||||
# but most of them just without adding by mistake, and we should add it later. |
||||
task_without_example = { |
||||
"sql", |
||||
"http", |
||||
"sub_process", |
||||
"python", |
||||
"procedure", |
||||
} |
||||
@ -0,0 +1,56 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one |
||||
# or more contributor license agreements. See the NOTICE file |
||||
# distributed with this work for additional information |
||||
# regarding copyright ownership. The ASF licenses this file |
||||
# to you under the Apache License, Version 2.0 (the |
||||
# "License"); you may not use this file except in compliance |
||||
# with the License. You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, |
||||
# software distributed under the License is distributed on an |
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||
# KIND, either express or implied. See the License for the |
||||
# specific language governing permissions and limitations |
||||
# under the License. |
||||
|
||||
"""Handle path related issue in test module.""" |
||||
|
||||
from pathlib import Path |
||||
from typing import Any, Generator |
||||
|
||||
path_code_tasks = Path(__file__).parent.parent.parent.joinpath( |
||||
"src", "pydolphinscheduler", "tasks" |
||||
) |
||||
path_example = Path(__file__).parent.parent.parent.joinpath( |
||||
"src", "pydolphinscheduler", "examples" |
||||
) |
||||
path_doc_tasks = Path(__file__).parent.parent.parent.joinpath("docs", "source", "tasks") |
||||
|
||||
|
||||
def get_all_examples() -> Generator[Path, Any, None]: |
||||
"""Get all examples files path in examples directory.""" |
||||
return (ex for ex in path_example.iterdir() if ex.is_file()) |
||||
|
||||
|
||||
def get_tasks(ignore_name: set = None) -> Generator[Path, Any, None]: |
||||
"""Get all tasks files path in src/pydolphinscheduler/tasks directory.""" |
||||
if not ignore_name: |
||||
ignore_name = set() |
||||
return ( |
||||
ex |
||||
for ex in path_code_tasks.iterdir() |
||||
if ex.is_file() and ex.name not in ignore_name |
||||
) |
||||
|
||||
|
||||
def get_doc_tasks(ignore_name: set = None) -> Generator[Path, Any, None]: |
||||
"""Get all tasks document path in docs/source/tasks directory.""" |
||||
if not ignore_name: |
||||
ignore_name = set() |
||||
return ( |
||||
ex |
||||
for ex in path_doc_tasks.iterdir() |
||||
if ex.is_file() and ex.name not in ignore_name |
||||
) |
||||
@ -0,0 +1,93 @@
|
||||
/* |
||||
* Licensed to the Apache Software Foundation (ASF) under one or more |
||||
* contributor license agreements. See the NOTICE file distributed with |
||||
* this work for additional information regarding copyright ownership. |
||||
* The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
* (the "License"); you may not use this file except in compliance with |
||||
* the License. You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
package org.apache.dolphinscheduler.server.config; |
||||
|
||||
import org.springframework.context.annotation.PropertySource; |
||||
import org.springframework.stereotype.Component; |
||||
import org.springframework.beans.factory.annotation.Value; |
||||
|
||||
@Component |
||||
@PropertySource(value = "python-gateway.properties") |
||||
public class PythonGatewayConfig { |
||||
|
||||
@Value("${gateway.server.address:0.0.0.0}") |
||||
private String gatewayServerAddress; |
||||
|
||||
@Value("${gateway.server.port:25333}") |
||||
private int gatewayServerPort; |
||||
|
||||
@Value("${python.address:127.0.0.1}") |
||||
private String pythonAddress; |
||||
|
||||
@Value("${python.port:25334}") |
||||
private int pythonPort; |
||||
|
||||
@Value("${connect.timeout:0}") |
||||
private int connectTimeout; |
||||
|
||||
@Value("${read.timeout:0}") |
||||
private int readTimeout; |
||||
|
||||
public String getGatewayServerAddress() { |
||||
return gatewayServerAddress; |
||||
} |
||||
|
||||
public void setGatewayServerAddress(String gatewayServerAddress) { |
||||
this.gatewayServerAddress = gatewayServerAddress; |
||||
} |
||||
|
||||
public int getGatewayServerPort() { |
||||
return gatewayServerPort; |
||||
} |
||||
|
||||
public void setGatewayServerPort(int gatewayServerPort) { |
||||
this.gatewayServerPort = gatewayServerPort; |
||||
} |
||||
|
||||
public String getPythonAddress() { |
||||
return pythonAddress; |
||||
} |
||||
|
||||
public void setPythonAddress(String pythonAddress) { |
||||
this.pythonAddress = pythonAddress; |
||||
} |
||||
|
||||
public int getPythonPort() { |
||||
return pythonPort; |
||||
} |
||||
|
||||
public void setPythonPort(int pythonPort) { |
||||
this.pythonPort = pythonPort; |
||||
} |
||||
|
||||
public int getConnectTimeout() { |
||||
return connectTimeout; |
||||
} |
||||
|
||||
public void setConnectTimeout(int connectTimeout) { |
||||
this.connectTimeout = connectTimeout; |
||||
} |
||||
|
||||
public int getReadTimeout() { |
||||
return readTimeout; |
||||
} |
||||
|
||||
public void setReadTimeout(int readTimeout) { |
||||
this.readTimeout = readTimeout; |
||||
} |
||||
} |
||||
@ -0,0 +1,38 @@
|
||||
# |
||||
# Licensed to the Apache Software Foundation (ASF) under one or more |
||||
# contributor license agreements. See the NOTICE file distributed with |
||||
# this work for additional information regarding copyright ownership. |
||||
# The ASF licenses this file to You under the Apache License, Version 2.0 |
||||
# (the "License"); you may not use this file except in compliance with |
||||
# the License. You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, software |
||||
# distributed under the License is distributed on an "AS IS" BASIS, |
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
# See the License for the specific language governing permissions and |
||||
# limitations under the License. |
||||
# |
||||
|
||||
# The address of Python gateway server start. Set its value to `0.0.0.0` if your Python API run in different |
||||
# between Python gateway server. It could be be specific to other address like `127.0.0.1` or `localhost` |
||||
#gateway.server.address=0.0.0.0 |
||||
|
||||
# The port of Python gateway server start. Define which port you could connect to Python gateway server from |
||||
# Python API side. |
||||
#gateway.server.port=25333 |
||||
|
||||
# The address of Python callback client. |
||||
#python.address=127.0.0.1 |
||||
|
||||
# The port of Python callback client. |
||||
#python.port=25334 |
||||
|
||||
# Close connection of socket server if no other request accept after x milliseconds. Define value is (0 = infinite), |
||||
# and socket server would never close even though no requests accept |
||||
#connect.timeout=0 |
||||
|
||||
# Close each active connection of socket server if python program not active after x milliseconds. Define value is |
||||
# (0 = infinite), and socket server would never close even though no requests accept |
||||
#read.timeout=0 |
||||
Loading…
Reference in new issue