Browse Source

[python] Add missing doc about config and connect remote server (#9443)

which includes `configuration`, `run example`
`how to connect remote server`

close: #9286, #9284, #8917
3.0.0/version-upgrade
Jiajie Zhong 3 years ago committed by GitHub
parent
commit
aaf2042ec4
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
  1. 96
      dolphinscheduler-python/pydolphinscheduler/docs/source/config.rst
  2. 26
      dolphinscheduler-python/pydolphinscheduler/docs/source/howto/index.rst
  3. 51
      dolphinscheduler-python/pydolphinscheduler/docs/source/howto/remote-submit.rst
  4. 2
      dolphinscheduler-python/pydolphinscheduler/docs/source/index.rst
  5. 32
      dolphinscheduler-python/pydolphinscheduler/docs/source/start.rst

96
dolphinscheduler-python/pydolphinscheduler/docs/source/config.rst

@ -0,0 +1,96 @@
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
Configuration
=============
Export Configuration File
-------------------------
pydolphinscheduler allows you to change the built-in configurations via CLI or editor you like. pydolphinscheduler
integrated built-in configurations in its package, but you could also export it locally by CLI
.. code-block:: bash
$ pydolphinscheduler config --init
And it will create a new YAML file in the path `~/pydolphinscheduler/config.yaml` by default. If you want to export
it to another path, you should set `PYDOLPHINSCHEDULER_HOME` before you run command :code:`pydolphinscheduler config --init`.
.. code-block:: bash
$ export PYDOLPHINSCHEDULER_HOME=<CUSTOM_PATH>
$ pydolphinscheduler config --init
After that, your configuration file will export into `<CUSTOM_PATH>/config.yaml` instead of the default path.
Change Configuration
--------------------
In section `export configuration file`_ you export the configuration file locally, and as a local file, you could
edit it with any editor you like. After you save your change in your editor, the latest configuration will work
when you run your workflow code.
You could also query or change the configuration via CLI :code:`config --get <config>` or :code:`config --get <config> <val>`.
Both `--get` and `--set` could be call one or more times in single command, and you could only set the leaf
node of the configuration but could get the parent configuration, there are simple examples below:
.. code-block:: bash
# Get single configuration in the leaf node
$ pydolphinscheduler config --get java_gateway.address
The configuration query as below:
java_gateway.address = 127.0.0.1
# Get multiple configuration in the leaf node
$ pydolphinscheduler config --get java_gateway.address --get java_gateway.port
The configuration query as below:
java_gateway.address = 127.0.0.1
java_gateway.port = 25333
# Get parent configuration which contain multiple leaf nodes
$ pydolphinscheduler config --get java_gateway
The configuration query as below:
java_gateway = ordereddict([('address', '127.0.0.1'), ('port', 25333), ('auto_convert', True)])
# Set single configuration
$ pydolphinscheduler config --set java_gateway.address 192.168.1.1
Set configuration done.
# Set multiple configuration
$ pydolphinscheduler config --set java_gateway.address 192.168.1.1 --set java_gateway.port 25334
Set configuration done.
# Set configuration not in leaf node will fail
$ pydolphinscheduler config --set java_gateway 192.168.1.1,25334,True
Raise error.
For more information about our CLI, you could see document :doc:`cli`.
All Configurations
------------------
Here are all our configurations for pydolphinscheduler.
.. literalinclude:: ../../src/pydolphinscheduler/core/default_config.yaml
:language: yaml
:lines: 18-

26
dolphinscheduler-python/pydolphinscheduler/docs/source/howto/index.rst

@ -0,0 +1,26 @@
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
How To
======
In this section
.. toctree::
:maxdepth: 2
remote-submit

51
dolphinscheduler-python/pydolphinscheduler/docs/source/howto/remote-submit.rst

@ -0,0 +1,51 @@
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
Submit Your Code from Different machine
=======================================
Generally, we use pydolphinscheduler as a client to DolphinScheduler, and consider we may change our workflow
code frequently, the best practice is running :ref:`python gateway service <start:start python gateway service>`
in your server machine and submit the workflow code from your development machine, like a laptop or PC. This behavior
is supported by pydolphinscheduler out of box with one or two single command lines.
Export Configuration File
-------------------------
.. code-block:: bash
$ pydolphinscheduler config --init
your could find more detail in :ref:`configuration exporting <config:export configuration file>`
Run API Server in Other Host
----------------------------
.. code-block:: bash
$ pydolphinscheduler config --set java_gateway.address <your-api-server-ip-or-hostname>
your could find more detail in :ref:`configuration setting <config:change configuration>`
Run API Server in Other Port
----------------------------
.. code-block:: bash
$ pydolphinscheduler config --set java_gateway.port <your-python-gateway-service-port>
your could find more detail in :ref:`configuration setting <config:change configuration>`

2
dolphinscheduler-python/pydolphinscheduler/docs/source/index.rst

@ -32,7 +32,9 @@ then go and see :doc:`tutorial` for more detail.
tutorial
concept
tasks/index
howto/index
cli
config
api
Indices and tables

32
dolphinscheduler-python/pydolphinscheduler/docs/source/start.rst

@ -107,6 +107,37 @@ the server is health if keyword `ApiApplicationServer` in the console.
yaml config path `python-gateway.enabled : true` in api-server's configuration path in `api-server/conf/application.yaml`.
The default value is true and Python gateway service start when api server is been started.
Run an Example
--------------
Before run an example for pydolphinscheduler, you should get the example code from it source code. You could run
single bash command to get it
.. code-block:: bash
$ wget https://raw.githubusercontent.com/apache/dolphinscheduler/dev/dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/examples/tutorial.py
or you could copy-paste the content from `tutorial source code`_. And then you could run the example in your
terminal
.. code-block:: bash
$ python tutorial.py
If you want to submit your workflow to a remote API server, which means that your workflow script is different
from the API server, you should first change pydolphinscheduler configuration and then submit the workflow script
.. code-block:: bash
$ pydolphinscheduler config --init
$ pydolphinscheduler config --set java_gateway.address <your-api-server-ip-or-hostname>
$ python tutorial.py
.. note::
You could see more information in :doc:`config` about all the configurations pydolphinscheduler supported.
What's More
-----------
@ -117,3 +148,4 @@ maybe you could go and play with all :doc:`tasks/index` *PyDolphinScheduler* sup
.. _`instructions for all platforms here`: https://wiki.python.org/moin/BeginnersGuide/Download
.. _`Apache DolphinScheduler`: https://dolphinscheduler.apache.org
.. _`install Apache DolphinScheduler`: https://dolphinscheduler.apache.org/en-us/docs/latest/user_doc/guide/installation/standalone.html
.. _`tutorial source code`: https://raw.githubusercontent.com/apache/dolphinscheduler/dev/dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/examples/tutorial.py

Loading…
Cancel
Save