From aaf2042ec46ced05e4d0d8f98ee8b0192b78a0f5 Mon Sep 17 00:00:00 2001 From: Jiajie Zhong Date: Tue, 12 Apr 2022 09:42:28 +0800 Subject: [PATCH] [python] Add missing doc about config and connect remote server (#9443) which includes `configuration`, `run example` `how to connect remote server` close: #9286, #9284, #8917 --- .../pydolphinscheduler/docs/source/config.rst | 96 +++++++++++++++++++ .../docs/source/howto/index.rst | 26 +++++ .../docs/source/howto/remote-submit.rst | 51 ++++++++++ .../pydolphinscheduler/docs/source/index.rst | 2 + .../pydolphinscheduler/docs/source/start.rst | 32 +++++++ 5 files changed, 207 insertions(+) create mode 100644 dolphinscheduler-python/pydolphinscheduler/docs/source/config.rst create mode 100644 dolphinscheduler-python/pydolphinscheduler/docs/source/howto/index.rst create mode 100644 dolphinscheduler-python/pydolphinscheduler/docs/source/howto/remote-submit.rst diff --git a/dolphinscheduler-python/pydolphinscheduler/docs/source/config.rst b/dolphinscheduler-python/pydolphinscheduler/docs/source/config.rst new file mode 100644 index 0000000000..bfba2ecfde --- /dev/null +++ b/dolphinscheduler-python/pydolphinscheduler/docs/source/config.rst @@ -0,0 +1,96 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Configuration +============= + +Export Configuration File +------------------------- + +pydolphinscheduler allows you to change the built-in configurations via CLI or editor you like. pydolphinscheduler +integrated built-in configurations in its package, but you could also export it locally by CLI + +.. code-block:: bash + + $ pydolphinscheduler config --init + +And it will create a new YAML file in the path `~/pydolphinscheduler/config.yaml` by default. If you want to export +it to another path, you should set `PYDOLPHINSCHEDULER_HOME` before you run command :code:`pydolphinscheduler config --init`. + +.. code-block:: bash + + $ export PYDOLPHINSCHEDULER_HOME= + $ pydolphinscheduler config --init + +After that, your configuration file will export into `/config.yaml` instead of the default path. + +Change Configuration +-------------------- + +In section `export configuration file`_ you export the configuration file locally, and as a local file, you could +edit it with any editor you like. After you save your change in your editor, the latest configuration will work +when you run your workflow code. + +You could also query or change the configuration via CLI :code:`config --get ` or :code:`config --get `. +Both `--get` and `--set` could be call one or more times in single command, and you could only set the leaf +node of the configuration but could get the parent configuration, there are simple examples below: + +.. code-block:: bash + + # Get single configuration in the leaf node + $ pydolphinscheduler config --get java_gateway.address + The configuration query as below: + + java_gateway.address = 127.0.0.1 + + # Get multiple configuration in the leaf node + $ pydolphinscheduler config --get java_gateway.address --get java_gateway.port + The configuration query as below: + + java_gateway.address = 127.0.0.1 + java_gateway.port = 25333 + + # Get parent configuration which contain multiple leaf nodes + $ pydolphinscheduler config --get java_gateway + The configuration query as below: + + java_gateway = ordereddict([('address', '127.0.0.1'), ('port', 25333), ('auto_convert', True)]) + + # Set single configuration + $ pydolphinscheduler config --set java_gateway.address 192.168.1.1 + Set configuration done. + + # Set multiple configuration + $ pydolphinscheduler config --set java_gateway.address 192.168.1.1 --set java_gateway.port 25334 + Set configuration done. + + # Set configuration not in leaf node will fail + $ pydolphinscheduler config --set java_gateway 192.168.1.1,25334,True + Raise error. + +For more information about our CLI, you could see document :doc:`cli`. + +All Configurations +------------------ + +Here are all our configurations for pydolphinscheduler. + +.. literalinclude:: ../../src/pydolphinscheduler/core/default_config.yaml + :language: yaml + :lines: 18- + + diff --git a/dolphinscheduler-python/pydolphinscheduler/docs/source/howto/index.rst b/dolphinscheduler-python/pydolphinscheduler/docs/source/howto/index.rst new file mode 100644 index 0000000000..e83b1631cf --- /dev/null +++ b/dolphinscheduler-python/pydolphinscheduler/docs/source/howto/index.rst @@ -0,0 +1,26 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +How To +====== + +In this section + +.. toctree:: + :maxdepth: 2 + + remote-submit diff --git a/dolphinscheduler-python/pydolphinscheduler/docs/source/howto/remote-submit.rst b/dolphinscheduler-python/pydolphinscheduler/docs/source/howto/remote-submit.rst new file mode 100644 index 0000000000..43d03cce38 --- /dev/null +++ b/dolphinscheduler-python/pydolphinscheduler/docs/source/howto/remote-submit.rst @@ -0,0 +1,51 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Submit Your Code from Different machine +======================================= + +Generally, we use pydolphinscheduler as a client to DolphinScheduler, and consider we may change our workflow +code frequently, the best practice is running :ref:`python gateway service ` +in your server machine and submit the workflow code from your development machine, like a laptop or PC. This behavior +is supported by pydolphinscheduler out of box with one or two single command lines. + +Export Configuration File +------------------------- + +.. code-block:: bash + + $ pydolphinscheduler config --init + +your could find more detail in :ref:`configuration exporting ` + +Run API Server in Other Host +---------------------------- + +.. code-block:: bash + + $ pydolphinscheduler config --set java_gateway.address + +your could find more detail in :ref:`configuration setting ` + +Run API Server in Other Port +---------------------------- + +.. code-block:: bash + + $ pydolphinscheduler config --set java_gateway.port + +your could find more detail in :ref:`configuration setting ` diff --git a/dolphinscheduler-python/pydolphinscheduler/docs/source/index.rst b/dolphinscheduler-python/pydolphinscheduler/docs/source/index.rst index ad4c93d1f9..24ad107ad6 100644 --- a/dolphinscheduler-python/pydolphinscheduler/docs/source/index.rst +++ b/dolphinscheduler-python/pydolphinscheduler/docs/source/index.rst @@ -32,7 +32,9 @@ then go and see :doc:`tutorial` for more detail. tutorial concept tasks/index + howto/index cli + config api Indices and tables diff --git a/dolphinscheduler-python/pydolphinscheduler/docs/source/start.rst b/dolphinscheduler-python/pydolphinscheduler/docs/source/start.rst index 489f4e8eb3..e411d7bf72 100644 --- a/dolphinscheduler-python/pydolphinscheduler/docs/source/start.rst +++ b/dolphinscheduler-python/pydolphinscheduler/docs/source/start.rst @@ -107,6 +107,37 @@ the server is health if keyword `ApiApplicationServer` in the console. yaml config path `python-gateway.enabled : true` in api-server's configuration path in `api-server/conf/application.yaml`. The default value is true and Python gateway service start when api server is been started. +Run an Example +-------------- + +Before run an example for pydolphinscheduler, you should get the example code from it source code. You could run +single bash command to get it + +.. code-block:: bash + + $ wget https://raw.githubusercontent.com/apache/dolphinscheduler/dev/dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/examples/tutorial.py + +or you could copy-paste the content from `tutorial source code`_. And then you could run the example in your +terminal + +.. code-block:: bash + + $ python tutorial.py + +If you want to submit your workflow to a remote API server, which means that your workflow script is different +from the API server, you should first change pydolphinscheduler configuration and then submit the workflow script + +.. code-block:: bash + + $ pydolphinscheduler config --init + $ pydolphinscheduler config --set java_gateway.address + $ python tutorial.py + +.. note:: + + You could see more information in :doc:`config` about all the configurations pydolphinscheduler supported. + + What's More ----------- @@ -117,3 +148,4 @@ maybe you could go and play with all :doc:`tasks/index` *PyDolphinScheduler* sup .. _`instructions for all platforms here`: https://wiki.python.org/moin/BeginnersGuide/Download .. _`Apache DolphinScheduler`: https://dolphinscheduler.apache.org .. _`install Apache DolphinScheduler`: https://dolphinscheduler.apache.org/en-us/docs/latest/user_doc/guide/installation/standalone.html +.. _`tutorial source code`: https://raw.githubusercontent.com/apache/dolphinscheduler/dev/dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/examples/tutorial.py