Browse Source
which includes `configuration`, `run example` `how to connect remote server` close: #9286, #9284, #89173.0.0/version-upgrade
Jiajie Zhong
3 years ago
committed by
GitHub
5 changed files with 207 additions and 0 deletions
@ -0,0 +1,96 @@ |
|||||||
|
.. Licensed to the Apache Software Foundation (ASF) under one |
||||||
|
or more contributor license agreements. See the NOTICE file |
||||||
|
distributed with this work for additional information |
||||||
|
regarding copyright ownership. The ASF licenses this file |
||||||
|
to you under the Apache License, Version 2.0 (the |
||||||
|
"License"); you may not use this file except in compliance |
||||||
|
with the License. You may obtain a copy of the License at |
||||||
|
|
||||||
|
.. http://www.apache.org/licenses/LICENSE-2.0 |
||||||
|
|
||||||
|
.. Unless required by applicable law or agreed to in writing, |
||||||
|
software distributed under the License is distributed on an |
||||||
|
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||||
|
KIND, either express or implied. See the License for the |
||||||
|
specific language governing permissions and limitations |
||||||
|
under the License. |
||||||
|
|
||||||
|
Configuration |
||||||
|
============= |
||||||
|
|
||||||
|
Export Configuration File |
||||||
|
------------------------- |
||||||
|
|
||||||
|
pydolphinscheduler allows you to change the built-in configurations via CLI or editor you like. pydolphinscheduler |
||||||
|
integrated built-in configurations in its package, but you could also export it locally by CLI |
||||||
|
|
||||||
|
.. code-block:: bash |
||||||
|
|
||||||
|
$ pydolphinscheduler config --init |
||||||
|
|
||||||
|
And it will create a new YAML file in the path `~/pydolphinscheduler/config.yaml` by default. If you want to export |
||||||
|
it to another path, you should set `PYDOLPHINSCHEDULER_HOME` before you run command :code:`pydolphinscheduler config --init`. |
||||||
|
|
||||||
|
.. code-block:: bash |
||||||
|
|
||||||
|
$ export PYDOLPHINSCHEDULER_HOME=<CUSTOM_PATH> |
||||||
|
$ pydolphinscheduler config --init |
||||||
|
|
||||||
|
After that, your configuration file will export into `<CUSTOM_PATH>/config.yaml` instead of the default path. |
||||||
|
|
||||||
|
Change Configuration |
||||||
|
-------------------- |
||||||
|
|
||||||
|
In section `export configuration file`_ you export the configuration file locally, and as a local file, you could |
||||||
|
edit it with any editor you like. After you save your change in your editor, the latest configuration will work |
||||||
|
when you run your workflow code. |
||||||
|
|
||||||
|
You could also query or change the configuration via CLI :code:`config --get <config>` or :code:`config --get <config> <val>`. |
||||||
|
Both `--get` and `--set` could be call one or more times in single command, and you could only set the leaf |
||||||
|
node of the configuration but could get the parent configuration, there are simple examples below: |
||||||
|
|
||||||
|
.. code-block:: bash |
||||||
|
|
||||||
|
# Get single configuration in the leaf node |
||||||
|
$ pydolphinscheduler config --get java_gateway.address |
||||||
|
The configuration query as below: |
||||||
|
|
||||||
|
java_gateway.address = 127.0.0.1 |
||||||
|
|
||||||
|
# Get multiple configuration in the leaf node |
||||||
|
$ pydolphinscheduler config --get java_gateway.address --get java_gateway.port |
||||||
|
The configuration query as below: |
||||||
|
|
||||||
|
java_gateway.address = 127.0.0.1 |
||||||
|
java_gateway.port = 25333 |
||||||
|
|
||||||
|
# Get parent configuration which contain multiple leaf nodes |
||||||
|
$ pydolphinscheduler config --get java_gateway |
||||||
|
The configuration query as below: |
||||||
|
|
||||||
|
java_gateway = ordereddict([('address', '127.0.0.1'), ('port', 25333), ('auto_convert', True)]) |
||||||
|
|
||||||
|
# Set single configuration |
||||||
|
$ pydolphinscheduler config --set java_gateway.address 192.168.1.1 |
||||||
|
Set configuration done. |
||||||
|
|
||||||
|
# Set multiple configuration |
||||||
|
$ pydolphinscheduler config --set java_gateway.address 192.168.1.1 --set java_gateway.port 25334 |
||||||
|
Set configuration done. |
||||||
|
|
||||||
|
# Set configuration not in leaf node will fail |
||||||
|
$ pydolphinscheduler config --set java_gateway 192.168.1.1,25334,True |
||||||
|
Raise error. |
||||||
|
|
||||||
|
For more information about our CLI, you could see document :doc:`cli`. |
||||||
|
|
||||||
|
All Configurations |
||||||
|
------------------ |
||||||
|
|
||||||
|
Here are all our configurations for pydolphinscheduler. |
||||||
|
|
||||||
|
.. literalinclude:: ../../src/pydolphinscheduler/core/default_config.yaml |
||||||
|
:language: yaml |
||||||
|
:lines: 18- |
||||||
|
|
||||||
|
|
@ -0,0 +1,26 @@ |
|||||||
|
.. Licensed to the Apache Software Foundation (ASF) under one |
||||||
|
or more contributor license agreements. See the NOTICE file |
||||||
|
distributed with this work for additional information |
||||||
|
regarding copyright ownership. The ASF licenses this file |
||||||
|
to you under the Apache License, Version 2.0 (the |
||||||
|
"License"); you may not use this file except in compliance |
||||||
|
with the License. You may obtain a copy of the License at |
||||||
|
|
||||||
|
.. http://www.apache.org/licenses/LICENSE-2.0 |
||||||
|
|
||||||
|
.. Unless required by applicable law or agreed to in writing, |
||||||
|
software distributed under the License is distributed on an |
||||||
|
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||||
|
KIND, either express or implied. See the License for the |
||||||
|
specific language governing permissions and limitations |
||||||
|
under the License. |
||||||
|
|
||||||
|
How To |
||||||
|
====== |
||||||
|
|
||||||
|
In this section |
||||||
|
|
||||||
|
.. toctree:: |
||||||
|
:maxdepth: 2 |
||||||
|
|
||||||
|
remote-submit |
@ -0,0 +1,51 @@ |
|||||||
|
.. Licensed to the Apache Software Foundation (ASF) under one |
||||||
|
or more contributor license agreements. See the NOTICE file |
||||||
|
distributed with this work for additional information |
||||||
|
regarding copyright ownership. The ASF licenses this file |
||||||
|
to you under the Apache License, Version 2.0 (the |
||||||
|
"License"); you may not use this file except in compliance |
||||||
|
with the License. You may obtain a copy of the License at |
||||||
|
|
||||||
|
.. http://www.apache.org/licenses/LICENSE-2.0 |
||||||
|
|
||||||
|
.. Unless required by applicable law or agreed to in writing, |
||||||
|
software distributed under the License is distributed on an |
||||||
|
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
||||||
|
KIND, either express or implied. See the License for the |
||||||
|
specific language governing permissions and limitations |
||||||
|
under the License. |
||||||
|
|
||||||
|
Submit Your Code from Different machine |
||||||
|
======================================= |
||||||
|
|
||||||
|
Generally, we use pydolphinscheduler as a client to DolphinScheduler, and consider we may change our workflow |
||||||
|
code frequently, the best practice is running :ref:`python gateway service <start:start python gateway service>` |
||||||
|
in your server machine and submit the workflow code from your development machine, like a laptop or PC. This behavior |
||||||
|
is supported by pydolphinscheduler out of box with one or two single command lines. |
||||||
|
|
||||||
|
Export Configuration File |
||||||
|
------------------------- |
||||||
|
|
||||||
|
.. code-block:: bash |
||||||
|
|
||||||
|
$ pydolphinscheduler config --init |
||||||
|
|
||||||
|
your could find more detail in :ref:`configuration exporting <config:export configuration file>` |
||||||
|
|
||||||
|
Run API Server in Other Host |
||||||
|
---------------------------- |
||||||
|
|
||||||
|
.. code-block:: bash |
||||||
|
|
||||||
|
$ pydolphinscheduler config --set java_gateway.address <your-api-server-ip-or-hostname> |
||||||
|
|
||||||
|
your could find more detail in :ref:`configuration setting <config:change configuration>` |
||||||
|
|
||||||
|
Run API Server in Other Port |
||||||
|
---------------------------- |
||||||
|
|
||||||
|
.. code-block:: bash |
||||||
|
|
||||||
|
$ pydolphinscheduler config --set java_gateway.port <your-python-gateway-service-port> |
||||||
|
|
||||||
|
your could find more detail in :ref:`configuration setting <config:change configuration>` |
Loading…
Reference in new issue