You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
75 lines
3.0 KiB
75 lines
3.0 KiB
2 years ago
|
.. Licensed to the Apache Software Foundation (ASF) under one
|
||
|
|
or more contributor license agreements. See the NOTICE file
|
||
|
|
distributed with this work for additional information
|
||
|
|
regarding copyright ownership. The ASF licenses this file
|
||
|
|
to you under the Apache License, Version 2.0 (the
|
||
|
|
"License"); you may not use this file except in compliance
|
||
|
|
with the License. You may obtain a copy of the License at
|
||
|
|
|
||
|
|
.. http://www.apache.org/licenses/LICENSE-2.0
|
||
|
|
|
||
|
|
.. Unless required by applicable law or agreed to in writing,
|
||
|
|
software distributed under the License is distributed on an
|
||
|
|
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||
|
|
KIND, either express or implied. See the License for the
|
||
|
|
specific language governing permissions and limitations
|
||
|
|
under the License.
|
||
|
|
|
||
|
|
ResourcePlugin
|
||
|
|
==============
|
||
|
|
|
||
|
|
`ResourcePlugin` is an abstract class of resource plug-in parameters of task subclass and workflow.
|
||
|
|
All resource plugins need to inherit and override its abstract methods.
|
||
|
|
|
||
|
|
Code
|
||
|
|
----
|
||
|
|
.. literalinclude:: ../../../src/pydolphinscheduler/core/resource_plugin.py
|
||
|
|
:start-after: [start resource_plugin_definition]
|
||
|
|
:end-before: [end resource_plugin_definition]
|
||
|
|
|
||
|
|
Dive Into
|
||
|
|
---------
|
||
|
|
It has the following key functions.
|
||
|
|
|
||
|
|
- Method `__init__`: The `__init__` function has STR type parameter `prefix`, which means the prefix of the resource.
|
||
|
|
|
||
|
|
You can rewrite this function if necessary.
|
||
|
|
|
||
|
|
.. literalinclude:: ../../../src/pydolphinscheduler/core/resource_plugin.py
|
||
|
|
:start-after: [start init_method]
|
||
|
|
:end-before: [end init_method]
|
||
|
|
|
||
|
|
- Method `read_file`: Get content from the given URI, The function parameter is the suffix of the file path.
|
||
|
|
|
||
|
|
The file prefix has been initialized in init of the resource plug-in.
|
||
|
|
|
||
|
|
The prefix plus suffix is the absolute path of the file in this resource.
|
||
|
|
|
||
|
|
It is an abstract function. You must rewrite it
|
||
|
|
|
||
|
|
.. literalinclude:: ../../../src/pydolphinscheduler/core/resource_plugin.py
|
||
|
|
:start-after: [start abstractmethod read_file]
|
||
|
|
:end-before: [end abstractmethod read_file]
|
||
|
|
|
||
|
|
.. automodule:: pydolphinscheduler.core.resource_plugin
|
||
|
|
|
||
|
|
How to use
|
||
|
|
----------
|
||
|
|
Resource plug-ins can be used in task subclasses and workflows. You can use the resource plug-ins by adding the `resource_plugin` parameter when they are initialized.
|
||
|
|
For example, local resource plug-ins, add `resource_plugin = Local("/tmp")`.
|
||
|
|
|
||
|
|
The resource plug-ins we currently support is `local`.
|
||
|
|
|
||
|
|
Here is an example.
|
||
|
|
|
||
|
|
.. literalinclude:: ../../../src/pydolphinscheduler/examples/tutorial_resource_plugin.py
|
||
|
|
:start-after: [start workflow_declare]
|
||
|
|
:end-before: [end task_declare]
|
||
|
|
|
||
|
|
When the resource_plugin parameter is defined in both the task subclass and the workflow, the resource_plugin defined in the task subclass is used first.
|
||
|
|
|
||
|
|
If the task subclass does not define resource_plugin, but the resource_plugin is defined in the workflow, the resource_plugin in the workflow is used.
|
||
|
|
|
||
|
|
Of course, if neither the task subclass nor the workflow specifies resource_plugin, the command at this time will be executed as a script,
|
||
|
|
|
||
|
|
in other words, we are forward compatible.
|