You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
75 lines
3.1 KiB
75 lines
3.1 KiB
.. Licensed to the Apache Software Foundation (ASF) under one |
|
or more contributor license agreements. See the NOTICE file |
|
distributed with this work for additional information |
|
regarding copyright ownership. The ASF licenses this file |
|
to you under the Apache License, Version 2.0 (the |
|
"License"); you may not use this file except in compliance |
|
with the License. You may obtain a copy of the License at |
|
|
|
.. http://www.apache.org/licenses/LICENSE-2.0 |
|
|
|
.. Unless required by applicable law or agreed to in writing, |
|
software distributed under the License is distributed on an |
|
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
|
KIND, either express or implied. See the License for the |
|
specific language governing permissions and limitations |
|
under the License. |
|
|
|
ResourcePlugin |
|
============== |
|
|
|
`ResourcePlugin` is an abstract class of resource plug-in parameters of task subclass and workflow. |
|
All resource plugins need to inherit and override its abstract methods. |
|
|
|
Code |
|
---- |
|
.. literalinclude:: ../../../src/pydolphinscheduler/core/resource_plugin.py |
|
:start-after: [start resource_plugin_definition] |
|
:end-before: [end resource_plugin_definition] |
|
|
|
Dive Into |
|
--------- |
|
It has the following key functions. |
|
|
|
- Method `__init__`: The `__init__` function has STR type parameter `prefix`, which means the prefix of the resource. |
|
|
|
You can rewrite this function if necessary. |
|
|
|
.. literalinclude:: ../../../src/pydolphinscheduler/core/resource_plugin.py |
|
:start-after: [start init_method] |
|
:end-before: [end init_method] |
|
|
|
- Method `read_file`: Get content from the given URI, The function parameter is the suffix of the file path. |
|
|
|
The file prefix has been initialized in init of the resource plug-in. |
|
|
|
The prefix plus suffix is the absolute path of the file in this resource. |
|
|
|
It is an abstract function. You must rewrite it |
|
|
|
.. literalinclude:: ../../../src/pydolphinscheduler/core/resource_plugin.py |
|
:start-after: [start abstractmethod read_file] |
|
:end-before: [end abstractmethod read_file] |
|
|
|
.. automodule:: pydolphinscheduler.core.resource_plugin |
|
|
|
How to use |
|
---------- |
|
Resource plugin can be used in task subclasses and workflows. You can use the resource plugin by adding the `resource_plugin` parameter when they are initialized. |
|
For example, local resource plugin, add `resource_plugin = Local("/tmp")`. |
|
|
|
The resource plugin we currently support are `local`, `github`, `gitlab`, `OSS`, `S3`. |
|
|
|
Here is an example. |
|
|
|
.. literalinclude:: ../../../src/pydolphinscheduler/examples/tutorial_resource_plugin.py |
|
:start-after: [start workflow_declare] |
|
:end-before: [end task_declare] |
|
|
|
When the resource_plugin parameter is defined in both the task subclass and the workflow, the resource_plugin defined in the task subclass is used first. |
|
|
|
If the task subclass does not define resource_plugin, but the resource_plugin is defined in the workflow, the resource_plugin in the workflow is used. |
|
|
|
Of course, if neither the task subclass nor the workflow specifies resource_plugin, the command at this time will be executed as a script, |
|
|
|
in other words, we are forward compatible. |