Browse Source

[fix] Python task can not pass the parameters to downstream task. (#12961)

3.2.0-release
JieguangZhou 2 years ago committed by GitHub
parent
commit
6e224d08a2
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
  1. 9
      docs/docs/en/guide/parameter/context.md
  2. 8
      docs/docs/zh/guide/parameter/context.md
  3. BIN
      docs/img/new_ui/dev/parameter/python_context_param.png
  4. 1
      dolphinscheduler-task-plugin/dolphinscheduler-task-python/src/main/java/org/apache/dolphinscheduler/plugin/task/python/PythonTask.java

9
docs/docs/en/guide/parameter/context.md

@ -13,6 +13,7 @@ DolphinScheduler allows parameter transfer between tasks. Currently, transfer di
* [Shell](../task/shell.md) * [Shell](../task/shell.md)
* [SQL](../task/sql.md) * [SQL](../task/sql.md)
* [Procedure](../task/stored-procedure.md) * [Procedure](../task/stored-procedure.md)
* [Python](../task/python.md)
When defining an upstream node, if there is a need to transmit the result of that node to a dependency related downstream node. You need to set an `OUT` direction parameter to [Custom Parameters] of the [Current Node Settings]. At present, we mainly focus on the SQL and shell nodes to pass parameters downstream. When defining an upstream node, if there is a need to transmit the result of that node to a dependency related downstream node. You need to set an `OUT` direction parameter to [Custom Parameters] of the [Current Node Settings]. At present, we mainly focus on the SQL and shell nodes to pass parameters downstream.
@ -76,3 +77,11 @@ The result of Node_mysql is as follows:
Even though output is assigned a value of 1 in Node_A's script, the log still shows a value of 100. But according to the principle from [parameter priority](priority.md): `Local Parameter > Parameter Context > Global Parameter`, the output value in Node_B is 1. It proves that the output parameter is passed in the workflow with reference to the expected value, and the query operation is completed using this value in Node_mysql. Even though output is assigned a value of 1 in Node_A's script, the log still shows a value of 100. But according to the principle from [parameter priority](priority.md): `Local Parameter > Parameter Context > Global Parameter`, the output value in Node_B is 1. It proves that the output parameter is passed in the workflow with reference to the expected value, and the query operation is completed using this value in Node_mysql.
But the output value 66 only shows in the Node_A, the reason is that the direction of value is selected as IN, and only when the direction is OUT will it be defined as a variable output. But the output value 66 only shows in the Node_A, the reason is that the direction of value is selected as IN, and only when the direction is OUT will it be defined as a variable output.
#### Pass parameter from Python task to downstream
Use `print('${setValue(key=%s)}' % value)`, DolphinScheduler will capture the `${setValue(key=value}` in the output to capture the parameters and pass them downstream.
For example
![img.png](../../../../img/new_ui/dev/parameter/python_context_param.png)

8
docs/docs/zh/guide/parameter/context.md

@ -13,6 +13,7 @@ DolphinScheduler 允许在任务间进行参数传递,目前传递方向仅支
* [Shell](../task/shell.md) * [Shell](../task/shell.md)
* [SQL](../task/sql.md) * [SQL](../task/sql.md)
* [Procedure](../task/stored-procedure.md) * [Procedure](../task/stored-procedure.md)
* [Python](../task/python.md)
当定义上游节点时,如果有需要将该节点的结果传递给有依赖关系的下游节点,需要在【当前节点设置】的【自定义参数】设置一个方向是 OUT 的变量。目前我们主要针对 SQL 和 SHELL 节点做了可以向下传递参数的功能。 当定义上游节点时,如果有需要将该节点的结果传递给有依赖关系的下游节点,需要在【当前节点设置】的【自定义参数】设置一个方向是 OUT 的变量。目前我们主要针对 SQL 和 SHELL 节点做了可以向下传递参数的功能。
@ -76,3 +77,10 @@ Node_mysql 运行结果如下:
虽然在 Node_A 的脚本中为 output 赋值为 1,但日志中显示的值仍然为 100。但根据[参数优先级](priority.md)的原则:`本地参数 > 上游任务传递的参数 > 全局参数`,在 Node_B 中输出的值为 1。则证明 output 参数参照预期的值在该工作流中传递,并在 Node_mysql 中使用该值完成查询操作。 虽然在 Node_A 的脚本中为 output 赋值为 1,但日志中显示的值仍然为 100。但根据[参数优先级](priority.md)的原则:`本地参数 > 上游任务传递的参数 > 全局参数`,在 Node_B 中输出的值为 1。则证明 output 参数参照预期的值在该工作流中传递,并在 Node_mysql 中使用该值完成查询操作。
但是 value 的值却只有在 Node_A 中输出为 66,其原因为 value 的方向选择为 IN,只有当方向为 OUT 时才会被定义为变量输出。 但是 value 的值却只有在 Node_A 中输出为 66,其原因为 value 的方向选择为 IN,只有当方向为 OUT 时才会被定义为变量输出。
#### Python 任务传递参数
使用 `print('${setValue(key=%s)}' % value)`,DolphinScheduler会捕捉输出中的 `${setValue(key=value}`来进行参数捕捉,从而传递到下游
![img.png](../../../../img/new_ui/dev/parameter/python_context_param.png)

BIN
docs/img/new_ui/dev/parameter/python_context_param.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 38 KiB

1
dolphinscheduler-task-plugin/dolphinscheduler-task-python/src/main/java/org/apache/dolphinscheduler/plugin/task/python/PythonTask.java

@ -114,6 +114,7 @@ public class PythonTask extends AbstractTask {
setExitStatusCode(taskResponse.getExitStatusCode()); setExitStatusCode(taskResponse.getExitStatusCode());
setProcessId(taskResponse.getProcessId()); setProcessId(taskResponse.getProcessId());
setVarPool(shellCommandExecutor.getVarPool()); setVarPool(shellCommandExecutor.getVarPool());
pythonParameters.dealOutParam(shellCommandExecutor.getVarPool());
} catch (Exception e) { } catch (Exception e) {
logger.error("python task failure", e); logger.error("python task failure", e);
setExitStatusCode(TaskConstants.EXIT_CODE_FAILURE); setExitStatusCode(TaskConstants.EXIT_CODE_FAILURE);

Loading…
Cancel
Save