Workflow orchestration is the process of managing and coordinating the execution of data pipelines which involves scheduling and automating the various tasks and processes involved in data pipelines, as well as error handling, alerting, and monitoring.
- Canvas with drag-and-drop interface for defining triggers, actions, and notifications
- Support for conditional execution of tasks and processes
- Scheduling and automation features
- Monitoring, alerting, and error handling tools
Substitutions are variables with values that you define here and use them within the workflow. When the workflow runs, the value replaces the variable.
When you create a job, you can override the variable values defined in the workflow. For example, you can define a directory variable and set it to different values depending which environment the job is set up to run. To use the directory variable within a workflow, you can refer it as ${directory}. You may use different values for development, staging and production runtimes.