Airflow: Operator

 

Operator defines a single task in the data-pipeline. For example, if you want to run a shell command, you can use ‘bash’ operator to execute the command.

 

An Operator describes single task in an airflow. In general, operators run independently. So parallel tasks can be scheduled on different worker nodes.

 

In case of task failure, airflow retry the task automatically.

 

Operator should be idempotent

Operator should produce the same result regardless of how many times it runs.

 

Can I create custom operator?

Yes, you can

 

Apache airflow support variety of operators that are sufficient for most of your needs. For example,

a.   Bash Operator: Execute single bash command

b.   Email Operator: Sends an email

c.    Sqlite, Postgres, MySQL Operator: Execute single SQL command

d.   Python Operator: Execute an arbitrary python function

 

Types of operators

There are three types of operators in Apache airflow.

a.   Action Operator: Execute an action. Example: Email operator and Bash operator

b.   Transfer Operator: Move data from one system to another. Example: sftp operator

c.    Sensor Operator: Wait for the data to arrive at a defined location.

 

Previous                                                    Next                                                    Home