Before building dataflows in Apache NiFi, it is important to understand two fundamental concepts:
· FlowFile: The data that moves through NiFi.
· Processor: The component that performs actions on that data.
Think of NiFi as a large package delivery system:
· FlowFiles are the packages being transported.
· Processors are the workers who receive, inspect, modify, route, and deliver those packages.
Once you understand these two concepts, the rest of NiFi becomes much easier to learn.
1. FlowFile
A FlowFile is the basic unit of data in Apache NiFi. Whenever data enters NiFi, it is wrapped inside a FlowFile and then moves through the dataflow from one Processor to another.
Examples of data that can become FlowFiles:
· CSV files
· JSON files
· XML documents
· Images
· Log files
· Database records
· Messages from Kafka
· API responses
In simple terms, a FlowFile represents a piece of data traveling through a NiFi flow.
2. Structure of a FlowFile
A FlowFile consists of two parts Content and Attributes.
a. Content
The Content is the actual data.
Example 1: JSON Content
{
"id": 1,
"name": "Krishna",
"city": "Bangalore"
}
Example 2: CSV Content
id,name,city 101,Krishna,Bangalore
Example 3: Log Content
2025-06-16 ERROR Database Connection Failed
Everything inside the data itself is called the Content.
b. Attributes
Attributes are metadata about the content. They are stored as key-value pairs. Following are some of the key attributes of FlowFile.
· path: The FlowFile's path indicates the relative directory to which a FlowFile belongs and does not contain the filename
· filename: The filename of the FlowFile. The filename should not contain any directory structure.
· uuid: A unique UUID assigned to this FlowFile
· priority: A numeric value indicating the FlowFile priority
· mime.type: The MIME Type of this FlowFile
· discard.reason: Specifies the reason that a FlowFile is being discarded
· alternate.identifier: Indicates an identifier other than the FlowFile's UUID that is known to refer to this FlowFile.
3. Real-World Example
Imagine a courier package. The package contains a Laptop. This is equivalent to the Content.
The shipping label contains:
· Sender: Amazon
· Receiver: Hari
· Destination: Chennai
· Priority: High
This is equivalent to the Attributes. Just as a courier company routes packages using labels, NiFi routes FlowFiles using attributes.
4. Why Attributes Are Important
Reading file content can be expensive. Instead of opening every file, NiFi can make decisions based on attributes.
For example: If country = UK, then route the FlowFile to the UK processing pipeline. If country = India, then route the FlowFile to the India processing pipeline. This makes processing much faster.
5. FlowFile Lifecycle
A typical FlowFile journey looks like this:
Data Source
↓
FlowFile Created
↓
Processor 1
↓
Processor 2
↓
Processor 3
↓
Destination
Example:
CSV File
↓
GetFile
↓
ValidateRecord
↓
ConvertRecord
↓
PutDatabaseRecord
At every step, the FlowFile continues traveling through the flow.
In the above flow diagram, a Processor is the component that performs work on FlowFiles. Processors are the building blocks of every NiFi dataflow. Without Processors, FlowFiles cannot move or be processed.
Think of a Processor as a worker in a factory. Each worker performs a specific task before passing the item to the next worker.
2. Processors
A Processor is the component that performs work on FlowFiles. Processors are the building blocks of every NiFi dataflow. Without Processors, FlowFiles cannot move or be processed.
Think of a Processor as a worker in a factory. Each worker performs a specific task before passing the item to the next worker.
A Processor can:
· Create FlowFiles (Ex: GenerateFlowFile)
· Receive FlowFiles (Ex: GetFile, ConsumeKafka, ListenHTTP, GetSFTP)
· Transform FlowFiles (Ex: Convert CSV → JSON, Convert XML → JSON, Modify Fields or content, Mask Sensitive Data)
· Route FlowFiles (Ex: Send data to different paths based on conditions)
· Split FlowFiles
· Merge FlowFiles
· Send FlowFiles to External Systems
2.1 Processor Analogy
Imagine an airport baggage handling system.
Suitcase Arrives
↓
Security Check
↓
Weight Check
↓
Destination Check
↓
Loaded onto Flight
Each station performs a specific task.
In NiFi:
FlowFile Arrives
↓
Processor A
↓
Processor B
↓
Processor C
↓
Destination
Each Processor performs one responsibility.
2.2 Commonly Used Processors
Following table summarizes commonly used processors by Beginners.
|
Processor |
Purpose |
|
GenerateFlowFile |
Create sample FlowFiles |
|
GetFile |
Read files from disk |
|
PutFile |
Write files to disk |
|
UpdateAttribute |
Add or modify attributes |
|
RouteOnAttribute |
Route based on attributes |
|
ReplaceText |
Modify content |
|
SplitJson |
Split JSON documents |
|
MergeContent |
Merge FlowFiles |
|
ConsumeKafka |
Read from Kafka |
|
PublishKafka |
Write to Kafka |
2.3 FlowFile and Processor Working Together
Consider a simple file ingestion flow:
Customer.csv
↓
GetFile
↓
UpdateAttribute
↓
RouteOnAttribute
↓
PutFile
Here:
· GetFile: Reads "Customer.csv" and creates a FlowFile.
· UpdateAttribute: Adds attributes like sourceSystem=CRM, country=UK etc.,
· RouteOnAttribute: Checks country=UK and routes to UK pipeline.
· PutFile: Stores the file in the target location.
+------------------+
| Customer.csv |
+------------------+
|
v
+------------------+
| GetFile |
+------------------+
|
v
+------------------+
| UpdateAttribute |
+------------------+
|
v
+------------------+
| RouteOnAttribute |
+------------------+
|
v
+------------------+
| PutFile |
+------------------+
The file travels as a FlowFile, while each box is a Processor performing an action.
In sumamry:
· FlowFile is the fundamental unit of data in NiFi. It contains Content (actual data) and attributes (metadata). FlowFile move throught the data flow.
· Processor is the Core building block of NiFi. It performs actions on FlowFiles and Processors can create, receive, transform, route, split, merge, and send data.
Understanding these two concepts is the foundation for learning Apache NiFi, because every NiFi flow is ultimately built by moving FlowFiles through a series of Processors.
Previous Next Home
No comments:
Post a Comment