Apache NiFi Tutorial: Building a Simple File Monitoring Pipeline

  

One of the best ways to learn Apache NiFi is by building a simple end-to-end data flow. In this post, we will create a basic flow that:

 

Reads files from a local directory using the GetFile processor.

Displays FlowFile metadata using the LogAttribute processor.

Runs the flow and verifies the results in the NiFi application logs.

 

This example introduces several core NiFi concepts:

 

·      Process Groups

·      Processors

·      FlowFiles

·      Connections

·      Data Flow Execution

·      FlowFile Attributes

 

According to the Apache NiFi Getting Started Guide, a FlowFile represents a piece of user data moving through NiFi, while a Processor is responsible for creating, receiving, transforming, routing, and processing FlowFiles. The GetFile processor is commonly used to ingest files from a local filesystem into NiFi.

 

Follow below step-by-step procedure to build the NiFi DataFlow.

 

Step 1: Open the NiFi UI

Start NiFi and open the web interface https://localhost:8443.

 

After logging in, you will see the NiFi canvas.

 

Step 2: Create a Process Group

Rather than building everything on the root canvas, it is a good practice to organize flows inside a Process Group.

 

Drag the Process Group icon onto the canvas.

 

Give the name as 'File Processing Demo'

 

Click on Add button.

 

You will now see a Process Group on the canvas.

 

Step 3: Enter the Process Group

Double-click the newly created Process Group.

 

You are now inside an isolated workspace where you can build your flow.

 

Think of a Process Group as a folder that contains related processors, connections, controller services, and other flow components.

 

Step 4: Add the GetFile Processor

The GetFile processor reads files from a directory and creates FlowFiles for each file found.

 

Drag a Processor onto the canvas.

 

Search for 'GetFile', Select GetFile.

 

Click on Add button.

 

Step 5: Configure GetFile

 

Right-click the processor and select -> Configure.

 

Navigate to the Properties tab. Set input directory to some valid folder path.

 

Click Apply.

 

Step 6: Add the LogAttribute Processor

The LogAttribute processor writes FlowFile attributes to the NiFi application log. It is extremely useful for learning how FlowFiles move through a data flow.

 

Drag another Processor onto the canvas.

 

Search for LogAttribute, Select it.

 

Click Add.

 

Step 7: Connect the Processors

Now we need to tell NiFi where FlowFiles should go after GetFile processes them.

 

Hover over GetFile.

 

Drag the connection arrow to LogAttribute.

 

A dialog appears showing relationships.

 

Select the checkbox against 'success'.

 

   

Click Add.

 

The flow now looks like below.

 

 

As you observe above DataFlow image, there is a warning message against LogAttribute processor. Hover mouse to see the actual error.

   

The error quotest like below:

 

'Relationship success' is invalid because Relationship 'success' is not connected to any component and is not auto-terminated'Relationship success' is invalid because Relationship 'success' is not connected to any component and is not auto-terminated

 

How to address above issue?

NiFi requires every relationship to be handled in one of two ways:

 

Connected to another processor

Marked as Auto-Terminate

 

If neither is done, NiFi doesn't know what to do with the FlowFile, so the processor is considered invalid.

 

To address this issue, right click on LogAttribute processor -> Configure. Click on Relationships tab and select terminate checkbox.

 

Click on Apply button.

 

You can observe that the warning message disappears now.

 

   

Step 8: Start the Processors

 

Right click on the canvas -> Start.

 

It starts both the processors.

 

The processor icons should turn green. This indicates that NiFi is actively polling the input directory and processing files.

 

Step 9: Create a Test File

 

Open a terminal and create a sample file:

echo "Hello NiFi" > /Users/Shared/NiFi/inputDir/sample.txt

 

Replace the directory ‘/Users/Shared/NiFi/inputDir/’ with the input directory you set while configuring GetFile processor.

 

The GetFile processor will detect the file and convert it into a FlowFile. The FlowFile is then transferred to LogAttribute.

 

Step 10: Verify the Flow

 

The LogAttribute processor writes FlowFile information to the NiFi application log. Open a terminal and run:

 

tail -f logs/nifi-app.log

 

You can able to see following log messages.

 

--------------------------------------------------
FlowFile Properties
Key: 'entryDate'
	Value: 'Tue Jun 16 14:43:25 IST 2026'
Key: 'lineageStartDate'
	Value: 'Tue Jun 16 14:43:25 IST 2026'
Key: 'fileSize'
	Value: '11'
FlowFile Attribute Map Content
Key: 'absolute.path'
	Value: '/Users/Shared/NiFi/inputDir/'
Key: 'file.creationTime'
	Value: '2026-06-16T14:43:25+0530'
Key: 'file.group'
	Value: 'wheel'
Key: 'file.lastAccessTime'
	Value: '2026-06-16T14:43:25+0530'
Key: 'file.lastModifiedTime'
	Value: '2026-06-16T14:43:25+0530'
Key: 'file.owner'
	Value: 'h0g01ex'
Key: 'file.permissions'
	Value: 'rw-r--r--'
Key: 'filename'
	Value: 'sample.txt'
Key: 'path'
	Value: '/'
Key: 'uuid'
	Value: '0b9c32ed-daa5-4d62-a527-bfa9f8196ded'
--------------------------------------------------

   

This simple flow forms the foundation for almost every NiFi pipeline. Once comfortable with GetFile and LogAttribute, you can continue by adding processors such as:

 

·      UpdateAttribute

·      RouteOnAttribute

·      ReplaceText

·      PutFile

·      ExecuteSQL

·      PutDatabaseRecord

 

to build more advanced data integration workflows.

  

Previous                                                    Next                                                    Home