In the previous article, we explored the Foundation Model, the brain of an AI agent. While the brain is responsible for reasoning and decision-making, it cannot operate in isolation. Just like humans rely on their senses to understand the world, AI agents need a way to receive, interpret, and organize information before they can think.
This is the responsibility of the Perception Module, the component that bridges the gap between the messy real world and the language model.
In this post, we'll explore how AI agents perceive their environment, why perception is much more than a simple chat message, and how raw data is transformed into something an LLM can actually understand.
1. Perception: The Starting Point of Every AI Agent
In the previous post, we learned that the Foundation Model (LLM) serves as the brain of an AI agent. It can analyze information, reason about problems, generate plans, and make decisions. However, even the most powerful brain is useless if it has no awareness of the world around it.
Think about a human being. Our brain doesn't magically know what's happening around us. It relies on our eyes to see, our ears to hear, and our sense of touch to feel our surroundings. These senses continuously collect information from the environment and send it to the brain, enabling us to understand what is happening and decide what to do next.
An AI agent works in exactly the same way. The language model is responsible for thinking, but it cannot think without first receiving information. This responsibility belongs to the Perception Module, which acts as the agent's senses. It is the gateway through which the agent observes and understands its environment.
1.1 The Agentic Loop
Every AI agent operates in a continuous cycle often referred to as the Agentic Loop.
Perceive
↓
Reason
↓
Plan
↓
Act
↓
Perceive Again
Each stage has a distinct responsibility:
· Perceive: Gather information from the environment
· Reason: Analyse the information and understand the situation.
· Plan: Decide the best course of action.
· Act: Execute the chosen action using tools, APIs, or external systems.
· Perceive Again: Observe the results of the action and continue the cycle.
Notice that everything begins with perception. Without new information, there is nothing to reason about, no plan to create, and no action to perform. Every iteration of an AI agent starts by observing its environment.
1.2 Perception Is the Trigger That Wakes Up the Agent
A useful way to think about perception is as the trigger that starts the entire reasoning process. Imagine someone knocking on your front door. Until you hear that knock, your brain has no reason to react. The knock becomes the event that captures your attention and causes you to think about what to do next.
Perception plays exactly the same role in an AI agent. Whenever the agent receives new information, it is "woken up". That new information becomes the starting point for the reasoning process. Without a trigger, the agent remains idle because there is nothing new to process.
In other words, perception is the front door through which every piece of information enters the system.
1.3 Perception Is More Than a Chat Message
When people first learn about AI agents, they often assume perception simply means a user typing a message into a chatbot.
While a chat message is certainly one form of perception, it is only one of many possible triggers. From an architectural perspective, perception is any event that changes what the agent knows about its environment.
That event may originate from a human, another application, a device, or even a scheduled task.
Some common examples include:
· A user sends a message through a chat interface.
· A new email arrives in an inbox.
· An external API returns updated information.
· A webhook notifies the agent that an order has been placed.
· A scheduled job runs every morning at 9:00 AM.
· A monitoring system reports that a server has become unhealthy.
· An IoT sensor detects an abnormal temperature.
· A customer uploads a PDF document or an image.
· A database receives a new record that requires processing.
Each of these events provides new information that may require the agent to reason and take action. This event-driven view is one of the key differences between modern AI agents and traditional chatbots. An AI agent is not limited to waiting for user messages, it can continuously observe multiple information sources and react whenever something meaningful happens.
As software architects, it's important to view the Perception Module as much more than an input box. It is the interface between the AI agent and the real world.
Every piece of incoming information, whether it comes from a user, a business application, a cloud service, a database, or a physical sensor passes through this module before reaching the language model.
In the next section, we'll explore why this translation layer is necessary and examine one of the biggest challenges in agent design: the real world is messy, unstructured, and multimodal, while language models reason most effectively using clean textual input.
2. The Challenge: The Real World Is Messy
Now that we understand how perception serves as the starting point of every AI agent, let's examine the biggest challenge faced by the Perception Module.
The language model is an incredibly powerful reasoning engine, but it was designed to process information in a very specific way. It excels at understanding relationships, drawing conclusions, generating explanations, and making decisions. However, there is one important limitation that every architect should understand:
The real world doesn't communicate in the format that an LLM prefers.
The outside world is messy, noisy, unstructured, and often contains information spread across multiple formats. Before the language model can reason about anything, that information must first be transformed into something it can understand.
2.1 The Real World Doesn't Speak the Language of LLMs
Imagine asking a customer support agent to investigate a product issue.
The customer might:
· Type a message describing the problem.
· Upload a photograph of the damaged product.
· Attach a warranty PDF.
· Record a voice message explaining the issue.
· Include an order number returned from an API.
· Share a video demonstrating the problem.
To a human, combining these pieces of information feels natural. We instinctively read the message, look at the image, skim the document, listen to the audio, and mentally combine everything into one understanding of the situation.
A language model cannot perform this process automatically. Each input arrives in a different format and requires specialized processing before it can become useful for reasoning.
2.2 AI Agents Receive Many Different Types of Inputs
Modern AI agents rarely interact with a single source of information. Instead, they continuously receive data from multiple systems, users, and applications.
Some common input types include:
a. Text
This is the simplest form of perception.
Examples include:
· Chat messages
· Emails
· Slack or Microsoft Teams conversations
· Web forms
· Log files
· Database records
Even text, however, is often imperfect. It may contain spelling mistakes, incomplete sentences, abbreviations, or multiple languages.
b. Images
Users frequently communicate by sharing screenshots, photographs, scanned documents, receipts, invoices, handwritten notes, or pictures of damaged products.
Although humans immediately recognize the contents of an image, an LLM cannot directly reason over raw image pixels.
The image must first be analyzed by a vision model that converts visual information into a textual description.
c. PDF Documents
PDFs are one of the most common enterprise data formats.
They may contain:
· Contracts
· Research papers
· Financial reports
· User manuals
· Medical records
· Policies and procedures
A PDF is much more than plain text. It contains formatting, tables, diagrams, headers, footers, page numbers, and images. Before an LLM can understand its contents, the useful information must be extracted and organized.
d. Audio
Many business workflows involve spoken conversations.
Examples include:
· Customer support calls
· Voice notes
· Meeting recordings
· Interviews
· Podcasts
An LLM cannot reason directly from an audio waveform. The audio must first pass through a Speech-to-Text system, producing a transcript that captures what was said.
e. Videos
Videos combine several information sources into a single file.
A video may contain:
· Visual scenes
· Spoken language
· Text displayed on the screen
· Human actions
· Facial expressions
· Background events
Understanding a video often requires extracting frames, recognizing objects, transcribing speech, and combining all these observations into a coherent summary.
f. JSON and API Responses
Enterprise AI agents frequently communicate with business applications through APIs. A single API response may contain thousands of lines of JSON representing:
· Customer information
· Orders
· Products
· Transactions
· Sensor readings
· System metrics
While JSON is structured for computers, it is not an ideal format for reasoning. Instead of processing hundreds of nested fields, the agent benefits from receiving a concise textual summary containing only the information relevant to the current task.
2.3 The Real World Is Unstructured, Noisy, and Multimodal
One of the biggest architectural challenges is that real-world information rarely arrives in a clean, consistent format.
Instead, it is often:
· Unstructured: Documents, emails, chat conversations, and images lack a fixed schema.
· Noisy: Data may contain spelling mistakes, irrelevant details, duplicate information, poor image quality, or incomplete records.
· Multimodal: Information is spread across text, images, audio, video, documents, and structured data sources.
An AI agent must combine all these different forms of information into one coherent understanding before reasoning can begin.
This process is similar to how humans use multiple senses simultaneously. We don't think separately about what we see, hear, and read, we combine everything into a single mental model of the situation.
An effective AI agent must do the same.
2.4 Why LLMs Ultimately Reason Using Text
This brings us to one of the most important concepts in AI agent architecture. Although today's models are often described as multimodal, their reasoning process is still fundamentally driven by textual representations.
Whether the original input is:
· an image
· a PDF
· an audio recording
· a video
· or a JSON document
the information is first translated into a textual form that captures its meaning.
For example:
· An image becomes a descriptive caption
· A voice recording becomes a transcript
· A PDF becomes extracted and cleaned text
· A JSON payload becomes a concise summary
· A video becomes a sequence of observations and spoken dialogue
Once these different inputs have been translated into text, they can be merged into a single prompt that provides the language model with a complete understanding of the situation.
In other words, the LLM is not reasoning over pixels, sound waves, or PDF pages. It is reasoning over the meaning of those inputs after they have been expressed as text.
This is precisely why the Perception Module is so important. Its primary responsibility is not simply receiving information, it is transforming the complexity of the real world into a clean, consistent representation that the language model can understand and reason about.
In the next section, we'll explore how the Perception Module performs this translation through specialized processors, normalization, and preprocessing pipelines that convert diverse inputs into a unified, LLM-readable prompt.
3. The Perception Module: Translating the World
In the previous section, we learned that the real world is messy, unstructured, and multimodal, while a language model performs its best reasoning when information is presented in a clear textual form.
This creates a fundamental architectural challenge. How do we bridge the gap between the complexity of the outside world and the structured reasoning capabilities of an LLM?
The answer is the Perception Module. Rather than simply forwarding incoming data to the language model, the Perception Module acts as an intelligent interpreter. Its responsibility is to understand the type of input it has received, process it using the appropriate tools, remove unnecessary complexity, and translate it into a form that the language model can immediately reason about.
You can think of it as a translator standing between two people who speak completely different languages. One side speaks in images, documents, audio recordings, API responses, and sensor data. The other side, the LLM—understands textual information. The Perception Module ensures that both sides can communicate effectively.
3.1 The Interpreter Between the Real World and the LLM
Imagine a customer submits the following support request:
A short chat message:
· My shirt arrived damaged
· A photograph showing a torn sleeve
· A PDF invoice
· An API response containing the customer's purchase history
Although these inputs describe the same issue, they are completely different formats. If we simply forwarded all of them directly to the language model, it would receive a collection of unrelated files with little context.
Instead, the Perception Module first interprets each input independently, extracts the relevant information, and then combines everything into one coherent understanding.
The language model never needs to worry about file formats, image pixels, PDF layouts, or nested JSON structures. It simply receives the information it needs to solve the problem.
This separation of responsibilities is one of the key architectural principles behind modern AI agents.
· The Perception Module understands data formats.
· The Language Model understands meaning.
Keeping these responsibilities separate results in systems that are easier to maintain, extend, and optimize.
3.2 Normalization: Speaking One Common Language
The primary responsibility of the Perception Module is normalization. Normalization is the process of converting diverse, inconsistent inputs into a single standardized representation.
Think of it as converting multiple human languages into one common language before starting a conversation. Regardless of whether the original input is an image, PDF, audio recording, or JSON document, the output should always be something the language model can easily understand.
Normalization may involve several preprocessing steps, including:
· Correcting spelling mistakes and grammatical errors
· Removing duplicate or irrelevant information
· Cleaning noisy OCR output
· Extracting useful content from documents
· Converting speech into text
· Summarizing large datasets or API responses
· Standardizing dates, currencies, and measurement units
· Combining information from multiple sources into a single narrative
The goal is not only to extract information but to make that information clear, concise, and useful for reasoning.
3.3 Specialized Processors for Different Types of Inputs
Since every input format is different, a single processor cannot handle them all. Instead, the Perception Module is typically built as a pipeline of specialized processors, each designed for a particular type of data.
a. Vision Processor
When the input is an image, the Perception Module first sends it to a vision model. The vision model identifies important objects, scenes, text, colors, relationships, and visual details before generating a textual description.
For example, instead of sending an image file to the LLM, the Vision Processor might produce, "The uploaded image shows a blue shirt with a tear approximately five centimeters long on the left sleeve". This textual description becomes part of the final prompt.
b. Speech-to-Text Processor
If the user uploads an audio recording, the first step is converting spoken words into text.
For example, a customer might say, "Hi, I received my package today, but the product inside is broken". The Speech-to-Text processor converts this into a transcript that the language model can understand.
Additional processing may remove filler words, restore punctuation, or detect the language being spoken.
c. OCR and PDF Processing
PDF documents often contain much more than plain text. They may include:
· Tables
· Images
· Headers
· Footers
· Signatures
· Multi-column layouts
· Scanned pages
The Perception Module uses Optical Character Recognition (OCR) and document parsing techniques to extract meaningful content.
For example, a five-page invoice might be reduced to:
Customer: John Smith
Invoice Number: INV-24581
Product: Premium Shirt
Purchase Date: July 10
Warranty: 30 Days
Rather than forcing the language model to read an entire document, only the relevant information is preserved.
d. JSON Processing
Enterprise AI agents frequently receive information from APIs in JSON format. Consider an order management API that returns hundreds of nested fields. Although the data is perfectly structured for software applications, it is unnecessarily detailed for reasoning.
Instead of passing the raw JSON to the language model, the Perception Module extracts only the important facts. For example, instead of processing hundreds of lines of JSON, the LLM might receive, Customer has placed three previous orders. The latest order contains one blue shirt purchased yesterday. Payment has been completed, and the shipment is currently in transit.
The meaning remains intact while unnecessary complexity is removed.
3.4 Producing a Unified Prompt
Once each processor has completed its task, the Perception Module combines all of the extracted information into a single unified prompt. Instead of receiving multiple disconnected inputs, the language model receives one coherent description of the situation.
Imagine the agent receives:
· Chat message
· Product image
· Invoice PDF
· Customer order details from an API
The Perception Module may produce a prompt like this:
"The customer reports that a recently purchased blue shirt arrived with a tear on the left sleeve. The uploaded image confirms visible damage. The attached invoice shows the purchase was made three days ago and is still within the return period. Customer purchase history indicates this is their third order with no previous return requests."
Although the original information came from four completely different sources, the language model sees a single, consistent narrative.
This dramatically improves the quality of reasoning because the model no longer has to interpret multiple disconnected inputs, it can immediately focus on solving the customer's problem.
3.5 Why This Matters for AI Agent Architecture
One of the biggest mistakes beginners make is assuming that the Perception Module simply forwards data to the language model. In reality, perception is an intelligent preprocessing pipeline.
It identifies the input type, routes it to the appropriate processor, extracts meaningful information, removes unnecessary complexity, normalizes the content, and finally constructs a clean prompt for the LLM.
By the time information reaches the language model, the difficult work of understanding raw data has already been completed. This separation of concerns allows each component to focus on what it does best.
The Perception Module understands files, documents, images, audio, APIs, and other real-world data sources. The Language Model understands language, reasons about context, makes decisions, and generates intelligent responses.
Together, these components enable AI agents to interact naturally with the complex, multimodal world while maintaining a clean and efficient reasoning process.
4. Architecture Walkthrough: A Multimodal Support Agent
So far, we've discussed the purpose of the Perception Module and how it translates different types of inputs into a format that the language model can understand. Now let's see how this works in a real-world application.
Imagine we're building an AI-powered customer support agent for an online clothing store. The agent doesn't just answer questions—it can inspect product images, understand customer messages, analyze documents, and decide the appropriate action.
In this example, a customer receives a damaged shirt and contacts customer support. Instead of relying on only a text conversation, the AI agent can combine multiple sources of information to gain a complete understanding of the problem.
Step 1: The Customer Perceives the Agent
The interaction begins when the customer sends two different inputs.
First, they type a message like "my new shrit has hole on it".
Notice that the message contains several spelling mistakes. Next, they upload a photograph showing the damaged shirt. These two actions represent two different forms of perception.
Sense 1: Text input
Sense 2: Image upload
Together, they trigger the agentic loop and activate the Perception Module. At this point, the language model has not yet received anything.
Step 2: The Perception Module Takes Control
Instead of forwarding the raw inputs directly to the LLM, the Perception Module first determines the type of each input and routes it to the appropriate processor. Think of the Perception Module as an intelligent traffic controller.
Each input follows a different processing pipeline based on its format. The text message is sent to a Text Processor, while the uploaded image is sent to a Vision Processor. Both processors work independently before their results are combined.
Step 3: Cleaning the Text
The Text Processor performs several preprocessing tasks.
It may:
· Correct spelling mistakes.
· Restore punctuation.
· Normalize grammar.
· Remove unnecessary words.
· Detect the language.
· Expand abbreviations if required.
The original message, "my new shrit has hole on it" becomes "My new shirt has a hole in it". Although the meaning hasn't changed, the sentence is now much clearer and easier for the language model to understand.
This simple normalization step improves reasoning quality without changing the customer's intent.
Step 4: Understanding the Image
While the text is being cleaned, the Vision Processor analyzes the uploaded photograph. Unlike humans, the language model cannot directly interpret image pixels. Instead, the Vision Processor examines the image and generates a textual description.
For example, "The uploaded image shows a blue shirt with a tear approximately five centimetres long on the left sleeve. The damage appears to be a manufacturing defect."
Notice what happened. The image itself never reaches the language model. Only its textual interpretation does. From the perspective of the LLM, it simply receives another piece of text describing what was observed.
Step 5: Building a Unified Understanding
Now comes the most important responsibility of the Perception Module. It combines the outputs from all processors into a single, coherent prompt.
Instead of sending multiple disconnected pieces of information, it creates one unified description of the customer's issue. The final prompt might look something like this:
· Customer Message: "My new shirt has a hole in it."
· Image Analysis: The uploaded image confirms a tear on the left sleeve of a blue shirt. The damage appears consistent with a manufacturing defect.
· Summary: The customer reports receiving a damaged product and has provided photographic evidence supporting the claim.
This is the only information sent to the Foundation Model. Notice how clean, organized, and easy it is to understand.
The language model no longer needs to interpret spelling mistakes or analyze an image. That work has already been completed by the Perception Module.
Step 6: The Foundation Model Begins Reasoning
With a normalized prompt, the Foundation Model can now focus on what it does best—reasoning.
It may analyze questions such as:
· Is the customer's complaint valid?
· Does the uploaded image support the claim?
· Is the issue covered under the return policy?
· What is the most appropriate next action?
· Should the customer receive a replacement or a refund?
· Is additional information required?
Since the model receives clean and structured information, it can spend its computational effort making decisions instead of interpreting raw data.
Step 7: The Agent Decides the Next Action
Based on its reasoning, the agent may choose one of several actions.
For example:
· Approve a replacement order
· Initiate a refund
· Generate a return shipping label
· Request additional photographs
· Escalate the case to a human support representative
· Ask the customer for their order number
The exact action depends on the business rules, available tools, and the reasoning performed by the language model.
One of the most important lessons from this example is that the Foundation Model never interacts directly with raw inputs.
· It never sees image pixels
· It never reads PDF layouts
· It never listens to audio recordings
· It never processes raw API responses
Instead, the Perception Module acts as an intelligent translation layer that converts every incoming signal into a consistent, textual representation.
This separation of responsibilities is what makes modern AI agents scalable and extensible. As new input types are introduced such as videos, IoT sensor data, or handwritten documents, you simply add new processors to the Perception Module. The Foundation Model remains unchanged because it continues to receive the same clean, normalized context regardless of where the information originated.
This design allows the agent to "see," "hear," and "understand" an increasingly complex world without increasing the complexity of the reasoning engine itself.
Previous Next Home
No comments:
Post a Comment