GlossaryAI search
What is an AI agent?
Also called: agentic AI, LLM agent, browser agent
Definition
An AI agent is a system in which a language model decides its own next steps and uses tools, such as web browsing, search, code or APIs, to complete a task with limited human direction.
AI agent, explained
Anthropic's guide "Building effective agents" draws a useful line. Workflows are systems where LLMs and tools are orchestrated through predefined code paths. Agents are systems where LLMs dynamically direct their own processes and tool usage, keeping control over how they accomplish a task. A fixed pipeline that summarizes a page is a workflow. A model that decides which pages to read, what to compare and when it's done is an agent.
Agents already touch your website. Research features like deep research browse dozens of pages to write a report. Coding assistants read documentation to integrate a library. Browser agents fill forms, compare prices and click through checkouts on a user's behalf. They fetch pages with user-triggered crawlers that often behave differently from search bots.
What makes a site agent-friendly is mostly what makes it accessible: real text in the HTML, clear headings, labeled form fields, predictable URLs, and facts like pricing and limits stated plainly rather than hidden in images or behind scripts. Docs with clean markdown versions or an llms.txt file help coding agents in particular.
There are decisions to make, too. Some sites block automated agents at the CDN, sometimes by accident. If you sell to developers or businesses, blocking agents may block the buyers who send them. Check what your bot protection does to user-triggered fetchers like ChatGPT-User, Claude-User and Perplexity-User.
For builders, the same guide's advice holds up: start with the simplest workflow that works, and only add agentic autonomy when the task genuinely needs it.
Why it matters for founders
Buyers increasingly send agents to research and compare products for them. If agents can't read your pricing or docs, you're left out of the shortlist before a human looks.
Example
A buyer asks an agent to shortlist three analytics tools under a budget. The agent reads each vendor's pricing page. One vendor's prices only appear after a JavaScript toggle, so the agent marks it "pricing unclear".
Common mistakes
- Hiding pricing and limits in images or behind interactions agents can't perform.
- Blocking user-triggered AI fetchers at the CDN without meaning to.
- Building a fully autonomous agent where a fixed workflow would do.
Sources
Checked
Related terms
- Deep research (AI)Deep research is an AI assistant mode that runs many web searches over several minutes, reads a large number of sources, and writes a long report with citations, instead of a quick single answer.
- Model Context Protocol (MCP)The Model Context Protocol (MCP) is an open standard for connecting AI applications to external data sources, tools and workflows, so an assistant can read data or take actions in another product through one common interface.
- AI crawlersAI crawlers are bots run by AI companies that fetch web pages, either to collect training data, to build an index for AI search answers, or to read a page a user asked about. Each has its own robots.txt name.
- llms.txtllms.txt is a proposed markdown file at a site's root, /llms.txt, that summarizes the site and links to its most useful pages for AI agents. It's voluntary, grants no access, and Google Search ignores it.