GlossarySEO
What is a user agent?
Also called: user-agent string, UA string, user-agent token
Definition
A user agent is the software making a web request, identified by the User-Agent HTTP header. Browsers and crawlers each send their own string, and robots.txt rules target crawlers by their user-agent token.
User agent, explained
Every request includes a User-Agent header describing the client. A browser's string names the browser engine and operating system. A crawler's string names the bot, like Googlebot, bingbot or GPTBot, usually with a link to its documentation. Servers use it for analytics, bot detection and, occasionally, serving different layouts.
In robots.txt, each group starts with a User-agent: line. That line uses a short product token rather than the full string: User-agent: Googlebot, User-agent: GPTBot, User-agent: * for everyone else. A crawler follows the most specific group that matches its token and ignores the others, so a bot with its own group doesn't also read the * group. That catches people out when they add AI bot rules and accidentally drop their general rules for those bots.
User-agent strings are self-reported and easy to fake. Scrapers often pretend to be Googlebot to get past blocks. For decisions that matter, like rate limiting or allowlisting, verify the IP: Google and Bing both document reverse DNS checks, and several AI companies publish IP ranges.
Serving different content based on user agent is risky. Showing crawlers something substantially different from users is cloaking under Google's spam policies. Responsive design avoids the question entirely, because every client gets the same HTML and the layout adapts in CSS.
Why it matters for founders
The user agent is how you tell Googlebot from GPTBot from a scraper. It's the key to reading your logs and writing robots.txt rules that do what you meant.
Example
You want to block OpenAI's training crawler but not ChatGPT search. You add a User-agent: GPTBot group with Disallow: / and a separate User-agent: OAI-SearchBot group with Allow: /.
Common mistakes
- Trusting the user-agent string for security decisions.
- Forgetting that a bot-specific group replaces the
*group for that bot. - Serving crawlers different content than users.
- Using the full UA string instead of the product token in robots.txt.
Sources
- Google Search Central: How Google interprets the robots.txt specification
- Google Search Central: Verifying Googlebot and other Google crawlers
Checked
Related terms
- Web crawlerA web crawler is a program that fetches web pages automatically, follows their links to find more pages, and passes what it finds to a search engine or AI system for indexing or analysis.
- GooglebotGooglebot is Google's main web crawler for Search. It comes in a smartphone and a desktop version, and since mobile-first indexing most sites are crawled by Googlebot Smartphone.
- robots.txtrobots.txt is a plain-text file at the root of a domain that tells crawlers which URLs they may and may not fetch. It controls crawling, not indexing, and well-behaved bots follow it voluntarily.
- CloakingCloaking is showing search engines different content from what users see, in order to manipulate rankings. It's a named violation in Google's spam policies and can lead to a manual action.