User Agent

What is a User Agent?

The User-Agent string tells servers what software is making the request - browser, app, bot, or tool. It’s like an ID badge that describes the client’s identity.

MDN Web DocsRead more on User-Agent.

Anatomy of a User-Agent String

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36

Breaking this down:

Mozilla/5.0 - Historical compatibility token
Windows NT 10.0; Win64; x64 - Operating system
AppleWebKit/537.36 - Rendering engine
Chrome/91.0.4472.124 - Browser and version
Safari/537.36 - Compatibility indicator

Common User-Agent Types

Legitimate Browsers

Chrome:  Mozilla/5.0 ... Chrome/120.0.0.0 Safari/537.36
Firefox: Mozilla/5.0 ... Firefox/120.0
Safari:  Mozilla/5.0 ... Version/17.0 Safari/605.1.15
Edge:    Mozilla/5.0 ... Edg/120.0.0.0

Mobile Apps

iOS:     MyApp/1.0 (iPhone; iOS 17.0)
Android: MyApp/1.0 (Linux; Android 14)

Bots & Crawlers

Googlebot:   Googlebot/2.1
Bingbot:     Mozilla/5.0 ... bingbot/2.0
Python:      python-requests/2.28.0
curl:        curl/7.84.0

Automated Tools

Postman:     PostmanRuntime/7.29.0
Selenium:    Mozilla/5.0 ... HeadlessChrome/91.0
Puppeteer:   Mozilla/5.0 ... HeadlessChrome/91.0

Using User-Agent in Policies

Esper supports string matching on User-Agent values with:

equals
not equals
contains
starts with
ends with

Basic Examples

Block known bad bots:

Field Type: User Agent
Operator: contains
Value: BadBot
Action: Block

Detect automated tools:

Field Type: User Agent
Operator: contains
Value: HeadlessChrome
OR
Value: PhantomJS
OR
Value: Selenium
Action: Challenge

Advanced Patterns

Require modern browsers:

Field Type: User Agent
Operator: contains
Value: Chrome/
AND
Field Type: User Agent
Operator: not equals
Value: Chrome/49.0
Action: Allow

API client validation:

Field Type: Request Path
Operator: equals
Value: /api
AND
Field Type: User Agent
Operator: starts with
Value: MyApp/
Action: Block

Traffic Protection TipMany malicious bots use fake or outdated User-Agent strings. Look for inconsistencies like Chrome version 40 (from 2015) or misspellings.

Security Patterns

Empty or Missing User-Agents

Field Type: User Agent
Operator: equals
Value: ""
OR
Operator: equals
Value: -
Action: Challenge

Suspicious Patterns

# Contains SQL/script keywords
Field Type: User Agent
Operator: contains
Value: SELECT
OR
Value: <script
Action: Block

Known Attack Tools

Field Type: User Agent
Operator: contains
Value: sqlmap
OR
Operator: contains
Value: nikto
OR
Operator: contains
Value: burp
Action: Block

Best Practices

DO:

Check for empty/null - Many attacks omit User-Agent
Validate format - Real browsers have consistent patterns
Monitor changes - Sudden User-Agent switches are suspicious
Allow legitimate bots - Google, Bing need access for SEO
Log everything - User-Agents help with forensics
Update patterns - Browsers release new versions monthly

DON’T:

Trust blindly - User-Agents are easily spoofed
Block all bots - Some are necessary (search engines)
Use exact matching - Versions change frequently
Ignore mobile - Different patterns than desktop
Forget about apps - Native apps have custom User-Agents

Bot Management

Good Bots to Allow

Bot	Purpose	User-Agent Pattern
Googlebot	Search indexing	`Googlebot/`
Bingbot	Search indexing	`bingbot/`
Slackbot	Link previews	`Slackbot`
FacebookBot	Social previews	`facebookexternalhit`
TwitterBot	Tweet cards	`Twitterbot`

Bad Bot Indicators

Generic: Mozilla/5.0 only
Outdated: Browser versions from years ago
Malformed: Missing expected components
Tools: Development/testing tools
Suspicious: Known attack signatures

User-Agent Consistency Checks

Session Validation

# User-Agent changed mid-session
Field Type: Cookie
Field Reference: session_id
Operator: is present
AND
User-Agent different from session start
Action: Challenge

Browser Feature Validation

# Claims Chrome but wrong headers
Field Type: User Agent
Operator: contains
Value: Chrome
AND
Field Type: Header
Field Reference: Sec-CH-UA
Operator: is not present
Action: Monitor  # Chrome sends Client Hints

Mobile Detection

Identifying Mobile Traffic

Field Type: User Agent
Operator: contains
Value: Mobile
OR
Operator: contains
Value: Android
OR
Operator: contains
Value: iPhone
OR
Operator: contains
Value: iPad
Action: Apply mobile rules

Tablet vs Phone

# iPads
Field Type: User Agent
Operator: contains
Value: iPad

# iPhones
Field Type: User Agent
Operator: contains
Value: iPhone

Common Attack Patterns

User-Agent Injection

# Detect injection attempts
Field Type: User Agent
Operator: contains
Value: \n
OR
Operator: contains
Value: \r
Action: Block # Contains line breaks

Crawler Pretending to be Browser

# Fast requests with browser User-Agent
Field Type: User Agent
Operator: contains
Value: Chrome
AND
Request rate > 60/minute
Action: Challenge

Troubleshooting

“Blocking real users”

Check for browser updates
Mobile apps might have custom strings
Browser extensions can modify User-Agent
Privacy tools randomize User-Agents

“Bots bypassing detection”

They might rotate User-Agents
Using real browser automation
Copying legitimate User-Agents
Need additional signals beyond User-Agent

Advanced Detection

Browser Fingerprinting

Combine User-Agent with:

JavaScript capabilities
Screen resolution
Timezone
Installed plugins
Canvas fingerprint

Behavioral Analysis

# Browser User-Agent but API behavior
Field Type: User Agent
Operator: contains
Value: Mozilla
AND
Never loads images/CSS/JS
Action: Challenge

Headers - Modern browsers send additional identity headers
Client IP - Combine for stronger traffic protection
Request Path - Bots often hit specific paths
Referrer - Bots often lack proper referrers

Settings

Policies

Services

Integrations

CLI

What is a User Agent?

Anatomy of a User-Agent String

Common User-Agent Types

Legitimate Browsers

Mobile Apps

Bots & Crawlers

Automated Tools

Using User-Agent in Policies

Basic Examples

Advanced Patterns

Security Patterns

Empty or Missing User-Agents

Suspicious Patterns

Known Attack Tools

Best Practices

DO:

DON’T:

Bot Management

Good Bots to Allow

Bad Bot Indicators

User-Agent Consistency Checks

Session Validation

Browser Feature Validation

Mobile Detection

Identifying Mobile Traffic

Tablet vs Phone

Common Attack Patterns

User-Agent Injection

Crawler Pretending to be Browser

Troubleshooting

Advanced Detection

Browser Fingerprinting

Behavioral Analysis

​What is a User Agent?

​Anatomy of a User-Agent String

​Common User-Agent Types

​Legitimate Browsers

​Mobile Apps

​Bots & Crawlers

​Automated Tools

​Using User-Agent in Policies

​Basic Examples

​Advanced Patterns

​Security Patterns

​Empty or Missing User-Agents

​Suspicious Patterns

​Known Attack Tools

​Best Practices

​DO:

​DON’T:

​Bot Management

​Good Bots to Allow

​Bad Bot Indicators

​User-Agent Consistency Checks

​Session Validation

​Browser Feature Validation

​Mobile Detection

​Identifying Mobile Traffic

​Tablet vs Phone

​Common Attack Patterns

​User-Agent Injection

​Crawler Pretending to be Browser

​Troubleshooting

​Advanced Detection

​Browser Fingerprinting

​Behavioral Analysis

​Related Fields

What is a User Agent?

Anatomy of a User-Agent String

Common User-Agent Types

Legitimate Browsers

Mobile Apps

Bots & Crawlers

Automated Tools

Using User-Agent in Policies

Basic Examples

Advanced Patterns

Security Patterns

Empty or Missing User-Agents

Suspicious Patterns

Known Attack Tools

Best Practices

DO:

DON’T:

Bot Management

Good Bots to Allow

Bad Bot Indicators

User-Agent Consistency Checks

Session Validation

Browser Feature Validation

Mobile Detection

Identifying Mobile Traffic

Tablet vs Phone

Common Attack Patterns

User-Agent Injection

Crawler Pretending to be Browser

Troubleshooting

Advanced Detection

Browser Fingerprinting

Behavioral Analysis

Related Fields