Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.esperr.com/llms.txt

Use this file to discover all available pages before exploring further.

​
What is a User Agent?

The User-Agent string tells servers what software is making the request - browser, app, bot, or tool. It’s like an ID badge that describes the client’s identity.
MDN Web DocsRead more on User-Agent.

​
Anatomy of a User-Agent String

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36
Breaking this down:
  • Mozilla/5.0 - Historical compatibility token
  • Windows NT 10.0; Win64; x64 - Operating system
  • AppleWebKit/537.36 - Rendering engine
  • Chrome/91.0.4472.124 - Browser and version
  • Safari/537.36 - Compatibility indicator

​
Common User-Agent Types

​
Legitimate Browsers

Chrome:  Mozilla/5.0 ... Chrome/120.0.0.0 Safari/537.36
Firefox: Mozilla/5.0 ... Firefox/120.0
Safari:  Mozilla/5.0 ... Version/17.0 Safari/605.1.15
Edge:    Mozilla/5.0 ... Edg/120.0.0.0

​
Mobile Apps

iOS:     MyApp/1.0 (iPhone; iOS 17.0)
Android: MyApp/1.0 (Linux; Android 14)

​
Bots & Crawlers

Googlebot:   Googlebot/2.1
Bingbot:     Mozilla/5.0 ... bingbot/2.0
Python:      python-requests/2.28.0
curl:        curl/7.84.0

​
Automated Tools

Postman:     PostmanRuntime/7.29.0
Selenium:    Mozilla/5.0 ... HeadlessChrome/91.0
Puppeteer:   Mozilla/5.0 ... HeadlessChrome/91.0

​
Using User-Agent in Policies

Esper supports string matching on User-Agent values with:
  • equals
  • not equals
  • contains
  • starts with
  • ends with

​
Basic Examples

Block known bad bots: 
Field Type: User Agent
Operator: contains
Value: BadBot
Action: Block
Detect automated tools: 
Field Type: User Agent
Operator: contains
Value: HeadlessChrome
OR
Value: PhantomJS
OR
Value: Selenium
Action: Challenge

​
Advanced Patterns

Require modern browsers: 
Field Type: User Agent
Operator: contains
Value: Chrome/
AND
Field Type: User Agent
Operator: not equals
Value: Chrome/49.0
Action: Allow
API client validation: 
Field Type: Request Path
Operator: equals
Value: /api
AND
Field Type: User Agent
Operator: starts with
Value: MyApp/
Action: Block
Traffic Protection TipMany malicious bots use fake or outdated User-Agent strings. Look for inconsistencies like Chrome version 40 (from 2015) or misspellings.

​
Security Patterns

​
Empty or Missing User-Agents

Field Type: User Agent
Operator: equals
Value: ""
OR
Operator: equals
Value: -
Action: Challenge

​
Suspicious Patterns

# Contains SQL/script keywords
Field Type: User Agent
Operator: contains
Value: SELECT
OR
Value: <script
Action: Block

​
Known Attack Tools

Field Type: User Agent
Operator: contains
Value: sqlmap
OR
Operator: contains
Value: nikto
OR
Operator: contains
Value: burp
Action: Block

​
Best Practices

​
DO:

  • Check for empty/null - Many attacks omit User-Agent
  • Validate format - Real browsers have consistent patterns
  • Monitor changes - Sudden User-Agent switches are suspicious
  • Allow legitimate bots - Google, Bing need access for SEO
  • Log everything - User-Agents help with forensics
  • Update patterns - Browsers release new versions monthly

​
DON’T:

  • Trust blindly - User-Agents are easily spoofed
  • Block all bots - Some are necessary (search engines)
  • Use exact matching - Versions change frequently
  • Ignore mobile - Different patterns than desktop
  • Forget about apps - Native apps have custom User-Agents

​
Bot Management

​
Good Bots to Allow

BotPurposeUser-Agent Pattern
GooglebotSearch indexingGooglebot/
BingbotSearch indexingbingbot/
SlackbotLink previewsSlackbot
FacebookBotSocial previewsfacebookexternalhit
TwitterBotTweet cardsTwitterbot

​
Bad Bot Indicators

  • Generic: Mozilla/5.0 only
  • Outdated: Browser versions from years ago
  • Malformed: Missing expected components
  • Tools: Development/testing tools
  • Suspicious: Known attack signatures

​
User-Agent Consistency Checks

​
Session Validation

# User-Agent changed mid-session
Field Type: Cookie
Field Reference: session_id
Operator: is present
AND
User-Agent different from session start
Action: Challenge

​
Browser Feature Validation

# Claims Chrome but wrong headers
Field Type: User Agent
Operator: contains
Value: Chrome
AND
Field Type: Header
Field Reference: Sec-CH-UA
Operator: is not present
Action: Monitor  # Chrome sends Client Hints

​
Mobile Detection

​
Identifying Mobile Traffic

Field Type: User Agent
Operator: contains
Value: Mobile
OR
Operator: contains
Value: Android
OR
Operator: contains
Value: iPhone
OR
Operator: contains
Value: iPad
Action: Apply mobile rules

​
Tablet vs Phone

# iPads
Field Type: User Agent
Operator: contains
Value: iPad

# iPhones
Field Type: User Agent
Operator: contains
Value: iPhone

​
Common Attack Patterns

​
User-Agent Injection

# Detect injection attempts
Field Type: User Agent
Operator: contains
Value: \n
OR
Operator: contains
Value: \r
Action: Block # Contains line breaks

​
Crawler Pretending to be Browser

# Fast requests with browser User-Agent
Field Type: User Agent
Operator: contains
Value: Chrome
AND
Request rate > 60/minute
Action: Challenge

​
Troubleshooting

“Blocking real users”
  • Check for browser updates
  • Mobile apps might have custom strings
  • Browser extensions can modify User-Agent
  • Privacy tools randomize User-Agents
“Bots bypassing detection”
  • They might rotate User-Agents
  • Using real browser automation
  • Copying legitimate User-Agents
  • Need additional signals beyond User-Agent

​
Advanced Detection

​
Browser Fingerprinting

Combine User-Agent with:
  • JavaScript capabilities
  • Screen resolution
  • Timezone
  • Installed plugins
  • Canvas fingerprint

​
Behavioral Analysis

# Browser User-Agent but API behavior
Field Type: User Agent
Operator: contains
Value: Mozilla
AND
Never loads images/CSS/JS
Action: Challenge
  • Headers - Modern browsers send additional identity headers
  • Client IP - Combine for stronger traffic protection
  • Request Path - Bots often hit specific paths
  • Referrer - Bots often lack proper referrers