Skip to main content
POST
Scrape URL

Request Body

string
required
The URL to scrape. Must be a valid HTTP or HTTPS URL. Private IPs and cloud metadata endpoints are blocked for security.
string
default:"auto"
Scraping engine tier to use. ScrapeBadger automatically selects the best approach.In auto mode, simple pages use fast HTTP (1 credit) and JavaScript-heavy pages use a browser (5 credits). When escalate is enabled, the system can further escalate to premium browsers (10 credits) for heavily protected sites.
string
default:"html"
Output format for the scraped content.
  • html — Raw HTML of the page
  • markdown — Converted to clean Markdown
  • text — Plain text with HTML tags stripped
boolean
default:false
Force JavaScript rendering before extracting content. Automatically switches to the browser engine. Use this for single-page applications or pages that load content dynamically.
string
CSS selector or XPath expression to wait for before extracting content. Only works with browser engines (browser, patchright, windows_chrome). If render_js is false and this is set, JS rendering is forced automatically.
integer
default:30000
Maximum time in milliseconds to wait for the wait_for selector to appear. Range: 1000120000.
integer
Additional milliseconds to wait after the page has finished loading, before extracting content. Useful for pages with animations or delayed rendering. Only works with browser engines. Range: 030000.
array
A list of browser actions to perform before extracting content. Forces the browser engine. Each action is an object with a type and action-specific parameters.Supported actions:
string
A unique identifier to persist cookies, fingerprint, and browser storage across multiple requests. Use the same session_id to maintain login state or continue a browsing session.
integer
default:3
Maximum number of retry attempts when a blocking page is detected. Retries are free — you only pay for the final successful engine. Range: 010.
boolean
default:true
Whether to automatically retry when a blocking page is detected. Set to false to get the blocked response immediately.
string
ISO 3166-1 alpha-2 country code for proxy geo-targeting. Routes the request through a proxy in the specified country.
Common values: US, GB, DE, FR, JP, BR, AU.
object
Additional HTTP headers to include in the request to the target URL. Accepts a key-value object.
boolean
default:false
Capture a full-page screenshot (PNG). Forces the browser engine. Returned as base64 in the screenshot_url response field.
boolean
default:false
Record a video of the browser session (animated GIF). Forces the browser engine. Returned as base64 in the video_url response field. Adds +3 credits. Useful for debugging, visual verification, or monitoring how a page loads.
boolean
default:false
Attempt to bypass detected anti-bot protection using registered solvers. Adds +5 credits to the request cost when a solver is invoked. Only triggered when blocking is actually detected.
boolean
default:false
Allow automatic escalation to more powerful engines when the initial engine is blocked.Escalation path: curl_cffibrowserwindows_chromeYou only pay for the engine that succeeds — costs are not cumulative. Without this flag, only the selected engine is tried.
integer
Maximum credits to spend on this request. The request fails with a 400 error if the estimated cost would exceed this budget. Useful for controlling costs when using escalate or anti_bot. Minimum: 1.
boolean
default:false
Run AI-powered extraction on the scraped content using the instruction in ai_prompt. Adds +2 credits to the request cost. The scrape result is still returned even if AI extraction fails.
string
Natural language instruction for AI data extraction. Required when ai_extract is true. Maximum 2000 characters.

Response

boolean
Whether the scrape completed successfully. false when all retries are exhausted and the page is still blocked.
string
The final URL after any redirects.
integer
HTTP status code from the target URL.
string
The scraped content in the requested format. null when success is false.
string
The output format used: html, markdown, or text.
string
The engine tier that produced the final result.
integer
Total credits charged for this request, including engine cost, solver, and AI extraction.
integer
Total request processing time in milliseconds.
integer
Number of retry attempts performed. 0 if the first attempt succeeded.
integer
Size of the returned content in bytes.
string
Base64-encoded PNG screenshot of the page. Only present when screenshot: true was requested.
string
Base64-encoded animated GIF of the browser session. Only present when video: true was requested.
object
HTTP response headers from the target URL.
boolean
Whether a blocking page was detected during scraping.
object
Details about the detected blocking page. Only present when blocking_detected is true.
array
List of anti-bot systems detected on the page.
array
List of CAPTCHA systems detected on the page.
boolean
Whether the anti-bot solver successfully bypassed the protection.
string
Name of the solver that successfully bypassed the block. null if no solver was used.
object | string | array
Structured data extracted by the LLM based on ai_prompt. The shape depends on your prompt. null when ai_extract is false or extraction failed.
string
The LLM model used for extraction (e.g., gpt-4o-mini). null when AI extraction was not used.
string
Error message if AI extraction failed. The scrape result is still returned. null on success.

Examples

Basic scrape

JavaScript rendering with wait

AI extraction

Full anti-bot bypass with budget

Browser automation scenario

Error Responses

Authorizations

x-api-key
string
header
required

Body

application/json
url
string
required

The URL to scrape. Must be a valid HTTP or HTTPS URL.

engine
enum<string>
default:auto

Scraping engine tier to use.

Available options:
auto,
browser
format
enum<string>
default:html

Output format for the scraped content.

Available options:
html,
markdown,
text
render_js
boolean
default:false

Force JavaScript rendering.

wait_for
string

CSS selector or XPath to wait for before extracting.

wait_timeout
integer
default:30000

Max wait time in ms for wait_for selector.

wait_after_load
integer

Additional ms to wait after page load.

js_scenario
object[]

Browser actions to perform before extracting.

session_id
string

Persist cookies and state across requests.

retry_count
integer
default:3

Max retry attempts on blocking detection.

retry_on_block
boolean
default:true

Auto-retry on blocking page detection.

country
string

ISO 3166-1 alpha-2 country code for proxy geo-targeting.

custom_headers
object

Additional HTTP headers for the target request.

screenshot
boolean
default:false

Capture a full-page PNG screenshot.

video
boolean
default:false

Record browser session as animated GIF (+3 credits).

anti_bot
boolean
default:false

Attempt anti-bot bypass when blocking detected.

escalate
boolean
default:false

Allow auto-escalation to stronger engines.

max_cost
integer

Maximum credits budget for this request.

ai_extract
boolean
default:false

Run AI extraction on scraped content.

ai_prompt
string

Natural language instruction for AI extraction.

Response

200 - application/json

Successful scrape

success
boolean
url
string
status_code
integer
content
string
format
string
engine_used
string
credits_used
integer
duration_ms
integer
retries_used
integer
content_length
integer
screenshot_url
string | null
video_url
string | null
headers
object
blocking_detected
boolean
blocking_details
object | null
antibot_systems
array
captcha_systems
array
anti_bot_solved
boolean
solver_used
string | null
ai_extraction
unknown
ai_model
string | null
ai_error
string | null