Web Scraping — Adaptive Strategy

What it does

This skill guides an agent through an adaptive, phased web-scraping reconnaissance and implementation workflow. It starts with lightweight HTTP checks (curl), escalates to browser reconnaissance and traffic interception when needed, discovers APIs/endpoints, validates selectors/JSON paths, and documents a repeatable extraction strategy. It also includes guidance for turning a working scraper into a production Apify Actor (TypeScript-first).

When to use it

Use when you need to extract structured data from a website, investigate blocking/403 issues, find APIs behind a site, or convert an ad-hoc scraper into a production actor. Triggers include: "scrape [site]", "extract data from", "I'm getting blocked", and "Make this an Apify Actor".

What's included

• Scripts: none bundled in the repo (has_scripts=false) but the skill contains runnable examples and patterns (examples/*, apify templates).
• References: in-repo references and strategy docs (has_references=false for top-level flag, but many subfiles are referenced in the SKILL.md).
• Instructions: a detailed 6-phase reconnaissance workflow (Phases 0-5), validation steps, protection-testing guidance, report schema, and productionization checklist for Apify Actors.

Compatible agents

Works well for agents that can run shell and Node workflows (Claude Code, Copilot/Code-writing agents, agents that can run Playwright/Crawlee examples).