Build Scrapy spiders with AI
Scrapy Agent Skills started as a Claude Code plugin from Zyte. They're now published as open Agent Skills, so the same schema discovery, scrapy-poet page objects, and pytest fixtures work wherever you already build.
How the skill works
Scrapy Skills, Claude Skills, and Copilot Skills all run the exact same pipeline underneath — only the packaging changes to fit your agent.
Schema definition
You give it a URL and describe what you want in plain English. It examines a detail page, discovers the available fields, and asks you to approve the schema before generating any code.
Website analysis
It downloads a handful of pages across the site, compares variants, and validates the extraction spec — with an optional browser review if you want to check it visually.
Project setup
Sets up a real Scrapy project with the dependencies it needs, including scrapy-poet and Zyte API support.
Code generation
Generates web-poet page objects from the validated spec — no boilerplate, no manual selector hunting.
Spider creation
Wires the page objects together into a runnable spider, plus a passing test suite you can run straight away.
Pick your agent
Scrapy Skills
The open Agent Skills package. Install it for whichever agent you use from a single registry.
$npx skills add zytedata/skills
Claude Skills
Install as a Claude Code plugin for schema discovery, page objects, and tests generated straight in your IDE.
$claude plugin marketplace add zytedata/claude-skills$claude plugin install zyte-web-data@zyte-ai
Copilot Skills
The same skillset packaged for GitHub Copilot's coding agent.
$copilot plugin marketplace add zytedata/copilot-skills$copilot plugin install zyte-web-data@zyte-ai
Also available: a dedicated VS Code extension
Prefer a point-and-click workflow inside your editor?
Web Scraping Copilot
Generate, test, and deploy production-ready Scrapy spiders without leaving your IDE — with full developer control.
- Generate selectors
- Unblock sites
- Deploy to Scrapy Cloud