Web Scraping Skill v3.0

Usage

/web-scrape <url> [options]

Options:

• --format=markdown|json|text - Output format (default: markdown)
• --full - Include full page content (skip smart extraction)
• --screenshot - Also save a screenshot
• --scroll - Scroll to load dynamic content (infinite scroll pages)

Examples:

/web-scrape https://example.com/article
/web-scrape https://news.site.com/story --format=json
/web-scrape https://spa-app.com/page --scroll --screenshot

---

Execution Flow

Phase 1: Navigate and Load

1. mcp__playwright__browser_navigate
   url: "<target URL>"

2. mcp__playwright__browser_wait_for
   time: 2  (allow initial render)

If --scroll option: Execute scroll sequence to trigger lazy loading:

3. mcp__playwright__browser_evaluate
   function: "async () => {
     for (let i = 0; i < 3; i++) {
       window.scrollTo(0, document.body.scrollHeight);
       await new Promise(r => setTimeout(r, 1000));
     }
     window.scrollTo(0, 0);
   }"

Phase 2: Capture Content

4. mcp__playwright__browser_snapshot
   → Returns full accessibility tree with all text content

If --screenshot option:

5. mcp__playwright__browser_take_screenshot
   filename: "scraped_<domain>_<timestamp>.png"
   fullPage: true

Phase 3: Close Browser

6. mcp__playwright__browser_close

---

Smart Content Extraction

After getting the snapshot, apply intelligent extraction:

Step 1: Identify Content Type

Page Type	Indicators	Extraction Strategy
Article/Blog	<article>, long paragraphs, date/author	Extract main article body
Product Page	Price, "Add to Cart", specs	Extract title, price, description, specs
Documentation	Code blocks, headings hierarchy	Preserve structure and code
List/Search	Repeated item patterns	Extract as structured list
Landing Page	Hero section, CTAs	Extract key messaging

Step 2: Filter Noise

ALWAYS REMOVE these elements from output:

• Navigation menus and breadcrumbs
• Footer content (copyright, links)
• Sidebars (ads, related articles, social links)
• Cookie banners and popups
• Comments section (unless specifically requested)
• Share buttons and social widgets
• Login/signup prompts

Step 3: Structure the Content

For Articles:

# [Title]

**Source:** [URL]
**Date:** [if available]
**Author:** [if available]

---

[Main content in clean markdown]

For Product Pages:

# [Product Name]

**Price:** [price]
**Availability:** [in stock/out of stock]

## Description
[product description]

## Specifications
| Spec | Value |
|------|-------|
| ... | ... |

---

Output Formats

Markdown (default)

Clean, readable markdown with proper headings, lists, and formatting.

JSON

{
  "url": "https://...",
  "title": "Page Title",
  "type": "article|product|docs|list",
  "content": {
    "main": "...",
    "metadata": {}
  },
  "extracted_at": "ISO timestamp"
}

Text

Plain text with minimal formatting, suitable for further processing.

---

Error Handling

Navigation Errors

Error	Detection	Action
Timeout	Page doesn't load in 30s	Report error, suggest retry
404 Not Found	"404" in title/content	Report "Page not found"
403 Forbidden	"403", "Access Denied"	Report access restriction
CAPTCHA	"captcha", "verify you're human"	Report CAPTCHA detected, cannot proceed
Paywall	"subscribe", "premium content"	Extract visible content, note paywall

Recovery Actions

If page load fails:
1. Report the specific error to user
2. Suggest: "Try again?" or "Different URL?"
3. Close browser cleanly

If content is blocked:
1. Report what was detected (CAPTCHA/paywall/geo-block)
2. Extract any available preview content
3. Suggest alternatives if applicable

---

Advanced Scenarios

Single Page Applications (SPA)

1. Navigate to URL
2. Wait longer (3-5 seconds) for JS hydration
3. Use browser_wait_for with specific text if known
4. Then snapshot

Infinite Scroll Pages

1. Navigate
2. Execute scroll loop (see Phase 1)
3. Snapshot after scrolling completes

Pages with Click-to-Reveal Content

1. Snapshot first to identify clickable elements
2. Use browser_click on "Read more" / "Show all" buttons
3. Wait briefly
4. Snapshot again for full content

Multi-page Articles

1. Scrape first page
2. Identify "Next" or pagination links
3. Ask user: "Article has X pages. Scrape all?"
4. If yes, iterate through pages and combine

---

Performance Guidelines

Metric	Target	How
Speed	< 15 seconds	Minimal waits, parallel where possible
Token Usage	< 5000 tokens	Smart extraction, not full DOM
Reliability	> 95% success	Proper error handling

---

Security Notes

• Never execute arbitrary JavaScript from the page
• Don't follow redirects to suspicious domains
• Don't submit forms or click login buttons
• Don't scrape pages that require authentication (unless user provides credentials flow)
• Respect robots.txt when mentioned by user

---

Quick Reference

Minimum viable scrape (4 tool calls):

1. browser_navigate → 2. browser_wait_for → 3. browser_snapshot → 4. browser_close

Full-featured scrape (with scroll + screenshot):

1. browser_navigate
2. browser_wait_for
3. browser_evaluate (scroll)
4. browser_snapshot
5. browser_take_screenshot
6. browser_close

Remember: The goal is to deliver clean, useful content to the user, not raw HTML/DOM dumps.