What you need

Python 3.9 or newer and two libraries: requests downloads pages and beautifulsoup4 parses HTML. Install them in a virtual environment:

Shell
python -m venv venv
source venv/bin/activate   # on Windows: venv\Scripts\activate
pip install requests beautifulsoup4

We will scrape books.toscrape.com, a sandbox store built for practising scraping: 1,000 books across 50 pages, each with a title, price, star rating and stock status.

Step 1: Inspect the page

Open the page in your browser, right-click a book and choose Inspect. Each book is an <article class="product_pod">. Inside it, the title is in the title attribute of the h3 a link, the price is in .price_color, the rating is a class such as star-rating Three, and the next page link is li.next a. These CSS selectors are all the scraper needs.

Step 2: Web scraping with Python and BeautifulSoup

The script below downloads each page, extracts every book, follows the next-page link until there is none, waits one second between requests and saves everything to books.csv:

Python · requests + BeautifulSoup
import csv
import time
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup

START_URL = "https://books.toscrape.com/catalogue/page-1.html"
HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; my-scraper/1.0)"}
RATINGS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}

def scrape_page(url):
    response = requests.get(url, headers=HEADERS, timeout=20)
    response.raise_for_status()
    soup = BeautifulSoup(response.content, "html.parser")  # bytes: lets bs4 read the page charset

    books = []
    for card in soup.select("article.product_pod"):
        link = card.select_one("h3 a")
        books.append({
            "title": link["title"],
            "price": float(card.select_one(".price_color").get_text(strip=True).lstrip("£")),
            "rating": RATINGS.get(card.select_one("p.star-rating")["class"][1]),
            "in_stock": "In stock" in card.select_one(".availability").get_text(),
            "url": urljoin(url, link["href"]),
        })

    next_link = soup.select_one("li.next a")
    next_url = urljoin(url, next_link["href"]) if next_link else None
    return books, next_url

all_books, url = [], START_URL
while url:
    books, url = scrape_page(url)
    all_books.extend(books)
    print(f"{len(all_books)} books so far")
    time.sleep(1)  # be polite: one request per second

with open("books.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=all_books[0].keys())
    writer.writeheader()
    writer.writerows(all_books)

Running it prints a running count and finishes with 1,000 rows in about a minute. Three details matter on real sites:

  • response.content, not response.text. This site does not declare its character set in the HTTP headers, so requests guesses wrong and the pound sign comes out as £. Passing bytes lets BeautifulSoup read the charset from the page itself.
  • urljoin for links. Links on the page are relative; joining them with the current URL gives absolute links that work.
  • raise_for_status() and a timeout. The script stops on errors instead of silently saving empty data, and never hangs forever.

Step 3: Scrape JavaScript-rendered pages

Many modern sites send an almost empty HTML page and build the content with JavaScript. The JavaScript version of the practice site, quotes.toscrape.com/js, shows ten quotes in a browser, but requests plus BeautifulSoup finds zero div.quote elements in the downloaded HTML. You need a real browser. Install Playwright with pip install playwright and playwright install chromium, then:

Python · Playwright
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://quotes.toscrape.com/js/")
    page.wait_for_selector("div.quote")  # wait until JavaScript has rendered the quotes

    quotes = []
    for quote in page.locator("div.quote").all():
        quotes.append({
            "text": quote.locator("span.text").inner_text(),
            "author": quote.locator("small.author").inner_text(),
            "tags": quote.locator("a.tag").all_inner_texts(),
        })
    browser.close()

print(len(quotes), quotes[0])

This prints 10 quotes, each with text, author and tags. Browsers are much slower and heavier than plain requests, so use them only for pages that need them. Before reaching for a browser, check the Network tab in developer tools: many JavaScript sites load their data from a JSON endpoint you can call directly with requests.

Web scraping in JavaScript

The same approach works for web scraping in JS with Node.js. Node 18 and newer have fetch built in, and cheerio, a popular JavaScript web scraping library, offers jQuery-style selectors (npm install cheerio):

JavaScript · Node.js + cheerio
import * as cheerio from "cheerio";

const url = "https://books.toscrape.com/";
const html = await (await fetch(url)).text();
const $ = cheerio.load(html);

const books = $("article.product_pod").map((_, card) => ({
  title: $(card).find("h3 a").attr("title"),
  price: $(card).find(".price_color").text().trim(),
  url: new URL($(card).find("h3 a").attr("href"), url).href,
})).get();

console.log(books.length, books[0]);

For JavaScript-rendered pages in Node, Playwright and Puppeteer play the role Playwright plays in Python.

Scrape responsibly

  • Read robots.txt and the terms of use. They tell you which paths the site does not want crawled and whether automated access is allowed.
  • Slow down. One request per second is a reasonable default for small sites. Never hammer a site in parallel.
  • Identify yourself with a descriptive User-Agent, and stop if the site asks you to.
  • Avoid personal data unless you have a lawful reason. Laws such as GDPR apply to scraped data just as to any other.
  • Prefer an official API when the site offers one with the data you need.

What changes on real websites

ProblemWhat you seeTypical fix
Blocking403 errors, empty pages, CAPTCHAsSlower requests, realistic headers, rotating proxies
JavaScript renderingSelectors find nothingFind the JSON endpoint, or use Playwright
Layout changesFields suddenly emptyMonitor output and update selectors
Pagination limitsResults stop at a fixed numberSplit queries by category, area or date
ScaleHours of runtime, memory issuesQueues, retries, concurrency limits, incremental saves

None of these is hard alone, but web scraping automation at scale turns a 40-line script into a system you have to maintain. Large platforms change their pages and anti-bot measures often, which is where most of the ongoing cost of a home-made scraper goes.

Python scraper, scraping API or ready-made scraper?

Your own Python scraperUnblocking APIReady-made scraper
Best forSmall sites, learning, one-off jobsMany different sites, your own parsingBig platforms: Google Maps, search, social media, reviews
You handleEverythingParsingNothing; you get rows
CostFree plus your time and proxiesPer requestPer result row
Breaks when the site changesYes, you fix itParsing breaks, you fix itThe provider fixes it

For example, collecting 10,000 businesses from Google Maps with your own script means a browser, proxies and constant selector fixes; the Google Maps Scraper returns them for $2.00. To turn a list of website pages into clean text, Website to Markdown crawls them for you. Our comparison of web scraping APIs covers the unblocking kind.

You can also call any Extracto scraper from Python. This uses the same requests library:

Python · Extracto API
import requests

response = requests.post(
    "https://extracto.cloud/api/v1/parsers/google-maps-leads-actor?wait=1",
    headers={"X-API-KEY": "ext_your_api_key", "Idempotency-Key": "dentists-austin-1"},
    json={"input": {"searchStrings": ["Dentist Austin TX"], "maxPlaces": 50}},
    timeout=600,
)
payload = response.json()
print(payload["count"], "places")
for place in payload["data"][:3]:
    print(place.get("title"), place.get("phone"), place.get("website"))

Synchronous calls return the rows in data. For longer runs use the queue endpoint and webhooks described in the API docs.