What you need
Python 3.9 or newer and two libraries: requests downloads pages and beautifulsoup4 parses HTML. Install them in a virtual environment:
python -m venv venv
source venv/bin/activate # on Windows: venv\Scripts\activate
pip install requests beautifulsoup4We will scrape books.toscrape.com, a sandbox store built for practising scraping: 1,000 books across 50 pages, each with a title, price, star rating and stock status.
Step 1: Inspect the page
Open the page in your browser, right-click a book and choose Inspect. Each book is an <article class="product_pod">. Inside it, the title is in the title attribute of the h3 a link, the price is in .price_color, the rating is a class such as star-rating Three, and the next page link is li.next a. These CSS selectors are all the scraper needs.
Step 2: Web scraping with Python and BeautifulSoup
The script below downloads each page, extracts every book, follows the next-page link until there is none, waits one second between requests and saves everything to books.csv:
import csv
import time
from urllib.parse import urljoin
import requests
from bs4 import BeautifulSoup
START_URL = "https://books.toscrape.com/catalogue/page-1.html"
HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; my-scraper/1.0)"}
RATINGS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}
def scrape_page(url):
response = requests.get(url, headers=HEADERS, timeout=20)
response.raise_for_status()
soup = BeautifulSoup(response.content, "html.parser") # bytes: lets bs4 read the page charset
books = []
for card in soup.select("article.product_pod"):
link = card.select_one("h3 a")
books.append({
"title": link["title"],
"price": float(card.select_one(".price_color").get_text(strip=True).lstrip("£")),
"rating": RATINGS.get(card.select_one("p.star-rating")["class"][1]),
"in_stock": "In stock" in card.select_one(".availability").get_text(),
"url": urljoin(url, link["href"]),
})
next_link = soup.select_one("li.next a")
next_url = urljoin(url, next_link["href"]) if next_link else None
return books, next_url
all_books, url = [], START_URL
while url:
books, url = scrape_page(url)
all_books.extend(books)
print(f"{len(all_books)} books so far")
time.sleep(1) # be polite: one request per second
with open("books.csv", "w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=all_books[0].keys())
writer.writeheader()
writer.writerows(all_books)
Running it prints a running count and finishes with 1,000 rows in about a minute. Three details matter on real sites:
response.content, notresponse.text. This site does not declare its character set in the HTTP headers, sorequestsguesses wrong and the pound sign comes out as£. Passing bytes lets BeautifulSoup read the charset from the page itself.urljoinfor links. Links on the page are relative; joining them with the current URL gives absolute links that work.raise_for_status()and a timeout. The script stops on errors instead of silently saving empty data, and never hangs forever.
Step 3: Scrape JavaScript-rendered pages
Many modern sites send an almost empty HTML page and build the content with JavaScript. The JavaScript version of the practice site, quotes.toscrape.com/js, shows ten quotes in a browser, but requests plus BeautifulSoup finds zero div.quote elements in the downloaded HTML. You need a real browser. Install Playwright with pip install playwright and playwright install chromium, then:
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.goto("https://quotes.toscrape.com/js/")
page.wait_for_selector("div.quote") # wait until JavaScript has rendered the quotes
quotes = []
for quote in page.locator("div.quote").all():
quotes.append({
"text": quote.locator("span.text").inner_text(),
"author": quote.locator("small.author").inner_text(),
"tags": quote.locator("a.tag").all_inner_texts(),
})
browser.close()
print(len(quotes), quotes[0])
This prints 10 quotes, each with text, author and tags. Browsers are much slower and heavier than plain requests, so use them only for pages that need them. Before reaching for a browser, check the Network tab in developer tools: many JavaScript sites load their data from a JSON endpoint you can call directly with requests.
Web scraping in JavaScript
The same approach works for web scraping in JS with Node.js. Node 18 and newer have fetch built in, and cheerio, a popular JavaScript web scraping library, offers jQuery-style selectors (npm install cheerio):
import * as cheerio from "cheerio";
const url = "https://books.toscrape.com/";
const html = await (await fetch(url)).text();
const $ = cheerio.load(html);
const books = $("article.product_pod").map((_, card) => ({
title: $(card).find("h3 a").attr("title"),
price: $(card).find(".price_color").text().trim(),
url: new URL($(card).find("h3 a").attr("href"), url).href,
})).get();
console.log(books.length, books[0]);
For JavaScript-rendered pages in Node, Playwright and Puppeteer play the role Playwright plays in Python.
Scrape responsibly
- Read robots.txt and the terms of use. They tell you which paths the site does not want crawled and whether automated access is allowed.
- Slow down. One request per second is a reasonable default for small sites. Never hammer a site in parallel.
- Identify yourself with a descriptive User-Agent, and stop if the site asks you to.
- Avoid personal data unless you have a lawful reason. Laws such as GDPR apply to scraped data just as to any other.
- Prefer an official API when the site offers one with the data you need.
What changes on real websites
| Problem | What you see | Typical fix |
|---|---|---|
| Blocking | 403 errors, empty pages, CAPTCHAs | Slower requests, realistic headers, rotating proxies |
| JavaScript rendering | Selectors find nothing | Find the JSON endpoint, or use Playwright |
| Layout changes | Fields suddenly empty | Monitor output and update selectors |
| Pagination limits | Results stop at a fixed number | Split queries by category, area or date |
| Scale | Hours of runtime, memory issues | Queues, retries, concurrency limits, incremental saves |
None of these is hard alone, but web scraping automation at scale turns a 40-line script into a system you have to maintain. Large platforms change their pages and anti-bot measures often, which is where most of the ongoing cost of a home-made scraper goes.
Python scraper, scraping API or ready-made scraper?
| Your own Python scraper | Unblocking API | Ready-made scraper | |
|---|---|---|---|
| Best for | Small sites, learning, one-off jobs | Many different sites, your own parsing | Big platforms: Google Maps, search, social media, reviews |
| You handle | Everything | Parsing | Nothing; you get rows |
| Cost | Free plus your time and proxies | Per request | Per result row |
| Breaks when the site changes | Yes, you fix it | Parsing breaks, you fix it | The provider fixes it |
For example, collecting 10,000 businesses from Google Maps with your own script means a browser, proxies and constant selector fixes; the Google Maps Scraper returns them for $2.00. To turn a list of website pages into clean text, Website to Markdown crawls them for you. Our comparison of web scraping APIs covers the unblocking kind.
You can also call any Extracto scraper from Python. This uses the same requests library:
import requests
response = requests.post(
"https://extracto.cloud/api/v1/parsers/google-maps-leads-actor?wait=1",
headers={"X-API-KEY": "ext_your_api_key", "Idempotency-Key": "dentists-austin-1"},
json={"input": {"searchStrings": ["Dentist Austin TX"], "maxPlaces": 50}},
timeout=600,
)
payload = response.json()
print(payload["count"], "places")
for place in payload["data"][:3]:
print(place.get("title"), place.get("phone"), place.get("website"))Synchronous calls return the rows in data. For longer runs use the queue endpoint and webhooks described in the API docs.