Why convert websites to Markdown for LLMs

A typical web page is mostly code: navigation, scripts, tracking, cookie banners and layout markup. Feeding that to a model wastes context and adds noise. Markdown keeps what matters for meaning: # headings show the structure, lists and tables stay intact, and links keep their targets. That is why website to Markdown for LLM pipelines is the first step of most RAG systems, AI agents that read the web, and documentation chatbots.

Website to Markdown in Python

For a single page, fetch the HTML, remove the boilerplate and convert the main content with markdownify:

Python · one page to Markdown
import requests
from bs4 import BeautifulSoup
from markdownify import markdownify

url = "https://go.dev/doc/"
html = requests.get(url, headers={"User-Agent": "Mozilla/5.0 (compatible; my-scraper/1.0)"}, timeout=20).text
soup = BeautifulSoup(html, "html.parser")
for tag in soup(["script", "style", "nav", "header", "footer", "aside"]):
    tag.decompose()  # drop menus and boilerplate before converting
main = soup.find("main") or soup.body
markdown = markdownify(str(main), heading_style="ATX")
print(markdown.strip()[:600])

Run on November 20, 2026, it printed the Go documentation page starting with # Documentation and its first paragraphs, the same text the extractor returns. It needs pip install requests beautifulsoup4 markdownify. Pages that build their content with JavaScript return little or nothing this way; they need a headless browser.

Crawl a whole website to Markdown

To crawl a website to Markdown, not just one page, you need a crawler that follows links, stays in scope and skips duplicates. The Website Markdown Extractor does that:

SettingWhat it does
Start URLsUp to 50 pages to start from
ScopeOnly pages under the start URL path (best for docs), anywhere on the same host, or on any subdomain
Max pages and depthStop after 1–1,000 pages and limit how many clicks from a start URL to follow
SitemapAlso add in-scope pages listed in the site's sitemap, including pages nothing links to
Include and exclude patternsCrawl only URLs matching patterns, or skip ones like **/changelog/**
Remove selectorsExtra CSS selectors to drop; navigation, headers, footers, sidebars and cookie banners are always removed
LinksOptionally list the links found on each page

Each page comes back with its URL, title, description, language, the Markdown, a word count, the HTTP status, the last-modified date when the site provides one and a content hash for spotting changes. 1,000 pages cost $0.19.

Website markdown extractor output for go.dev/doc showing the page converted to Markdown with headings and paragraphs
The Go documentation page converted by the Website Markdown Extractor, from the saved sample of November 20, 2026. The beginning of the Markdown is shown.
Website markdown extractor results for 11 pages with title, URL, language, word count and markdown
One row per page: title, URL, language, word count and the Markdown itself, here for documentation, encyclopedia, government and blog pages.

Website to Markdown converters compared

ToolHow it worksFree use
Python (markdownify, html2text)Code you run; one page at a time unless you write a crawlerFree; you handle blocking and JavaScript pages
Jina ReaderPrefix r.jina.ai/ to a URL to get LLM-friendly text20 requests a minute without a key; 10M free tokens with a new key; paid by tokens
FirecrawlAPI that scrapes and crawls pages to Markdown or JSON1,000 credits a month free; plans up to $749 for 1M credits
Extracto Markdown ExtractorCrawler with scope, sitemap and pattern controls$0.19 / 1,000 results, 2,500 trial credits
Browser extensionsWebsite to Markdown Chrome extension: converts the page you have openFree, one page at a time

A website to Markdown online converter or Chrome extension is the quickest way to grab one article. For a documentation site or a knowledge base, use a crawler so pages stay consistent and you can re-run it when the site changes.

For a detailed price and feature comparison with Firecrawl, see Extracto vs Firecrawl.

HTML to Markdown API

Run the extractor through the Extracto API to scrape website to Markdown as part of a pipeline. This request crawls a documentation section through its sitemap:

cURL · website-markdown-extractor
curl --request POST 'https://extracto.cloud/api/v1/parsers/website-markdown-extractor?wait=1' \
  --header 'X-API-KEY: ext_your_api_key' \
  --header 'Idempotency-Key: docs-markdown-2026-11-20' \
  --header 'Content-Type: application/json' \
  --data '{"input": {"startUrls": ["https://docs.example.com/guide"], "maxPages": 200, "crawlScope": "path", "useSitemap": true}}'

Check the scraper page for the exact input format. Use the contentHash field to re-embed only pages that changed since the last run.

Tips for LLM and RAG use

  • Split on headings, not fixed lengths, so each chunk keeps its context.
  • Keep the URL and title with every chunk for citations.
  • Remove repeated blocks (cookie text, related posts) with remove selectors before embedding.
  • Check the word count: very short pages are often error or login pages.
  • Respect robots.txt and the site's terms; for your own docs, publish an llms.txt to point models at the right pages.
This is general information, not legal advice. See Extracto's acceptable use policy for what the platform may not be used for.