Website Conversionfull website to markdown

How to Convert an Entire Website to Markdown

Learn how to collect website URLs, extract page content in bulk, and export multiple webpages as organized Markdown files.

WebToMarkdown Team· Batch Systems & Crawling
10 min read
#Batch Conversion#Whole Site#Sitemap#Documentation#Crawling

Single Page vs. Entire Website Conversion

Converting a single URL is straightforward. But when you need to ingest an entire API reference, a product knowledge base, or hundreds of technical tutorials, converting pages one by one is impractical.

Full website conversion requires a structured workflow:

  1. Discovering all relevant canonical URLs across the domain.
  2. Filtering out noisy duplicates, pagination pages, and archive tags.
  3. Extracting content in parallel while respecting rate limits.
  4. Packaging individual Markdown files into an organized folder structure with an index.md table of contents.

Step 1: Discover and Collect Website URLs

The most reliable way to find all pages on a website is using its XML sitemap.

Locating the Sitemap

Most sites publish sitemaps at standard locations:

  • https://example.com/sitemap.xml
  • https://example.com/sitemap_index.xml
  • Look inside https://example.com/robots.txt for a Sitemap: directive.

You can inspect the sitemap in your browser or extract all URLs using a simple command-line query:

bash
curl -s https://example.com/sitemap.xml | grep -oE 'https://[^<]+' | sort -u > urls.txt

Step 2: Filter and Refine the Extraction Scope

Before converting, clean the URL list to avoid burning quotas on unnecessary pages:

  • Exclude Non-Content Pages: Filter out /tag/, /author/, /search, /privacy, and /terms.
  • Target Subdirectories: If you only need documentation, keep only URLs starting with /docs/ or /api/.
  • Remove Query Parameters: Strip tracking tokens like ?utm_source= or ?ref= to prevent duplicate downloads.

Step 3: Execute Batch Markdown Conversion

Using the WebToMarkdown Batch Tool:

  1. Paste your clean list of URLs into the batch input area.
  2. Select your extraction profile:

- Standard Profile: Preserves standard web documentation layouts.

- LLM Profile: Optimized for maximum token density and clean headers.

  1. Configure rendering mode:

- Auto: Uses fast server-side fetching for static pages and headless browser rendering for dynamic pages.

  1. Click Convert Batch. The backend processes pages concurrently and tracks real-time progress.

Step 4: Organize Folder Structure and Table of Contents

A collection of 200 loose Markdown files in a single flat directory is difficult to navigate. A well-structured export mirrors the website's URL hierarchy:

text
docs-export/
├── index.md                  # Master Table of Contents
├── getting-started/
│   ├── installation.md
│   └── configuration.md
├── api-reference/
│   ├── authentication.md
│   └── endpoints.md
└── tutorials/
    └── building-an-app.md

The generated index.md contains direct Markdown links to each sub-page, allowing RAG vector databases and note apps like Obsidian to resolve internal links effortlessly.


Step 5: Export Markdown Files as a Bundled ZIP

Once the batch job finishes:

  • Review the summary: total pages converted, success count, and failed URLs.
  • Inspect any failed pages to retry with browser rendering if needed.
  • Click Download Archive (.zip) to save the entire package locally.

Overcoming Challenges in Full Website Extraction

  • Rate Limits & IP Blocking: Crawling too aggressively triggers HTTP 429 (Too Many Requests) or Cloudflare blocks. WebToMarkdown employs built-in request throttling and polite concurrency to protect availability.
  • Repeated Global Elements: When converting 100 pages from the same site, each page may repeat the company tagline or header. Our extraction algorithm detects and strips repeated domain-wide boilerplate across batches.
Product Tool

Batch Convert Entire Websites Online

Submit dozens or hundreds of URLs at once, run parallel extraction, and download a bundled ZIP archive with clean Markdown.

Frequently Asked Questions

How many URLs can I convert at once?

Free users can batch up to 5 URLs at a time with a small whole-site trial. Pro plans allow up to 100 URLs per batch, and Plus plans handle up to 200 URLs with 5 concurrent batches.

Will the exported Markdown files include internal links?

Yes. Internal links are preserved as clean relative or canonical Markdown links, making the folder ideal for Obsidian vaults or documentation generators.

What happens if a few pages in a large batch fail?

The batch process logs each failed URL with its specific error message. You can view the failed list and retry just those specific pages with one click.

Can I automate whole-site extraction with an API?

Yes. The WebToMarkdown Batch API (`POST /api/v1/convert/batch`) lets you submit lists of URLs programmatically and receive a download URL for the generated ZIP file.

W

WebToMarkdown Team

Batch Systems & Crawling at WebToMarkdown

Dedicated to helping developers, AI researchers, and knowledge workers extract, clean, and convert unstructured web content into portable Markdown.

Related Articles & Tutorials

All articles
AI & LLM10 min read
Sending documents as Markdown is 40%–80% more token-efficient and cost-effective than sending PDFs or Word files to LLMs. Discover how to eliminate the PDF vision tax, format tables and headings, and cut token costs.
Developer Guides8 min read
Integrate automated web-to-markdown capabilities directly into your AI applications, RAG pipelines, and automated bots using the WebToMarkdown REST API.
Markdown Guides9 min read
Have a folder of Markdown files converted from legacy websites? Learn how to turn them into a lightning-fast, searchable documentation website using modern SSGs.