Verify web links + SEO

Goal: keep a site honest. Crawl every page in the sitemap, follow internal links, catch broken URLs / dead anchors / stale redirects, and diff Open Graph and meta tags against fixtures. The site-health counterpart to test-fixture — same .crust.ts shape, same predicate-aware diff. Built on Bun’s native HTMLRewriter, no extra dependencies.

Run it

# Direct sitemap URL
verify-web-links https://example.com/sitemap.xml

# Auto-discover: probes /robots.txt for `Sitemap:` lines, falls back to /sitemap.xml
verify-web-links https://example.com

# Full SEO sweep: crawl + diff OG / meta against fixtures
verify-web-links https://example.com --fixtures site/*.meta.crust.ts

# Local sitemap (e.g. from `astro build` output). The URLs inside it are
# absolute, so no base URL is needed — and passing one is an error: the
# sitemap and base-url forms are mutually exclusive.
verify-web-links ./dist/sitemap-0.xml

# Sitemap-only, no recursion into internal links
verify-web-links ./public/sitemap.xml --no-recurse

// Programmatic — exit code drives CI
import { runCli } from "crust/verifyWebLinks/cli";

const code = await runCli([
  "--base-url", "https://example.com",
  "--fixtures", "site/*.meta.crust.ts",
  "--json",
]);
process.exit(code);

What gets checked

All four behaviors are on by default. Each has a --no-* opt-out.

External (off-origin) links are skipped by default. Pass --include-external to status-check them (never recursed).

Some URLs redirect because the site is working — WooCommerce bounces an empty-cart /checkout/ to /cart/, /wp-admin/ bounces to the login page. Exclude those subtrees instead of silencing all redirect warnings:

crust -c "verify-web-links https://shop.example.com \
  --exclude /checkout/ --exclude /wp-admin/"

OG images get an extra check: every <meta property="og:image" content="..."> URL is fetched and the response content-type must start with image/.

What counts as a page

A fetched document is crawled for links when crust can read it as HTML. Either the server says so — text/html or application/xhtml+xml, HTML’s own XML-serialised media type — or the declaration says otherwise and the document itself begins <!doctype html> / <html. That second case is printed rather than passed over quietly:

  note: 1 page(s) had to be recognised from their body or an xhtml content-type — the server
        serves HTML without saying text/html

The sniff is deliberately strict about position: the marker has to be the first thing in the body (after a BOM and whitespace), so a text/plain file that merely quotes an HTML snippet in its middle does not become a page whose every string is a URL to check. It costs nothing, either — the asset path already drains the body to avoid leaking the stream.

A page-shaped document that is neither (a JSON API response that ended up in a sitemap) is not crawled: its <a href> strings are data, not links. It is counted in totals.unparsedPages, which makes totals.complete false. Before that sniff existed, a sitemap of application/xhtml+xml pages reported 0 page(s), 1 asset(s), 0 failure(s) and exited 0 over a dead link two hops away — the header ended the crawl.

totals.complete is the machine-readable version of “nothing was left unchecked”: false when --max-pages cut the crawl short, when a #fragment link pointed at a page crust never parsed, or when unparsedPages is nonzero. Pass --strict to turn that same fact into exit 1. Prefer it over reading the JSON in CI: a builtin runs on its own line, so verify-web-links --json | assert (r => r.totals.complete) is refused outright, and a shell that has to remember each field will eventually forget one. --strict is current on main; it is not in the published release this page’s grammar gate lints against, which is why it appears here in prose and not in a runnable block.

Flags

Meta fixtures

Meta fixtures use the same .crust.ts default-export shape as test-fixture. Values can be literals (exact-equality) or single-arg functions (predicates) for anything you want to assert loosely:

// site/about.meta.crust.ts
export default {
  url: "https://example.com/about",
  meta: {
    title: "About Us — Example",
    description: (d: string) => d.length > 50 && d.length < 160,
    "og:title": "About Us",
    "og:type": "website",
    "og:image": (u: string) => u.startsWith("https://") && /\.(png|jpg|jpeg)$/.test(u),
    "twitter:card": "summary_large_image",
  },
};

Or export an array to cover several URLs from one file:

const base = "https://example.com";
export default [
  { url: `${base}/`,       meta: { title: "Example", "og:type": "website" } },
  { url: `${base}/about`,  meta: { title: "About Us — Example", "og:type": "website" } },
  { url: `${base}/blog`,   meta: { title: (t: string) => t.endsWith(" — Example") } },
];

Keys are the literal property/name attribute — title, description, og:title, og:image, twitter:card, article:published_time, etc.

Pairing with CI

verify-web-links exits non-zero on any failure, so it drops straight into a pipeline. Add --strict and it also fails when the run finished having left discovered links unchecked — a bounded crawl can no longer pass as a clean one:

# After a build, run against the deployed preview URL
verify-web-links "$PREVIEW_URL" --fixtures site/**/*.meta.crust.ts

Or combine the builtins into a full pre-deploy gate. The last line assumes the built site is already running: build the site and serve its output — astro preview, a static file server, or your pipeline’s preview step — on the port $PREVIEW_URL points at, the same premise as the example above. The gate never starts that site itself; if nothing is listening on $PREVIEW_URL’s port, the line fails on sitemap discovery rather than reporting the pages it was asked to check. Each builtin line is its own crust -c because a builtin runs in-process: mock-server … & would hand sh a name it does not have, and inside crust the & is refused outright.

crust -c 'mock-server ./openapi.yaml -p4000' &
crust -c 'wait :4000/pets --timeout 20s'
crust -c 'test-fixture fixtures/*.crust.ts'
crust -c 'verify-web-links "$PREVIEW_URL" --fixtures site/*.meta.crust.ts'

Exit codes

Related

• API smoke tests — test-fixture for the API surface; verify-web-links for the public site.
• Mock server — pair with verify-web-links http://localhost:PORT to smoke-test rendered pages against mocked backends.