Verify web links + SEO

Goal: keep a site honest. Crawl every page in the sitemap, follow internal links, catch broken URLs / dead anchors / stale redirects, and diff Open Graph and meta tags against fixtures. The site-health counterpart to test-fixture — same .crust.ts shape, same predicate-aware diff. Built on Bun’s native HTMLRewriter, no extra dependencies.

Run it

# Direct sitemap URL
verify-web-links https://example.com/sitemap.xml

# Auto-discover: probes /robots.txt for `Sitemap:` lines, falls back to /sitemap.xml
verify-web-links https://example.com

# Full SEO sweep: crawl + diff OG / meta against fixtures
verify-web-links https://example.com --fixtures site/*.meta.crust.ts

# Local sitemap (e.g. from `astro build` output). The URLs inside it are
# absolute, so no base URL is needed — and passing one is an error: the
# sitemap and base-url forms are mutually exclusive.
verify-web-links ./dist/sitemap-0.xml

# Sitemap-only, no recursion into internal links
verify-web-links ./public/sitemap.xml --no-recurse

// Programmatic — exit code drives CI
import { runCli } from "crust/verifyWebLinks";

const code = await runCli([
  "--base-url", "https://example.com",
  "--fixtures", "site/*.meta.crust.ts",
  "--json",
]);
process.exit(code);

What gets checked

All four behaviors are on by default. Each has a --no-* opt-out.

External (off-origin) links are skipped by default. Pass --include-external to status-check them (never recursed).

Some URLs redirect because the site is working — WooCommerce bounces an empty-cart /checkout/ to /cart/, /wp-admin/ bounces to the login page. Exclude those subtrees instead of silencing all redirect warnings:

crust -c "verify-web-links https://shop.example.com \
  --exclude /checkout/ --exclude /wp-admin/"

OG images get an extra check: every <meta property="og:image" content="..."> URL is fetched and the response content-type must start with image/.

Flags

Meta fixtures

Meta fixtures use the same .crust.ts default-export shape as test-fixture. Values can be literals (exact-equality) or single-arg functions (predicates) for anything you want to assert loosely:

// site/about.meta.crust.ts
export default {
  url: "https://example.com/about",
  meta: {
    title: "About Us — Example",
    description: (d: string) => d.length > 50 && d.length < 160,
    "og:title": "About Us",
    "og:type": "website",
    "og:image": (u: string) => u.startsWith("https://") && /\.(png|jpg|jpeg)$/.test(u),
    "twitter:card": "summary_large_image",
  },
};

Or export an array to cover several URLs from one file:

const base = "https://example.com";
export default [
  { url: `${base}/`,       meta: { title: "Example", "og:type": "website" } },
  { url: `${base}/about`,  meta: { title: "About Us — Example", "og:type": "website" } },
  { url: `${base}/blog`,   meta: { title: (t: string) => t.endsWith(" — Example") } },
];

Keys are the literal property/name attribute — title, description, og:title, og:image, twitter:card, article:published_time, etc.

Pairing with CI

verify-web-links exits non-zero on any failure, so it drops straight into a pipeline:

# After a build, run against the deployed preview URL
verify-web-links "$PREVIEW_URL" --fixtures site/**/*.meta.crust.ts

Or chain with other crust builtins for a full pre-deploy gate:

mock-server ./openapi.yaml -p4000 &
test-fixture fixtures/*.crust.ts
verify-web-links http://localhost:3000 --fixtures site/*.meta.crust.ts

Exit codes

Related

• API smoke tests — test-fixture for the API surface; verify-web-links for the public site.
• Mock server — pair with verify-web-links http://localhost:PORT to smoke-test rendered pages against mocked backends.