crust-verify-web-links

crust-verify-web-links turns “the site is probably fine” into an exit code. It starts from a sitemap and checks the graph underneath it: links, #fragment targets, redirect chains, OG/twitter images, and — against fixtures — the meta tags of specific pages.

Check	Fails when
internal + external links	any status that is not 2xx
#fragment targets	the id/name is not on the fetched page
redirect chains	a link 301s/302s somewhere (informational with --no-redirect-warnings)
OG / twitter images	the image URL is unreachable
meta vs --fixtures	a keyed value differs, or a predicate returns falsy

Exit codes: 0 all clear, 1 verification failures, 2 bad arguments or an unreachable sitemap. With --strict, 1 also covers a run that finished having left discovered links unchecked. That 2 also covers --site-map-url together with --base-url, neither of them given, no fixture files matched <glob>, --concurrency must be >= 1, and unexpected argument — so a typo never looks like a clean crawl.

Getting it pointed at the right thing

verify-web-links --base-url https://example.com
verify-web-links --site-map-url https://example.com/sitemap.xml
verify-web-links --site-map-url ./dist/sitemap.xml --no-recurse
verify-web-links --base-url https://staging.example.com --fixtures site/*.meta.crust.ts

--base-url probes robots.txt for Sitemap: lines, then /sitemap.xml, then /sitemap-index.xml, and when none answer it names all three rather than saying “no sitemap”. --site-map-url (aliased --sitemap) also takes a local path, which is the CI move: check the build output before publishing it, so a broken internal link never reaches a deployment. With --no-recurse only the URLs in the sitemap are fetched.

Green does not mean checked

The failure mode worth memorising: three ways to exit 0 having verified almost nothing. Read the note: lines and totals before believing a green run.

Symptom	What it means
N page(s) had to be recognised from their body	the server sends HTML without saying text/html. crust followed the links anyway — the mislabel is the site’s bug, and the note is there so a green run does not hide it
N page(s) NOT read as HTML	a sitemap entry or <a> target that is neither declared HTML nor smells like it (a JSON endpoint). Its links were not followed: the crawl ended there
a note naming a count (anchors skipped…, stopped at --max-pages…)	that work did not happen. --no-recurse and --no-anchors produce it deliberately; a cap produces it by accident
"parsed": false on a page in --json	no links and no ids were extracted from that page, so nothing was provable from it

--max-pages N is a safety valve, not a fast mode: the crawl stops early, the unchecked count is reported (note: --max-pages reached — 10 discovered URL(s) NOT checked, totals.dropped in JSON) — and the exit code still says 0. Treat a capped run as inconclusive.

totals.complete is the one field that means “nothing was left unchecked”: false on any of the above. It is what a gate should read, instead of a list of fields someone has to keep updated by hand.

The same honesty applies to anchors. With --no-recurse, a fragment link to a page outside the sitemap is not a failure, because crust never read that page: it is counted into totals.anchorsSkipped and printed as

verify-web-links: 4 page(s), 1 asset(s), 0 failure(s)
  note: 3 #fragment link(s) NOT checked — crust never read the destination's HTML (--no-recurse / --exclude / --max-depth, or an off-origin link without --include-external)

“Could not check, and here is why” is a different fact from “checked and clean”, and the report keeps them apart on purpose.

In CI

--strict is the CI flag. It exits 1 when the run finished having left discovered links unchecked — a --max-pages cut-off, an unchecked #fragment, a page crust could not read — so a bounded crawl cannot pass as a clean one: verify-web-links --base-url https://example.com --strict --max-pages 500.

A builtin runs in-process and cannot sit mid-pipeline: verify-web-links … | assert (…) is refused with “give it a line of its own”. Redirect the whole invocation, then judge the report:

crust -c 'verify-web-links --base-url https://example.com --json --no-progress' > report.json

read report.json | (s => JSON.parse(s)) | assert (r => r.totals.failures === 0)
read report.json | (s => JSON.parse(s)) | assert (r => r.totals.complete)

read yields each file’s contents as one item, so JSON.parse receives the whole document (a lines source would hand it one line at a time). The first line fails on a real failure; the second catches every inconclusive case at once — complete is false if anything was dropped, if any #fragment went unchecked, or if any page could not be read.

Meta fixtures

export default {
  url: "https://example.com/about",
  meta: {
    title: "About Us",
    description: (d: string) => d.length > 50 && d.length < 160,
    "og:image": (u: string) => u.endsWith(".png") || u.endsWith(".jpg"),
    "twitter:card": "summary_large_image",
  },
};

A value is compared for equality; a function is a predicate over the extracted value. A key a page does not have is asked about, not ignored — so asserting og:image on a page that lost its image is a failure, which is the entire point.

Fixture urls match URL-normalized: scheme+host case and default ports don’t matter, https://x.com and https://x.com/ are the same page, but a trailing slash on a non-root path stays significant (/login/ ≠ /login). A fixture whose url never appears in the crawl fails as meta-fixture-no-page — a fixture for a page you deleted is a failure, not a skip.

Tuning

verify-web-links --base-url https://example.com --exclude /checkout/ --exclude /go/
verify-web-links --base-url https://example.com --include-external --max-depth 2
verify-web-links --base-url https://example.com --concurrency 16 --timeout 3000

• --exclude <substring> is repeatable. Use it for subtrees that redirect by design (short-link senders, locale switches); otherwise a redirect chain is a failure. --no-redirect-warnings silences them globally — only when you mean it.
• Off-origin links are not queued at all without --include-external, and even with it they are never recursed into.
• --max-depth (default 5) bounds the crawl; --concurrency (default 4) and --timeout (default 10000 ms) bound each fetch.

Traps

• 0 page(s) is a bug in your target, not a clean site. Check the Content-Type before the link graph.
• A note is a finding. Any note: line naming a skipped or uncrawled count means the run’s green is narrower than it looks.
• --no-recurse shrinks the anchor check, loudly. It cannot check fragments on pages it never fetched; the count is in the report.
• A builtin has no pipes. No | assert, no > on the builtin itself — crust refuses both and prints the redirect that works. See crust-builtins.

Related

• Verify web links — the recipe: a live site, a built dist/, and CI wiring
• crust-cli — exit codes, and what --check can say about a line like these
• crust-builtins — why a builtin refuses a pipe