Testing playbook · intermediate

Archive to Live Surface Delta

Reconcile historical URLs, parameters, JavaScript, and screenshots with the current in-scope surface while preserving source time, present behavior, and a control for every promoted lead.

waybackcommon-crawlurlscanhistorical-urlssurface-diff
Stable IDplaybook:archive-to-live-surface-deltaSource record dateHuman reviewNot recordedProvenanceCurated workflow
Detection firstStop as soon as the evidence decisively supports or falsifies the hypothesis.

Every step requires authorization. Review side effects before running commands against a live system.

Define the boundary before the test.

Preconditions
Current scope, observed technology, reachable feature, controlled identities or data, and a recovery plan.
Expected evidence
One observable response, state, or callback that distinguishes the hypothesis from normal behavior.
Negative control
The same action with the security-sensitive variable removed, replaced, or bound to its rightful owner.
False-positive checks
Cache, retry, race, reflection, proxy rewriting, stale sessions, and unrelated asynchronous activity.
Chain gate
Record the proven primitive and the next missing prerequisite. Do not expand impact until scope and authorization still permit the next step.

Choose the evidence path that fits the boundary.

No lane is automatically stronger. Pick the least intrusive path that can distinguish the hypothesis, then confirm through an independent method before reporting.

01

Passive source merge

The asset graph is incomplete or stale.

Combine certificate transparency, passive DNS or approved APIs, archives, public repositories, and first-party documentation into an append-only source ledger.

Tradeoff
Low target traffic, but historical names, shared infrastructure, and vendor data are candidates rather than current scope.
Advance when
Every candidate retains source and time, then survives live brief, ownership, DNS, and wildcard checks.
02

Low-rate active confirmation

The live program permits resolution or probing of already approved candidates.

Resolve explicit DNS record types, then probe the current DNS view at a bounded rate for TLS and HTTP state with known wildcard controls.

Tradeoff
Establishes current behavior, but sends traffic and can misclassify shared edges or transient failures.
Advance when
A second resolver or repeat observation supports the state and the asset remains in scope.
03

Source assisted

Public docs, client code, contracts, archives, or authorized local source can narrow the test.

Extract operation names, routes, fields, versions, and trust decisions without executing untrusted artifacts, then test only the supported live hypothesis.

Tradeoff
Low traffic and high context, but sources can be stale, client-only, generic, or unrelated to the deployed product area.
Advance when
A current in-scope response or operation corroborates the source claim.
04

Independent replay

A candidate survives its first control and is nearing report status.

Repeat the smallest claim in a fresh session, browser profile, account pair, token set, or second tool while keeping the security-sensitive variable constant.

Tradeoff
Costs another clean setup, but catches cache, session, extension, proxy, and operator confounders.
Advance when
Fresh evidence supports the same primitive without relying on the first tool's interpretation.
01

Lock scope and archive the query manifest

Record approved roots, exclusions, query time, source URL, source index, filters, and rate budget before collection. Historical presence is evidence that a URL existed in an archive, not that it is live, owned, vulnerable, or currently authorized.

mkdir -p recon/archive/{raw,normalized,current,state}; printf 'source	query	retrieved_at	file
' > recon/archive/state/query-manifest.tsv
Evidence checkpointSave the request or command, raw result, timestamp, target, and a negative control before continuing.
02

Pull bounded Wayback CDX records

Request only the fields needed for reconciliation, cache the response, and keep timestamps and digests. Use collapse carefully because it can hide changes that matter to a delta review.

curl -fsS --get 'https://web.archive.org/cdx/search/cdx' --data-urlencode 'url=*.target.com/*' --data-urlencode 'output=json' --data-urlencode 'fl=timestamp,original,statuscode,mimetype,digest' --data-urlencode 'filter=statuscode:200' --data-urlencode 'limit=50000' > recon/archive/raw/wayback.json
Evidence checkpointSave the request or command, raw result, timestamp, target, and a negative control before continuing.
03

Query the current Common Crawl index

Resolve the newest index endpoint from the official collection list instead of hard-coding an index name. Store CDXJ exactly as returned so each normalized URL keeps its crawl name, timestamp, status, MIME type, and digest.

cc_index=$(curl -fsS https://index.commoncrawl.org/collinfo.json | jq -r '.[0]["cdx-api"]'); curl -fsS --get "$cc_index" --data-urlencode 'url=*.target.com/*' --data-urlencode 'output=json' --data-urlencode 'filter=status:200' > recon/archive/raw/commoncrawl.cdxj
Evidence checkpointSave the request or command, raw result, timestamp, target, and a negative control before continuing.
04

Use urlscan as a public artifact pivot

Search public scans for historical request graphs, DOM snapshots, technologies, and screenshots. Do not submit a private or sensitive target as a public scan. Record whether an artifact came from a public search result or a scan you were explicitly allowed to create.

curl -fsS --get 'https://urlscan.io/api/v1/search/' --data-urlencode 'q=domain:target.com' --data-urlencode 'size=100' > recon/archive/raw/urlscan-search.json
Evidence checkpointSave the request or command, raw result, timestamp, target, and a negative control before continuing.
05

Normalize without erasing provenance

Lowercase only the hostname, remove fragments, retain scheme, port, path, parameter names, source timestamp, source name, and original value. Group URLs into endpoint families, but keep the raw source row behind every family member.

jq -r '.[1:][]? | .[1]' recon/archive/raw/wayback.json | unfurl format '%s://%d%p?%q' | sort -u > recon/archive/normalized/wayback.urls
Evidence checkpointSave the request or command, raw result, timestamp, target, and a negative control before continuing.
06

Build a target-specific route and parameter queue

Rank endpoint families by independent-source agreement, sensitive operation nouns, parameter novelty, historical JavaScript references, and recency. Prefer a compact target-specific corpus over a broad public list.

cat recon/archive/normalized/*.urls | unfurl paths | sed 's#^/##' | awk 'NF' | sort -u > recon/archive/normalized/route-words.txt; cat recon/archive/normalized/*.urls | unfurl keys | awk 'NF' | sort -u > recon/archive/normalized/parameter-words.txt
Evidence checkpointSave the request or command, raw result, timestamp, target, and a negative control before continuing.
07

Revalidate with a low-rate live comparison

Probe only current in-scope URLs and retain status, redirect, title, content type, body length, and observation time. Use a current known route and a randomized nonexistent route as controls. A changed status alone is not proof that an old operation still exists.

httpx -l recon/archive/normalized/candidate-urls.txt -json -silent -rl 5 -sc -cl -ct -title -location -o recon/archive/current/http.jsonl
Evidence checkpointSave the request or command, raw result, timestamp, target, and a negative control before continuing.
08

Promote only evidence-backed deltas

A promoted lead includes original source, capture time, exact historical artifact, current request and response fingerprint, control result, confidence, and one safe next discriminator. Label archive-only, dead, redirected, current, and indeterminate states separately.

Evidence checkpointSave the request or command, raw result, timestamp, target, and a negative control before continuing.

Common guidance

  • Agreement between Wayback, Common Crawl, and urlscan increases provenance confidence, not vulnerability confidence.
  • Old JavaScript can reveal retired operation names and parameter shapes, but validate the current server behavior before testing a security boundary.
  • Do not replay archived POST bodies, tokens, personal data, or state-changing requests.
  • Store normalized facts and source references. Raw vendor responses should follow a defined retention limit.

Mapped workflow tools

References

Version history: normalized permanent page created 2026-08-20.