Archive to Live Surface Delta
Reconcile historical URLs, parameters, JavaScript, and screenshots with the current in-scope surface while preserving source time, present behavior, and a control for every promoted lead.
Every step requires authorization. Review side effects before running commands against a live system.
Define the boundary before the test.
- Preconditions
- Current scope, observed technology, reachable feature, controlled identities or data, and a recovery plan.
- Expected evidence
- One observable response, state, or callback that distinguishes the hypothesis from normal behavior.
- Negative control
- The same action with the security-sensitive variable removed, replaced, or bound to its rightful owner.
- False-positive checks
- Cache, retry, race, reflection, proxy rewriting, stale sessions, and unrelated asynchronous activity.
- Chain gate
- Record the proven primitive and the next missing prerequisite. Do not expand impact until scope and authorization still permit the next step.
Choose the evidence path that fits the boundary.
No lane is automatically stronger. Pick the least intrusive path that can distinguish the hypothesis, then confirm through an independent method before reporting.
Passive source merge
The asset graph is incomplete or stale.Combine certificate transparency, passive DNS or approved APIs, archives, public repositories, and first-party documentation into an append-only source ledger.
- Tradeoff
- Low target traffic, but historical names, shared infrastructure, and vendor data are candidates rather than current scope.
- Advance when
- Every candidate retains source and time, then survives live brief, ownership, DNS, and wildcard checks.
Low-rate active confirmation
The live program permits resolution or probing of already approved candidates.Resolve explicit DNS record types, then probe the current DNS view at a bounded rate for TLS and HTTP state with known wildcard controls.
- Tradeoff
- Establishes current behavior, but sends traffic and can misclassify shared edges or transient failures.
- Advance when
- A second resolver or repeat observation supports the state and the asset remains in scope.
Source assisted
Public docs, client code, contracts, archives, or authorized local source can narrow the test.Extract operation names, routes, fields, versions, and trust decisions without executing untrusted artifacts, then test only the supported live hypothesis.
- Tradeoff
- Low traffic and high context, but sources can be stale, client-only, generic, or unrelated to the deployed product area.
- Advance when
- A current in-scope response or operation corroborates the source claim.
Independent replay
A candidate survives its first control and is nearing report status.Repeat the smallest claim in a fresh session, browser profile, account pair, token set, or second tool while keeping the security-sensitive variable constant.
- Tradeoff
- Costs another clean setup, but catches cache, session, extension, proxy, and operator confounders.
- Advance when
- Fresh evidence supports the same primitive without relying on the first tool's interpretation.
Lock scope and archive the query manifest
Record approved roots, exclusions, query time, source URL, source index, filters, and rate budget before collection. Historical presence is evidence that a URL existed in an archive, not that it is live, owned, vulnerable, or currently authorized.
mkdir -p recon/archive/{raw,normalized,current,state}; printf 'source query retrieved_at file
' > recon/archive/state/query-manifest.tsvPull bounded Wayback CDX records
Request only the fields needed for reconciliation, cache the response, and keep timestamps and digests. Use collapse carefully because it can hide changes that matter to a delta review.
curl -fsS --get 'https://web.archive.org/cdx/search/cdx' --data-urlencode 'url=*.target.com/*' --data-urlencode 'output=json' --data-urlencode 'fl=timestamp,original,statuscode,mimetype,digest' --data-urlencode 'filter=statuscode:200' --data-urlencode 'limit=50000' > recon/archive/raw/wayback.jsonQuery the current Common Crawl index
Resolve the newest index endpoint from the official collection list instead of hard-coding an index name. Store CDXJ exactly as returned so each normalized URL keeps its crawl name, timestamp, status, MIME type, and digest.
cc_index=$(curl -fsS https://index.commoncrawl.org/collinfo.json | jq -r '.[0]["cdx-api"]'); curl -fsS --get "$cc_index" --data-urlencode 'url=*.target.com/*' --data-urlencode 'output=json' --data-urlencode 'filter=status:200' > recon/archive/raw/commoncrawl.cdxjUse urlscan as a public artifact pivot
Search public scans for historical request graphs, DOM snapshots, technologies, and screenshots. Do not submit a private or sensitive target as a public scan. Record whether an artifact came from a public search result or a scan you were explicitly allowed to create.
curl -fsS --get 'https://urlscan.io/api/v1/search/' --data-urlencode 'q=domain:target.com' --data-urlencode 'size=100' > recon/archive/raw/urlscan-search.jsonNormalize without erasing provenance
Lowercase only the hostname, remove fragments, retain scheme, port, path, parameter names, source timestamp, source name, and original value. Group URLs into endpoint families, but keep the raw source row behind every family member.
jq -r '.[1:][]? | .[1]' recon/archive/raw/wayback.json | unfurl format '%s://%d%p?%q' | sort -u > recon/archive/normalized/wayback.urlsBuild a target-specific route and parameter queue
Rank endpoint families by independent-source agreement, sensitive operation nouns, parameter novelty, historical JavaScript references, and recency. Prefer a compact target-specific corpus over a broad public list.
cat recon/archive/normalized/*.urls | unfurl paths | sed 's#^/##' | awk 'NF' | sort -u > recon/archive/normalized/route-words.txt; cat recon/archive/normalized/*.urls | unfurl keys | awk 'NF' | sort -u > recon/archive/normalized/parameter-words.txtRevalidate with a low-rate live comparison
Probe only current in-scope URLs and retain status, redirect, title, content type, body length, and observation time. Use a current known route and a randomized nonexistent route as controls. A changed status alone is not proof that an old operation still exists.
httpx -l recon/archive/normalized/candidate-urls.txt -json -silent -rl 5 -sc -cl -ct -title -location -o recon/archive/current/http.jsonlPromote only evidence-backed deltas
A promoted lead includes original source, capture time, exact historical artifact, current request and response fingerprint, control result, confidence, and one safe next discriminator. Label archive-only, dead, redirected, current, and indeterminate states separately.
Common guidance
- Agreement between Wayback, Common Crawl, and urlscan increases provenance confidence, not vulnerability confidence.
- Old JavaScript can reveal retired operation names and parameter shapes, but validate the current server behavior before testing a security boundary.
- Do not replay archived POST bodies, tokens, personal data, or state-changing requests.
- Store normalized facts and source references. Raw vendor responses should follow a defined retention limit.
Mapped workflow tools
References
- https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server ↗
- https://index.commoncrawl.org/ ↗
- https://commoncrawl.org/cdxj-index ↗
- https://urlscan.io/docs/api/ ↗
- https://urlscan.io/docs/search/ ↗
Version history: normalized permanent page created 2026-08-20.