waymore
Best-in-class historical URL collector. Pulls from Wayback, Common Crawl, AlienVault OTX, URLScan and VirusTotal, with response-body download if you want JS from the past.
Tags: urls, wayback, alienvault, commoncrawl
- Category
- crawl
- Maintenance signal
- active
Use this when: JavaScript. Collect URLs from a host that is in scope, then save them before you read them.
Install
pipx
pipx install waymorepip
pip install waymoreCommand templates
Collect URLs
waymore -i {domain} -mode U -oU {output}URLs + responses
waymore -i {domain} -mode B -oU {output}