I know it is possible to search by domain as mentioned at: How to search Internet Archive for all pages from a particular domain but this is not always feasible for huge domains like Twitter, where there will be way too many results for a given date.

For example I spent a long time looking for: https://web.archive.org/web/20191005025317if_/https://twitter.com/dmorey/status/1180312072027947008

I knew the user's URL: https://twitter.com/dmorey and the approximate date based on news reports, but not the Tweet URL since the Tweet had been deleted, so what I would like would be to search for any URL under https://twitter.com/dmorey

I tried to add an asterisk: https://twitter.com/dmorey/*: http://web.archive.org/web/20190615000000*/https://twitter.com/dmorey/* but that has no results.

How I ended up finding the page: http://archive.is has the asterisk feature, which pointed me to the desired exact URL.

Asked them on Twitter at: https://twitter.com/cirosantilli/status/1312690953875083265

1 Answer 1


2021-06 update

I saw that they added it to their UI by default now!

E.g. to search for all pages under https://askubuntu.com/users/52975/ you can use: https://web.archive.org/web/*/https://askubuntu.com/users/52975/*

If you search for an archive which they don't have, they might might suggest this subpath method automatically if they have subpath matches.

Only prefix searches as possible however, where prefix is the logical URL hierarchical order (com.domain.subdomain/path). Presumably because anything else would require a full-text search index over the URL dataset, which is much more expensive to maintain than the simpler sort by prefix index they currently have.

TODO: is there any way to download a dataset of all crawled URLs? Or alternatively query it? At 800B pages, 100 chars / URL, it would be 80 TB too which is large to download in full on a personal computer. But for a year maybe it can be just about handled. Full crawls are apparently present here: https://archive.org/details/widecrawl but with all website content they are at PiB range of course.

This is the type of dataset that would perfectly suit Google BigQuery or AWS Athena. https://commoncrawl.org/ is a more searchable related project, and it has an AWS Athena option: https://commoncrawl.org/2018/03/index-to-warc-files-and-urls-in-columnar-format/

I've also come across the "Wayback CDX Shards" dataset: https://archive.org/details/waybackcdx?tab=about which presumably contains the crawled CDX, but it is not publicly downloadable for some reason (also mentioned in collection description). It is however quite large as expected.

A UK CDX dump appears to be available for download at: https://data.webarchive.org.uk/opendata/ukwa.ds.2/cdx/

Original answer

I found a way. It is not amazing, but it would have worked. You can construct a search query of an URL of type:


and matchType=prefix will make it search by prefix, and from and to date ranges.

This gives a text list of what I presume are archived URLs, one of which was the desired one:

com,twitter)/dmorey/status/1180312072027947008 20191005025317 https://twitter.com/dmorey/status/1180312072027947008 text/html 200 L7ZPXVHCUPLD64RKA5SSIXAQ3I6Q56GM 27230

I found this by Googling into https://github.com/internetarchive/wayback/blob/master/wayback-cdx-server/README.md which documents those search parameters.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.