How to search the Internet Archive's Wayback Machine by URL prefix or pattern?

Question

I know it is possible to search by domain as mentioned at: How to search Internet Archive for all pages from a particular domain but this is not always feasible for huge domains like Twitter, where there will be way too many results for a given date.

For example I spent a long time looking for: https://web.archive.org/web/20191005025317if_/https://twitter.com/dmorey/status/1180312072027947008

I knew the user's URL: https://twitter.com/dmorey and the approximate date based on news reports, but not the Tweet URL since the Tweet had been deleted, so what I would like would be to search for any URL under https://twitter.com/dmorey

I tried to add an asterisk: https://twitter.com/dmorey/*: http://web.archive.org/web/20190615000000*/https://twitter.com/dmorey/* but that has no results.

How I ended up finding the page: http://archive.is has the asterisk feature, which pointed me to the desired exact URL.

Asked them on Twitter at: https://twitter.com/cirosantilli/status/1312690953875083265

Answer 1

2021-06 update

I saw that they added it to their UI by default now! E.g. to search for all pages under https://stackoverflow.com/users/895245 you can visit the regular search window https://web.archive.org/ and type the following in the search bar:

https://stackoverflow.com/users/895245/*

ending with the asterisk *. The final search URL with the results is: https://web.archive.org/web/*/https://stackoverflow.com/users/895245/*

Also if you search for an archive which they don't have, they might might suggest this subpath method automatically if they have subpath matches.

Only prefix searches as possible however, where prefix is the logical URL hierarchical order (com.domain.subdomain/path). Presumably because anything else would require a full-text search index over the URL dataset, which is much more expensive to maintain than the simpler sort by prefix index they currently have.

TODO: is there any way to download a dataset of all crawled URLs? Or alternatively query it? At 800B pages, 100 chars / URL, it would be 80 TB too which is large to download in full on a personal computer. But for a year maybe it can be just about handled. Full crawls are apparently present here: https://archive.org/details/widecrawl but with all website content they are at PiB range of course.

This is the type of dataset that would perfectly suit Google BigQuery or AWS Athena. https://commoncrawl.org/ is a more searchable related project, and it has an AWS Athena option: https://commoncrawl.org/2018/03/index-to-warc-files-and-urls-in-columnar-format/

I've also come across the "Wayback CDX Shards" dataset: https://archive.org/details/waybackcdx?tab=about which presumably contains the crawled CDX, but it is not publicly downloadable for some reason (also mentioned in collection description). It is however quite large as expected.

A UK CDX dump appears to be available for download at: https://data.webarchive.org.uk/opendata/ukwa.ds.2/cdx/

Original answer

I found a way. It is not amazing, but it works. You can construct a search query of an URL of type:

http://web.archive.org/cdx/search/cdx?url=twitter.com/dmorey/status&matchType=prefix&limit=1000&from=20191001&to=20191007

and matchType=prefix will make it search by prefix, and from and to date ranges.

This gives a text list of what I presume are archived URLs, one of which was the desired one:

com,twitter)/dmorey/status/1180312072027947008 20191005025317 https://twitter.com/dmorey/status/1180312072027947008 text/html 200 L7ZPXVHCUPLD64RKA5SSIXAQ3I6Q56GM 27230

I found this by Googling into https://github.com/internetarchive/wayback/blob/master/wayback-cdx-server/README.md which documents those search parameters.