I saw that they added it to their UI by default now!
E.g. to search for all pages under https://askubuntu.com/users/52975/ you can use: https://web.archive.org/web/*/https://askubuntu.com/users/52975/*
If you search for an archive which they don't have, they might might suggest this subpath method automatically if they have subpath matches.
Only prefix searches as possible however, where prefix is the logical URL hierarchical order (
com.domain.subdomain/path). Presumably because anything else would require a full-text search index over the URL dataset, which is much more expensive to maintain than the simpler sort by prefix index they currently have.
TODO: is there any way to download a dataset of all crawled URLs? Or alternatively query it? At 800B pages, 100 chars / URL, it would be 80 TB too which is large to download in full on a personal computer. But for a year maybe it can be just about handled. Full crawls are apparently present here: https://archive.org/details/widecrawl but with all website content they are at PiB range of course.
This is the type of dataset that would perfectly suit Google BigQuery or AWS Athena. https://commoncrawl.org/ is a more searchable related project, and it has an AWS Athena option: https://commoncrawl.org/2018/03/index-to-warc-files-and-urls-in-columnar-format/
I've also come across the "Wayback CDX Shards" dataset: https://archive.org/details/waybackcdx?tab=about which presumably contains the crawled CDX, but it is not publicly downloadable for some reason (also mentioned in collection description). It is however quite large as expected.
A UK CDX dump appears to be available for download at: https://data.webarchive.org.uk/opendata/ukwa.ds.2/cdx/
I found a way. It is not amazing, but it would have worked. You can construct a search query of an URL of type:
matchType=prefix will make it search by prefix, and
to date ranges.
This gives a text list of what I presume are archived URLs, one of which was the desired one:
com,twitter)/dmorey/status/1180312072027947008 20191005025317 https://twitter.com/dmorey/status/1180312072027947008 text/html 200 L7ZPXVHCUPLD64RKA5SSIXAQ3I6Q56GM 27230
I found this by Googling into https://github.com/internetarchive/wayback/blob/master/wayback-cdx-server/README.md which documents those search parameters.