Format for ID of YouTube video

Question

Every YouTube video has a unique ID which can be used to get at it. For example, the video at http://www.youtube.com/watch?v=aN46pEO_jX8 has the id aN46pEO_jX8.

After some observation, it looks to me like these IDs obey the following two rules:

• Exactly 11 characters
• Allowed symbols: a-z, A-Z, 0-9, -, and _

I want to know:

1. Whether these two rules are both always correct.
2. If there are any other rules that are important to follow.

Answer 1

According to the Youtube 2.0 API documentation and 3.0 API documentation, the videoId is a string, nothing is specified about the current set of characters used...

About the 11 characters length, a post from a Youtube API Team say :

I don't see anywhere in the documentation where we officially commit to a standard length of 11 characters for YouTube video ids. It's one of those things where we have a current implementation, and it may stay that way indefinitely. But we're not offering any official commitment to that, so proceed at your own risk.

And last but not least, another post clarify (or not) the format :

We don't make any public guarantees about the format for video ids. While they're currently 11 character strings that contain letters, numbers and some punctuation, I wouldn't recommend hardcoding that into your application (unless you have an easy way of changing it in the future).

The Youtube team seems to prefer to directly ask Youtube server if the Video_ID is correct or not (refer to an existing video) :

If you need to validate that random user input corresponds to a valid video id, I'd recommend doing an empirical test. Attempt to access

http://gdata.youtube.com/feeds/api/videos/VIDEO_ID

If you get a 200 response, then VIDEO_ID is valid. If you get a non-200 response, you have an invalid id. There are some edge cases for newly uploaded videos or private videos, but for most purposes I'd assume that would be okay.

Answer 2

Summary

---

YouTube videoId and channelId identifiers are single integer values represented in a slightly mod­ified version of Base64 encoding. One difference versus the IETF RFC4648 recommendations is the substitution of two characters in the encoding alphabet:

 Payload  ASCII/Unicode      Base64     YouTube
 -------  -------------     ---------  ---------
  0...25  \x41 ... \x5A     'A'...'Z'  'A'...'Z'
 26...51  \x61 ... \x7A     'a'...'z'  'a'...'z'
 52...61  \x30 ... \x39     '0'...'9'  '0'...'9'
    62    \x2F vs. \x2D  →   '/' (2F)   '-' (2D)
    63    \x2B vs. \x5F  →   '+' (2B)   '_' (5F)

The substitution is likely due to the fact that, for some reason RFC4648 selected two characters that already had prominent and well-established functions in URLs.^{[note 1.]} Obviously, for the usage under discussion here, that particular complication was best avoided.

Another difference from the official specification is that YouTube identifiers do not use the = padding character; it's not necessary because the encoded lengths expected per respective de­coded integer size are fixed and known (11 and 22 encoded 'digits' for 64 and 128 bits, respec­tively).

With one minor exception (explained below), the full details of the Base64 mapping can be inferred from publicly accessible data. It turns out that, with no proprietary knowledge—and essentially no guesswork—it must be the case that the Base64 scheme used in the videoId and channelId strings is as follows:

    ——₀————₁————₂————₃————₄————₅————₆————₇————₈————₉———₁₀———₁₁———₁₂———₁₃———₁₄———₁₅—
     00ᴴ  01ᴴ  02ᴴ  03ᴴ  04ᴴ  05ᴴ  06ᴴ  07ᴴ  08ᴴ  09ᴴ  0Aᴴ  0Bᴴ  0Cᴴ  0Dᴴ  0Eᴴ  0Fᴴ
00→ 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111
      A    B    C    D    E    F    G    H    I    J    K    L    M    N    O    P

    —₁₆———₁₇———₁₈———₁₉———₂₀———₂₁———₂₂———₂₃———₂₄———₂₅———₂₆———₂₇———₂₈———₂₉———₃₀———₃₁—
     10ᴴ  11ᴴ  12ᴴ  13ᴴ  14ᴴ  15ᴴ  16ᴴ  17ᴴ  18ᴴ  19ᴴ  1Aᴴ  1Bᴴ  1Cᴴ  1Dᴴ  1Eᴴ  1Fᴴ
01→ 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111
      Q    R    S    T    U    V    W    X    Y    Z    a    b    c    d    e    f

    —₃₂———₃₃———₃₄———₃₅———₃₆———₃₇———₃₈———₃₉———₄₀———₄₁———₄₂———₄₃———₄₄———₄₅———₄₆———₄₇—
     20ᴴ  21ᴴ  22ᴴ  23ᴴ  24ᴴ  25ᴴ  26ᴴ  27ᴴ  28ᴴ  29ᴴ  2Aᴴ  2Bᴴ  2Cᴴ  2Dᴴ  2Eᴴ  2Fᴴ
10→ 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111
      g    h    i    j    k    l    m    n    o    p    q    r    s    t    u    v

    —₄₈———₄₉———₅₀———₅₁———₅₂———₅₃———₅₄———₅₅———₅₆———₅₇———₅₈———₅₉———₆₀———₆₁———₆₂———₆₃—
     30ᴴ  31ᴴ  32ᴴ  33ᴴ  34ᴴ  35ᴴ  36ᴴ  37ᴴ  38ᴴ  39ᴴ  3Aᴴ  3Bᴴ  3Cᴴ  3Dᴴ  3Eᴴ  3Fᴴ
11→ 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111
      w    x    y    z    0    1    2    3    4    5    6    7    8    9    -    _

The analysis below also establishes further non-obvious constraints on the expressed form of the videoId and channelId identifiers. This (presumably novel) result is summarized by the following two regular expressions which I provide here for those who aren't interested in reading the formal analysis that follows.

RegEx

---

videoId:    [0-9A-Za-z_-]{10}[048AEIMQUYcgkosw]

channelId:  [0-9A-Za-z_-]{21}[AQgw]

Range and Distribution

---

When a new item is created, YouTube permanently assigns it an unused ID value of the appropriate type. Initial values appear to be randomly selected with a strongly uniform distribution across the entire bitness range (64- or 128-bit), allowing for up to 2⁶⁴ (18​ quintillion) or 2¹²⁸ (3.4 × 10³⁸) ID values respectively. No metadata is encoded; for example, a videoId value contains no information about what channel it comes from. It is likely that ID values of cancelled items are never reused or recycled. It also seems likely that certain ID values are excluded from ever being issued; for example, where the Base64 encoding happens to contain the spelling of a suggestive word in some natural language.

Discussion

---

The reason to believe that Base64 is being used is that, when we assume standard integer sizes of 64 and 128 bits for the encoder input, Base64 predicts the unusual character lengths (11 and 22 characters) of the YouTube channelId and videoId identifiers exactly. Furthermore, remainders calculated as per Base64 perfectly explain the observed distributional variation found in the l̲a̲s̲t̲ c̲h̲a̲r̲a̲c̲t̲e̲r̲ of each type of identifier string. Discussion of these points follows.

In both cases, the binary "data" that gets Base64-encoded is a s̲i̲n̲gl̲e̲ i̲n̲t̲e̲ge̲r̲, either 64 or 128 bits, for (respectively) videoId vs. channelId. Accordingly, by using a Base64 decoder, a single integer can be recovered from the respective s̲t̲r̲i̲n̲g i̲d̲e̲n̲t̲i̲f̲i̲e̲r̲. It can be quite useful to do this because, while each integer id contains exactly the same information as the Base64 string—and also allows the string to be recreated at any time—when compared to Base64 strings stored as Unicode, the binary representation is:

• 63% smaller,
• has the (obviously maximal) bit-density of 100%,
• aligns in memory better,
• sorts and hashes faster, and, perhaps most importantly,
• eliminates false collisions between identifiers that differ only in orthographic case.

This last problem, though extremely improbable numerically, nevertheless cannot be ruled out when Base64 IDs are treated as case-insensitive, as some filesystems do (e.g. Windows, dating back to DOS). That's kinda important: if you're using a videoId / channelId string as part of a Windows/‌NTFS filename, there's a vanishingly miniscule—but nevertheless non-zero—chance of filename collisions due to those filesystems deploying case-insensitive path and file naming.

If you're worried about the extremely remote possibility of filename collisions, one way to formally eliminate it would be to re-encode the decoded integers—still obtained as described in this article—into either a base-10 (decimal) or (thence case-agnostic) hexadecimal representation, for use in path or file names on such filesystems.^{[note 2.]} In this approach, the 64-bit videoId would need 20 decimal digits [0-9] or 8 hex digits [0-9,A-F] (vs. 11 Base64 digits). The 128-bit channelId would require a maximum of 39 decimal digits or 16 hex digits (vs. 22 Base64 digits).

Analysis

---

You might have noticed above that I wrote that you can recover "an" integer from the exposed string value. Wouldn't this simply be the value that was originally encoded? The answer is almost certainly "yes" (and this section will discuss the supporting analysis in some detail). Note, however, that while we can be almost certain that the we are obtaining the exact same binary value recorded "internally" in YouTube's master vault, this fact is also quite certainly moot.

In other words, while the evidence shown in the sections below is convincing and perhaps in­ter­esting or satisfying as well, it also doesn't matter if it is wrong. As long as YouTube never exposes the so-called "correct" values in binary form somewhere/anywhere publicly, then decoded 64- or 128-bit values have no use except as an opaque identifying token, and any binary re-encoding used privately can't possibly matter.

The only only true requirements for the transform are distinct encoding (no two unique tokens collide) and perfect reversibility (decoding recovers the original token identity). All we really care about is lossless round-tripping of the original Base64 string. Since Base64 is lossless and reversible (as long as you always stick to the same alphabet mapping and endianness assumption for both encoding and decoding) it satisfies our purposes.

There are a few clues that can tell us about the "true" Base64 mapping. The crux of the puzzle is that only certain mappings correctly predict the final-position characters that we observe, meaning the binary value for only those characters must have a certain number of LSB zeros. Heh. The mathematical details are specific to the each bit width, so I discuss the 64- and 128-bit identifiers separately in the next two sections.

Suffice it to say that, taken together with the overwhelmingly likely assumption that the alphabet and digit characters are mapped in ascending order, we can basically confirm the mapping to be what is shown in the tables above. The only remaining uncertainty about which the analysis below is math­ematically inconclusive concerns the mapping of the - and _ characters for encoding Base64 digits 62 and 63 (i.e., respectively?).^{[note 3.]}

videoId

---

The videoId identifier string encodes an 8-byte (64-bit) integer. Applying Base64-encoding to 8 bytes of data requires 11 characters. However, since each Base64 character conveys exactly 6 bits (viz., 2⁶ equals 64), this allocation could actually hold up to 11 × 6 = 66 bits—a surplus of 2 bits over the 64 bits our payload needs. The excess bits are set to zero, which has the effect of excluding certain characters from ever appearing in the last position of the encoded string. In par­ticular, the videoId is guaranteed to always end with one of the following characters:

{ A, E, I, M, Q, U, Y, c, g, k, o, s, w, 0, 4, 8 }

Thus, the maximally-constrained regular expression (RegEx) for the videoId would be as follows:

[0-9A-Za-z_-]{10}[048AEIMQUYcgkosw]

channelId

---

The channelId (and some types of playlistId) strings are produced by Base64-encoding a 128-bit (16-byte) binary integer. This gives a 22-character string which can be prefixed with either UC to identify the channel itself, or with UU to identify a full playlist of the videos it contains. These 24-character prefixed strings are used in URLs. For example, the following two URLs pertain to the same channel, since they have the same 22-digit Base64 suffix (K8sQmJBp8GCxrOtXWBpyEA, shown in bold):

Channel URL:         https://www.youtube.com/channel/UCK8sQmJBp8GCxrOtXWBpyEA

Playlist URL:  https://www.youtube.com/playlist?list=UUK8sQmJBp8GCxrOtXWBpyEA

Both of these URL formulations can be useful. For example, the playlist page includes information on the total number of videos in the channel,^{[see note 4.]} a useful piece of information which other channel pages don't seem to expose.

Similar to the case with the 11-character videoId, calculation per Base64 assuming an 8-byte integer correctly predicts the obser­ved string length of 22-characters. In this case, the output is capable of encoding 22 × 6 = 132 bits, a surplus of 4 bits, and those bits being zero ends up restricting m̲o̲s̲t̲ of the 64 alphabet symbols from appearing in the last position, with only 4 remaining eligible. We therefore know that the last char­ac­ter in a YouTube channelId string must be one of the following:

{ A, Q, g, w }

This gives us the maximally-constrained regular expression for a channelId:

[0-9A-Za-z_-]{21}[AQgw]

Remember, this regular expression describes the bare 22-character channelID value only, without any prefixes, slashes, separators, etc., that might be present in URLs, within HTML pages, and the other various uses. Indeed more generally, although the RegEx patterns I give here (and above) are mathematically minimal for identifier strings in isolation, they are still likely to generate a lot of false-positives (that is, incorrectly match spurious text) if used as-is without additional surrounding con­text. To avoid this problem in actual use, be sure to elaborate these regular expressions with as much of the expected adjacent or surrounding context as possible.

Implementation

---

For the 64-bit (videoId) case, decoding to binary is trivial because you can use a UInt64 (ulong in C#) to hold the resulting "native" binary value.

// Recover the unique 64-bit value from an 11-character videoID (native-endian)
//
// The method of padding shown here (i.e. 'b64pad') is provided to demonstrate the
// full and correct padding requirement for Base64 in general. For our cases:
//
//    videoId    →  11 chars  →  b64pad[11 % 3]  →  b64pad[2]  →  "="
//    channelId  →  22-chars  →  b64pad[22 % 3]  →  b64pad[1]  →  "=="
//
// Note however that, because it returns 'ulong', this function only works for videoId
// values, and the padding will always end up being "=". This is assumed in the revised
// version of this code given further below, by just hard-coding the value "=".

static ulong YtEnc_to_videoId(String ytId)
{
    String b64 = ytId.Replace('-', '+').Replace('_', '/') + b64pad[ytId.Length % 3];

    return BitConverter.ToUInt64(Convert.FromBase64String(b64), 0);
}

static String[] b64pad = { "", "==", "=" };

For the case of the 128-bit values, it's slightly trickier because, unless your compiler has an __int128 representation, you'll have to figure out a way to store the whole thing and keep it combobulated as you pass it around. A simple value type (or System.Numerics.Vectors.Vector<T>, which manifests as a 128-bit SIMD hardware register, when available) will do the trick in .NET (not shown).

A final technical point concerns endianness. You might in fact want to intentionally choose big-endian for your binary interpretation, even though it's less common than little-endian nowadays. The reason is that this is a case of dual views on the same value, such that the actual byte order is concomitantly exposed in the Base64 rendition. It's helpful and less confusing to keep the sort order consistent between the binary value and the (somewhat more) human-readable Base64 string, but the sort of the little-endian binary values is a non-trivial scramble of the desired ASCII/lexical sort.

Because of the irregular overlap between the 6-bit pattern and 8-bit bytes, there's no simple fix for the human-readable sorting problem once you encode to little-endian ID values (i.e. simply revers­ing the sort won't work). Instead, you have to plan ahead and reverse the bytes of each binary value prior to decoding, in other words, apply the endianness transform. So if you care about the alphabetical display matching the sorting of the binary values, you might want to alter the function shown above so that it decodes into big-endian ulong values instead. Here's that code:

// Recover the unique 64-bit value from an 11-character videoID (big-endian)

static ulong YtEnc_to_videoId(String ytId)
{
    ytId = ytId.Replace('-', '+').Replace('_', '/') + "=";
    var a = Convert.FromBase64String(ytId);
    if (BitConverter.IsLittleEndian)   // true for most computers nowadays
        Array.Reverse(a);
    return BitConverter.ToUInt64(a, 0);
}

---

Notes
_[ 1. ]
As promised above, here is an excerpt from the Base64 specification which discusses the con­sid­er­a­tions for selecting alphabet symbols. Individuals seeking to understand how the working group ultimately managed to recommend a character '/' which is fundamental to URL semantics—indeed one of a tiny few the citation itself identifies (twice!) as "particularly problematic"—may find the discus­sion somewhat unedifying, if not entirely bizarre:

3.4. Choosing the Alphabet

Different applications have different requirements on the characters in the alphabet. Here are a few requirements that determine which alphabet should be used:

• Handled by humans. The characters "0" and "O" are easily confused, as are "1", "l", and "I". In the base32 alphabet below, where 0 (zero) and 1 (one) are not present, a decoder may interpret 0 as O, and 1 as I or L depending on case. (However, by default it should not; see previous section.)

• Encoded into structures that mandate other requirements. For base 16 and base 32, this determines the use of upper- or lowercase alphabets. For base 64, the non-alphanumeric characters (in particular, "/") may be problematic in file names and URLs.

• Used as identifiers. Certain characters, notably "+" and "/" in the base 64 alphabet, are treated as word-breaks by legacy text search/index tools.

There is no universally accepted alphabet that fulfills all the requirements. For an example of a highly specialized variant, see [IMAP - RFC 3501 5.1.3.]. In this document, we document and name some currently used alphabets.

_[ 2. ]
Alternatively, to solve the problem of using Base64-encoded ID strings as "as-is" components of file or path names on the NTFS filesystem, which is case-insensitive by default (and thus technically risks incorrectly conflating one or more unrelated ID values), it so happens that NTFS can be configured with case-sensitive path/file naming on a per-volume basis. Enabling the non-default behavior may fix the problem described here, but is rarely recommended since it alters expectations for any/all the disparate applications that inspect or access the volume. If you're even considering this option, read and understand this first, and you'll probably change your mind.

_[ 3. ]
As the analysis shows, it's because neither - nor _ can ever appear as the final character of an ID that the small uncertainty remains.

_[ 4. ]
I believe the total number of videos shown the channel playlist page takes into account an exclu­sion for videos which are restricted according to the geographical region of the HTTP client. This accounts for any discrepancy between the number of videos listed for the playlist vs. channel. (update - May 2021: The web page for a playlist now seems to include an alert message when one or more of its videos are "unavailable".)