Ok, found it after digging in Telegram's bots api - File size limit.
There's a limit of 20 MB for files sent with an HTTP URL, and 50 MB if sent with multipart/form-data
when uploaded from the browser.
The file that was working was 12 MB, while the other was 54 MB 😕
Quoting:
Sending files
...
- Provide Telegram with an HTTP URL for the file to be sent. Telegram will download and send the file. 5 MB max size for photos and 20 MB max for other types of content.
- Post the file using multipart/form-data in the usual way that files are uploaded via the browser. 10 MB max size for photos, 50 MB for other files.