Can you speed up dropbox by replacing large collections of files with disk images?

Question

I (and lots of people where I work) use dropbox and keep our work source directories in dropbox. These directories contain close to 20k small files, and we'll typically have many of them hanging around as we work on different code branches.

Dropbox takes a long time to start up, and a long time to sync everything, probably because it has to download and index all those files, and it may not be very efficient to download small files one at a time. I have a theory that I could speed up the start time by:

1) developing a tool so that code checkouts create a loopback filesystem, and mount that filesystem

2) allow dropbox to sync only the filesystem blob at the block-level, ignoring the mounted filesystem directory.

Does dropbox even do block-level sync, and if so, would this improve performance?

Answer 1

Dropbox does do block-level sync, according to this answer to a ServerFault question.

The blocks in question are 4MB, apparently. Whether your idea of a loopback filesystem / disk image to dump all the 20K files into a single file for dropbox syncing would speed up dropbox sync would depend on the speed of your internet connection - you would be trading the local indexing (checking multiple files & protocol delays) for the raw upload bandwidth (even just a few changes would probably require re-uploading the entire disk image).

However, your idea is probably a non-starter for another reason, which is that most filesystems cannot handle external updates to their block storage; you'll get I/O errors and perhaps even kernel panics if Dropbox updates an ext3fs disk image you have loop-mounted. Even if you were to use a shared-disk filesystem that is designed to handle external updates, like GFS2, Dropbox's 4MB block size and delayed update synchronization would probably make this extremely unstable.