Unable to view filesystems other than / using Docker on Linux?

I was running a Docker container on my home machine today (Linux, x86_64). Usually Docker is trouble-free and reliable, but today I ended up navigating a deep labyrinth. Since I couldn't find any good documentation on this, and to avoid this problem in the future,  I wanted to document it.

The problem was that Docker refused to mount an external filesystem on the host as a directory on the image. I'll condense the problem to make it easy to understand. Here is the symptom:

Let's say /mnt has a filesystem that is mounted (an external disk, or a thumb drive). / is the root partition, and the only other physical partition. There are a few files on the thumb drive in /mnt/

host$ ls /mnt

fileA fileB directoryA/

The following command works file, and mounts /home/user/work-dir as /external

host$ docker run -it -v ~/work-dir:/external busybox

/external in the docker image correctly shows the contents of /home/user/work-dir, as expected.

But, if I point it to the external mount, the directory /external is empty in the docker container

host$ docker run -it -v /mnt:/external busybox

You'd expect /external in the busybox image to contain fileA, fileB and directoryA. But they don't. It gets weirder.

 If you export all of / as /external, then busybox can see everything in / except /mnt/

host$ docker run -it -v /:/external busybox

# ls /external/mnt

The output is empty, so /external/mnt/ exists, and is empty. /external/bin/ exists, and maps to /bin on the host machine, correctly, as expected.

I struggled with this for a long time. strace on docker clearly shows that it thinks /mnt doesn't exist, or is read only, even though it is definitely visible and writable in the host.

host$ date >> /mnt/out; cat /mnt/out

Sun 09 Aug 2020 04:26:16 PM PDT

So the directory structure is visible, writable, but never in a docker image. Doesn't help if you try all the commands as root. What gives?

The problem was that my docker installation was from snap. Snaps have limited visibility of the remaining system. This might be fine for a snap containing a calculator app, but docker needs a lot to function.

$ sudo snap remove docker; apt install docker-ce{,-cli}

When you do that, remember to log out because otherwise bash might have hashed the location of 'docker' to /snap/bin/docker, and it takes a logout to clear bash's state and realize that docker is now at /usr/bin/docker.

Snaps are a great idea, and I look forward to a time when they are functionally equivalent to packages installed through apt. For now, my experience with docker on snap makes me reluctant to use them. They're an additional layer of complexity to understand and debug, in a system that is already plenty complicated.