I deleted /usr/bin, and got it back

I ran rm -rf /usr/bin on an Ubuntu server I needed.

On Ubuntu /bin is a symlink to /usr/bin, so that took /bin/bash and /bin/sh with it. The shell I had open kept running, since it was already in memory, but only its builtins worked. No ls, no cp, no apt. A new SSH login failed too, because root’s login shell was gone.

This was a while ago and I don’t remember every step anymore. This is the path that works, and it’s close to what I did.

Getting in

The open shell can’t do much, and nothing new can log in, so the way in is the provider’s rescue mode. Boot the server into the rescue system and mount the real disk:

mount /dev/sda1 /mnt   # whatever your root partition is

The libraries in /mnt/usr/lib and the package database in /mnt/var/lib/dpkg are all still there. Only the binaries are missing.

Putting the basics back

Four packages are enough to get going: busybox-static for every basic command in one file with no dependencies, dpkg and apt so the package manager can run, and gpgv so apt can check signatures. They have to come from the same Ubuntu release as the server (check /mnt/etc/os-release). Download them from packages.ubuntu.com or the archive, then unpack them straight onto the disk:

mkdir -p /mnt/usr/bin   # /bin points here, it has to exist again
for deb in busybox-static_*.deb dpkg_*.deb apt_*.deb gpgv_*.deb; do
  dpkg-deb -x "$deb" /mnt
done

dpkg-deb -x only unpacks files, so this works from the rescue system without touching its own package state.

BusyBox decides what to be from the name it’s called by, so symlinks turn it into the missing tools:

chroot /mnt /bin/busybox --install -s /bin

That gives back sh, ls, cp, tar, xargs and the rest, enough for dpkg’s maintainer scripts to run.

Letting apt rebuild the rest

dpkg still knows every package that was installed, so the fix is to reinstall all of them from inside the system:

for d in dev proc sys; do mount --bind /$d /mnt/$d; done
cp --remove-destination /etc/resolv.conf /mnt/etc/resolv.conf
chroot /mnt /bin/sh
dpkg-query -W -f='${Package}\n' > /tmp/pkgs.txt
apt-get update
xargs -a /tmp/pkgs.txt apt-get install --reinstall -y

Some packages failed the first time because something they need wasn’t back yet. Running the last line again picked them up.

Before rebooting I checked for anything left half-done:

dpkg -C
apt-get check
ls /boot/vmlinuz-* /boot/initrd.img-*

After the reboot, systemctl --failed was empty.

Anything I had put in /usr/bin by hand, outside a package, was gone for good. Everything that came from a package came back, because the package database in /var/lib/dpkg was never touched.