Resolving Kali (udev/systemd) Boot Failure (part 1)
27 JAN 19 – 13:30 UPDATE: Before going down this road, please consider part duex as a starting point. It is much simpler (in comparison) and will likely work for non LVM encrypted volumes. LVM encryption is still hit or miss.
This… this sucked…
For a system and solution that is based on the stable trees coming out of Debian to have a catastrophic boot failure is at best, a damage of faith.
To set up the situation. Sometime back in the Christmas timeframe I was getting ready to participate in a “group” SANS holiday hack gathering and to prep I spun up my normal assessment system to pull a couple of the scripts and materials I had recently collected. While there, I decided, I’ll just give the thing a little update while I’m here…
What could it hurt? Right?
On begins the loss of 2 additional systems after updates in attempt to figure out what “I” did wrong. Only to discover this was an upstream bug in the udev/systemd packaging which affected some systems abilities to boot.
I have been left with a feeling much like Olaf Zaplinski (see below).
Eventually I spun a vanilla load without updates and moved on for the time being after trying to find quick fixes, of which there were none. I looked a few times over the past couple weeks in the spare few minutes I occasionally had to find more or less nothing new that was usable for a system that was not booting. All of the fixes seemed to depend on being able to already boot to the base OS. Which, in my case, and several others, that ship had sailed long ago.
Today I took nearly 8 solid hours to work on this problem and come up with a possible solution. While it is likely that many have already re-rolled systems and likely sacrificed data to arbitrary computer (or other) gods I wanted to post what I found that may help. Also, this might not fix your system; I have one I’m still working on which is still failing. I’ll follow up with another post if I find a way to address it.
BRIEF DISCLAIMER: This worked for me, it was not trivial in any way and this may not work for you at all, you might even make things worse and if bad enough, you might even lose your data so just be careful.
On with the show…
- FIRST: If you have an “old” functional kernel, this will show up on your boot screen under “Advanced options for Kali GNU/Linux”, use it and upgrade from there. That is by far the easiest and best way to resolve this problem. Furthermore, try to keep at least 1 prior “known good” kernel available for these types of problems. Recovering without one is no small task.
- Boot your system into a Live CD. In this case, I used Ubuntu 18.04.
- In my case, I had to decrypt the LVM volume as well. Assuming all goes well you should be able open the Disk Usage Analyzer program (or simply navigate to /media/ubuntu/<your disk>) and see your file system as you structured it.
- Next, you’ll want to pull down the systemd and udev (and related) files from the kali repository here (http://http.kali.org/kali/pool/main/s/systemd/) Make sure to pull the correct version both in a package sense as well as for your architecture since several different architectures are present.
- Now the real fun happens. Essentially, you’re going to use the dpkg program to “install” the packages into your root file system without having it mounted.
- You will probably run into problems, and almost certainly right away. The first will likely be the statoverride file located in /var/lib/dpkg/. It will probably say something like below:
dpkg: unrecoverable fatal error, aborting:
unknown group 'redis' in statoverride file
- It will probably be bold and red and scary. You’ll need to review your statoverride file for EACH offending entry and remove them (you might want to back up this file before doing removals).
- It should be noted that you can also use dpkg-statoverride –remove /<offending>/<path> but I felt better just removing the lines manually.
- Next, before you get to the install process you’re going to want to check your /dev/null file. Yes, that’s right, if this isn’t working right you’re done, the packages won’t install. The best way to do this is to simply cat the file out. If your receive an access denied error (like below) you have another problem to resolve.
- Also, remember that this is the /dev/null file on your attache drive, not on your live OS so make SURE you’re using that system, I move to /media/ubuntu/<drive id>/root/ and function with relative pathing to ensure I’m in the right spot
cat: ../dev/null: Permission denied
- This is NOT ideal, of course neither is this whole situation. First, you can try setting permissions or even recreating the file, see below:
rm -f /dev/null; mknod -m 666 /dev/null c 1 3
- Failing this, or using chmod, you can try symlinks, but that’s probably not going to work either. But check this out, there’s still one trick up your sleeve, you’re booted into a functional OS afterall right? Why not copy the null file from the running OS? Funny enough, this will probably work when others fail.
cp /dev/null /media/ubuntu/<drive id>/dev/
- Once you’ve completed the copy, there should be no output (especially errors) when you cat the null file.
- NOW you can move on to installs.
- dpkg comes with an important option for our use here –root. This option allows you to specify the base directory to install the programs at. This means that you can install .deb programs into the base (non-functional) Kali install without actually having to be booted into it.
- I would recommend navigating by shell to the location where you downloaded the packages from the Kali repository.
- Next, craft your command, it should look something like below (note the –simulate option as well, I would recommend starting with this and working out any additional errors; if you can’t simulate, you can’t install)
dpkg --install --root=/media/ubuntu/<drive id>/ --simulate libsystemd0_240-4_amd64.deb
- If you’re lucky enough to have the simulation go well, remove –simulate and run again to see if it actually installs.
- If you run into dependencies you’ll need to install them.
- You should start seeing some success once dependencies are met
- Continue this process until you have systemd and udev installed appropriately. If everything goes well you’ll end up with packages installed appropriately to the underlying Kali system.
If you’re lucky and things are not messed up too badly. You can reboot and get back into your OS. If not, then things are more dire and you should likely consider a reload and data recovery. I will be exploring what else can be done but almost all “solutions” that people propose involve having the ability to get booted into a minimally functional system vs. dropping to initramfs at boot time. If anyone has a viable solution I’m definitely open to suggestions.