Close

Recovering FreeNAS Configuration from ZFS Boot Drive

This article explains how to rescue a FreeNAS server with failed USB boot media and provides details on how to read files from ZFS volumes.

Just in time for the holiday break I had to deal with a FreeNAS server that failed to boot due to a defective USB boot drive. The drive started failing during a recent FreeNAS update. Of course, this happened after a number of configuration changes were made to the server and before any backup took place. To make matters worse, the server was located four thousand miles away with nobody nearby to repair or reinstall it on-site.

As a result, I had a not so welcome, but appreciated opportunity to learn about the inner workings of FreeNAS, and was able to prepare new USB boot media that successfully restored the server. For the benefit of others who may end up in similar situations, I am documenting my findings below.


Prerequisites

It is assumed that you are still able to read data off the drive containing the ZFS boot volume. You shouldn’t work directly on that drive, because it may damage it further and result in a complete data loss. Clone the drive to a second USB drive and work on that copy instead. This can be accomplished with the dd command on Linux and BSD based operating systems, and tools like Win32DiskImager on Windows.

I was able to restore my FreeNAS server without having to deal with the ZFS file system on the boot drive. You can skip to the very last section of this article if you’d like to try this automatic upgrade process. For everyone else and for future reference, I will include instructions on how to manually read files off the boot drive in the following paragraphs.

Accessing ZFS File Systems

The easiest way to read ZFS volumes is via FreeBSD, because it has ZFS support out of the box. It is also possible in Linux, but requires extra work, such as installing a kernel module. Windows does not have any ZFS support at this time and may not get it anytime soon.

If you do not have a working FreeBSD installation, you can download the latest image, burn it to a DVD or USB drive, and boot it in LiveCD mode:

  1. Boot the DVD or USB drive
  2. Select the Multi User boot option (default)
  3. Log in as root (no password)

Mounting the ZFS Volume

Remove all USB drives from your computer in order to avoid getting confused about which one is which. Insert the cloned USB drive. The USB mass storage driver should recognize it and add it to the device list as da0.

Now use the zpool command to discover ZFS pools on the drive. Zpool will automatically search all connected drives for available pools:

The pool on FreeNAS boot devices is called freenas-boot. By default, the mount points of the relevant datasets are set to the root directory, which is not what you want when mounting them into our LiveCD system. Import the pool with the mount points pointing to a temporary directory instead:

Unless you exported the pool on the FreeNAS server with zpool export, which is almost certainly not the case if the server does not boot anymore, the pool will already be marked as imported and online, so we use the -f option to force reimporting the pool into the LiveCD environment. The -R option is equivalent to "f -o altroot=" and sets the desired root mountpoint.

Let’s take a look at the pool’s status:

In my case it looks like the kernel and some other files in various snapshots are corrupted, but luckily the pool’s datasets are still intact and can be mounted. The zfs list command will output a list of all available datasets:

Note that the mountpoint property of mountable datasets are now pointing to /tmp/freenas instead of the root directory. To see all properties of the datasets (or a particular dataset) you can run the zfs get command:

So far we only imported the pool and haven’t mounted any datasets yet. The freenas-boot/ROOT/FreeNAS-9.3-STABLE-201509282017 dataset is what I am interested in, because it contains the main FreeNAS installation along with its configuration database. Depending on which version of FreeNAS you have installed, the name of your dataset may be slightly different.

Let’s mount the dataset and make its files available in /tmp/freenas:


Restoring the FreeNAS Server

At this point it may be tempting to mount any affected datasets and simply restore the corrupt files from the FreeNAS installer, but this is not ideal. On the one hand, repairing a broken ZFS file system is not straightforward, and there is no good mechanism to clear permanent errors from ZFS snapshots. The general process is to create a new pool and then migrate the old pool’s datasets to the new pool via zpool send/receive. On the other hand, with a dieing drive you never know what else might be broken, and you probably don’t want to take any chances.

FreeNAS offers two great alternatives. The naive solution is to install a fresh copy of FreeNAS on a new USB stick and import the old configuration via the web GUI. The configuration database is located in /data/freenas-v1.db. Copying it from the mounted dataset to a different drive is straightforward. In the following example I used a spare USB stick that happened to be formatted with an EXT3 file system:

Besides reimporting the configuration database there is an even simpler solution, and this is the one I would ultimately recommend, because no files need to be backed up, and no extra USB drive is required: the FreeNAS installer has an option to upgrade an existing installation. It can perform this in place with the backup USB drive containing the damaged copy of the original boot drive. It will also retain the old configuration database:

  1. Download the FreeNAS ISO, burn it to a CD or USB drive, and boot it
  2. In the boot menu, chose Install/Upgrade
  3. In the destination media menu, select the drive containing the copy of the damaged boot image
  4. Click OK. The installer will offer you to upgrade instead of reinstall, and to keep the original configuration

The installer will make a copy of the configuration database, then wipe the file system and reinstall FreeNAS, and finally restore the database on the drive. The drive is now ready to be used in the server. When booting it for the first time, FreeNAS will migrate the database to the installed version. After that, your system should be up and running, and all settings, including user accounts, network shares, plug-ins and jails should be working again.

FreeNAS is able to migrate earlier database versions to newer ones during the upgrade process, but in order to avoid any complications, it may be best to ‘upgrade’ FreeNAS to the same version as before, and then upgrade to a newer version via the web GUI after the system is up and running again.


Related Resources

3 Comments

  1. Thanks! I was able to recover my scripts and other important stuff from a corrupted FreeNAS USB-bootdrive with your instructions. On a side note: I was very glad to have a recent-enough backup of FreeNAS configuration (a .db file). I just installed a new installation of FreeNAS to a working USB stick, then booted it and imported the configuration. It did its magic and rebooted itself couple of times, and I was running the exact same configuration of my old FreeNAS setup with the old dataset imported and ready to use. Even all of the plugins were running without a hitch.

    Reply

  2. Thanks a lot for the detailed article. Saved me a few hours looking for the most appropriate way to recover the system. I went the path to reinstall on the copy of the damaged disk, back online fully operational 30 minutes later.

    Reply

  3. Appreciatᥱ this post. Will trу itt out.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *