Leapp or ELevate failed? Rebuilding a CentOS 7 server on EL 9 instead

You ran leapp preupgrade on a CentOS 7 server and the report came back with a list of inhibitors. Or you cleared them, ran leapp upgrade, rebooted, and the host came up half on EL 8 with packages from both releases installed. Either way, you are now deciding whether to keep fixing this machine or start over.

This post covers the inhibitors you are most likely looking at, what the quick fix for each one costs you, and the alternative that usually works out cheaper for anything that matters: build a fresh EL 9 server from a plan of what the old one does, and keep the old one running until the new one has been checked.

Why CentOS 7 to EL 9 is two upgrades, not one

CentOS 7 reached its end of life on June 30, 2024, and its repositories went offline. Before you can even install the upgrade tooling, you have to point the host at the CentOS vault or a mirror. AlmaLinux's ELevate guide gives its own CentOS 7 mirror for this.

ELevate is the AlmaLinux project that runs Red Hat's Leapp upgrade framework on RHEL rebuilds. It is built for one-step upgrades, so a CentOS 7 host goes to an EL 8 release first, and only then, in a second Leapp run, to EL 9. Red Hat's own path is the same shape: its in-place upgrade takes a RHEL 7.9 host as far as RHEL 8.10, and RHEL 9 is a second upgrade after that. So the in-place route means two pre-upgrade reports, two sets of inhibitors, two upgrade reboots, and two chances to stop half way, all on the production machine.

The inhibitors you are most likely looking at

On a default CentOS 7 install, the ELevate guide expects the pre-upgrade check to fail, and it lists the most common fixes. On a real server you will usually get those plus a few more. The report is in /var/log/leapp/leapp-report.txt, and the questions Leapp wants answered are in /var/log/leapp/answerfile.

Drivers that EL 8 no longer ships. Leapp stops if a loaded kernel module has no counterpart on the target. On CentOS 7 the usual one is pata_acpi, and the fix is rmmod pata_acpi. That is harmless on most VMs. On older hardware, check what the driver was for before you unload it, because the target kernel will not have it either.

Root login over SSH. Leapp stops when sshd_config does not set PermitRootLogin, because on EL 8 the default no longer lets root log in with a password. The ELevate guide's fix is to append PermitRootLogin yes. That passes the check by switching on exactly what the new default switches off, so decide what you actually want there and set that after the upgrade.

Unanswered questions. If the answer file has a question with no answer, Leapp stops. The common one on CentOS 7 is the pam_pkcs11 module, answered with leapp answer --section remove_pam_pkcs11_module_check.confirm=True. Red Hat warns that pam_pkcs11 and pam_krb5 can lock you out if they are marked required in your PAM stack, so look at how this host authenticates before you confirm.

Third-party packages. If a package you installed from another repository has the same name as one in the distribution, Red Hat's guide says the upgrade fails until you remove or replace it. Packages that do not clash are left behind as EL 7 builds, and the post-upgrade cleanup can remove some of them. After a 7-to-8 run, rpm -qa | grep el7 shows what was left over.

Network interface names. Interface names can change between the releases, and on hosts with network bonding Red Hat's guide has you set LEAPP_NO_NETWORK_RENAMING=1 before the upgrade and check that the host is still reachable afterwards. On a remote server with no console, that step is where people lose the box.

Authentication and name lookup. winbind or wins in /etc/nsswitch.conf makes the transaction fail until you take them out. The authconfig settings are not migrated at all; on EL 8 you set them up again with authselect.

Storage. Red Hat's guide says the in-place upgrade cannot be used on a system with full-disk, partition or file-system encryption, that network multipath and network storage over Ethernet or InfiniBand are not supported, and that it might fail on software RAID. A host with a lot of mounts can fail the pre-upgrade with unable to open database file.

Each fix is small on its own. The trouble is the total: you change the production machine to suit the upgrade tool, one workaround at a time, and every one of those changes is something you will have to explain later.

When the upgrade itself failed

There is no undo button. Red Hat's guide says to back up the system before upgrading, recommends a full backup such as ReaR or a VM snapshot, and warns that with LVM snapshots you might not be able to recover after some failures. ELevate's guide says much the same, and recommends a trial run in a sandbox before production. Going back means restoring the backup, and the restored host is the CentOS 7 server you started with, with the same inhibitors waiting for you.

Why rebuilding is often the cheaper fix

A rebuild goes from CentOS 7 to EL 9 in one step. You read what the old server does, write that down as a plan, build a new EL 9 server from the plan, check it, and switch traffic over. Compared with fixing the upgrade, that gets you:

• The old host as your rollback. Nothing is changed on it. If the new server is wrong, you point traffic back.
• No inhibitors to work around. Leapp's checks are there to protect a machine being transformed in place. A new machine has nothing to protect: the interface names, the PAM stack and the SSH settings are whatever the plan says.
• A clean target. No EL 7 packages left over, and none of the debris ten years of hand edits leave behind, only what you decided to carry.
• Something to compare against. With the old server still there, you can check the new one against it before cutover instead of finding the differences in production.

In-place still has its place. For a plain host close to the vendor defaults, or a lab machine, ELevate may well be the fastest route. The rebuild pays off on the servers that failed the pre-upgrade in the first place: the ones with third-party repositories, local changes and software nobody remembers installing.

How vrtmv does the rebuild

vrtmv turns the reading and the planning into a few commands. On the CentOS 7 host it changes nothing.

• Read the old server. vrtmv migrate --ssh admin@web01 --sudo --target alma9 -o ./out reads the host over SSH without an agent. To leave the running host alone, point --image at a copy of its disk instead.
• Read the plan. The Index translates packages, services and settings to the target. Some arrive under another name: ntp becomes chrony, authconfig becomes authselect, iptables-services becomes iptables-nft-services, and network.service is replaced by NetworkManager. What vrtmv will not guess at goes on a punch list for you: a Java 8 application moving to OpenJDK 17, or Python 2 code that has no interpreter package on EL 9.
• Build the new server. The output is an Ansible role (or Terraform, with --format terraform) that you review and run against a fresh EL 9 host.
• Move the data. vrtmv never uploads your data, and copying it is a step you start. vrtmv data plan writes the copy runbook, with a sha256 manifest of the source's files, and the export and import steps for each database it finds; you run them. Or vrtmv run --copy copies the paths you name for you. For PostgreSQL that means a dump and restore or a pg_upgrade, from 9.2 on CentOS 7 to the version you pick on EL 9.
• Check it. vrtmv verify --attestation out/vrtmv-attestation.json --ssh admin@web01-new --sudo checks the new server against the plan, from what it can observe there: its package database and the carried configuration paths. It writes a parity report and exits non-zero on any gap, so the cutover can wait on it.

Our migration guides cover what changes for each move, for example CentOS 7 to AlmaLinux 9 and CentOS 7 to RHEL 9; the rest are on the migration guides page. If you are choosing between the free rebuilds, see Rocky Linux vs AlmaLinux.

A rebuild checklist for one CentOS 7 server

• Keep the Leapp report. It is a useful list of what is unusual about this host.
• Read the source and save the inventory with the change ticket.
• Settle every punch-list item in the plan before you build.
• Build the EL 9 host from the generated role, and keep the role in version control.
• Move the data with checksums, after the final sync and before writes resume.
• Run verify, and resolve or explain every gap it reports.
• Cut over, and keep the old host powered off but intact until the new one has run cleanly for a while.

If an auditor will look at this change, what auditors want to see after a server migration covers the rest of the evidence.