Guide · Rocky Linux

How to install NVIDIA drivers and CUDA on Rocky Linux 8, 9 and 10Pick your version and setup, copy the commands and check that it works.

Getting an NVIDIA GPU working on Rocky Linux takes around a dozen commands, and which ones depends on the Rocky version, whether the machine has a desktop, Secure Boot and containers.

Below you choose your case and get only the commands that apply. They come from NVIDIA's installation guide for Rocky Linux.

Before you start

First make sure the system sees the card. If this prints nothing, the problem is in the hardware or the BIOS, and no driver will fix it:

lspci | grep -i nvidia

The packages in this guide use NVIDIA's open kernel modules, which are the ones NVIDIA recommends for GPUs from the Turing generation onwards. That covers data centre cards from the T4 up and the RTX 20 series and later. Older cards such as the P100 or V100 need the proprietary modules, which this guide does not cover.

Then check whether Secure Boot is on:

mokutil --sb-state

If it says enabled, tick Secure Boot in the guide. You will need access to the server console for the next reboot, because the key confirmation screen does not show up over SSH.

Install the driver and CUDA

Choose your Rocky version and tick what applies. The steps update as you go, and each command has its own copy button. If you are setting up several machines, copy the whole thing as a script.

Version

Your setup

Check that it works

After the reboot, nvidia-smi should list your GPUs with the driver version. The CUDA version it shows in the top right corner is the highest one that driver supports, not the toolkit you have installed, which confuses a lot of people.

To see the version of the module actually loaded in the kernel:

cat /proc/driver/nvidia/version

The toolkit goes into /usr/local/cuda and is not on the PATH by default. To use the compiler, add it to your profile and check it:

export PATH=/usr/local/cuda/bin:$PATH
nvcc --version

Update and remove the driver

To update within the same branch, a normal sudo dnf update is enough. When a new kernel comes in, DKMS builds the module for it. It is worth checking before you reboot that it has done so, because otherwise the server will come back without a GPU:

dkms status

On Rocky 8 and 9, moving to another driver branch means switching the module stream:

sudo dnf module reset nvidia-driver
sudo dnf module enable nvidia-driver:<stream>
sudo dnf update --allowerasing

Rocky 10 has no module streams. If you need to stay on one branch, pin it with dnf versionlock.

To remove the driver on Rocky 8 and 9:

sudo dnf module remove --all nvidia-driver
sudo dnf module reset nvidia-driver

And on Rocky 10:

sudo dnf remove nvidia-driver\*

In both cases only the driver goes. The CUDA toolkit stays until you remove it yourself.

When there is more than one server

On one workstation, all this takes a morning. On a rack of GPU servers it turns into maintenance. Every kernel update has to rebuild the modules on every node, CUDA and PyTorch have to keep matching the driver, and a node that boots without its GPU sits idle until someone notices.

That is the problem RLC Pro AI solves. It is CIQ's edition of Rocky Linux for AI servers, and it ships the data centre driver, CUDA, cuDNN, NCCL and PyTorch installed and tested together with its own kernel. We are a CIQ partner: we deploy it on your servers and support it. The rest of the editions are on our Rocky Linux support page.

Frequently asked questions

Do I need the CUDA toolkit to run PyTorch?

No. The PyTorch packages you install with pip bring the CUDA libraries they need, so the driver is enough. The toolkit is for compiling your own CUDA code with nvcc.

nvidia-smi says it cannot communicate with the NVIDIA driver. What happened?

It is usually a kernel update the module was not built for. Run dkms status, install the headers for the new kernel if they are missing and reboot.

Does this also work on AlmaLinux?

Yes. NVIDIA documents AlmaLinux 8, 9 and 10 with the same commands as Rocky, including the module streams on 8 and 9.

Which driver branch should I use?

The open-dkms stream installs the latest. In production, many teams pin a branch such as 580 so that an update does not bring a new major version without warning.

SIXE

Running GPU servers on Rocky?

Tell us how many servers you have, which GPUs and what you want to run on them, and we will tell you whether RLC Pro AI makes sense and what the move would look like.

Talk to SIXE WhatsApp

Rocky Linux, Red Hat, SUSE and Ubuntu support · sixe.eu