Skip to main content

Running CUDA Apps in Docker on Ubuntu (The Modern Way)

 


Running CUDA Apps in Docker on Ubuntu (The Modern Way)

These days, it’s hard to find a serious Machine Learning project that doesn’t expect an NVIDIA GPU. Whether you’re training models, running inference, or doing heavy compute workloads, CUDA is usually part of the stack.

The good news: you can run GPU workloads cleanly inside Docker — without turning your host machine into dependency spaghetti.

In this guide, I’ll show you how to set up Docker + NVIDIA GPU support on Ubuntu and verify everything works by running a CUDA container.

What You’ll Need (Prerequisites)

Before you start, make sure you have:

  • Ubuntu (x86_64) — Ubuntu 20.04+ recommended (22.04 and 24.04 work great)
  • An NVIDIA GPU with CUDA support
  • A working NVIDIA driver installed on the host
  • Docker Engine installed (Docker CE recommended)

    Note: Docker’s built-in --gpus support requires modern Docker (this has been standard for years now).

Step 1 — Install Docker (Modern Ubuntu Method)

This is the current official approach using Docker’s repository and keyring (not apt-key, which is deprecated).

1) Remove old Docker installs (optional but recommended)

sudo apt-get remove -y docker docker-engine docker.io containerd runc

2) Install required packages

sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg

3) Add Docker’s official GPG key

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

4) Add the Docker repository

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo ${VERSION_CODENAME}) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

5) Install Docker Engine

sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

6) Add your user to the Docker group (so you can run Docker without sudo)

sudo usermod -aG docker $USER

⚠️ Important: Log out and log back in (or reboot) for group changes to take effect.

To test Docker:

docker run --rm hello-world

Step 2 — Install NVIDIA Drivers (Host Side)

Docker containers do not install your GPU driver — the host provides it.

To confirm your NVIDIA driver is working:

nvidia-smi

If everything is installed correctly, you’ll see your GPU listed along with driver and CUDA info.

If nvidia-smi is missing

Install drivers from Ubuntu’s repo:

sudo apt-get update
sudo apt-get install -y nvidia-driver-535
sudo reboot

Driver versions change over time. 535 is common, but Ubuntu may offer newer ones depending on your version and hardware.

Step 3 — Install NVIDIA Container Toolkit (Required)

This is the modern replacement for older “nvidia-container-runtime” instructions you might find in older tutorials.

1) Add NVIDIA’s repository key + source list

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /etc/apt/keyrings/nvidia-container-toolkit.gpg

curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/etc/apt/keyrings/nvidia-container-toolkit.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

2) Install the toolkit

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

3) Configure Docker to use the NVIDIA runtime

sudo nvidia-ctk runtime configure --runtime=docker

4) Restart Docker

sudo systemctl restart docker

Step 4 — Test CUDA Inside Docker

Now for the moment of truth: run a CUDA container and call nvidia-smi inside it.

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

If everything is working, you’ll see the same GPU information you saw on the host.

Bonus: Selecting Specific GPUs

If your machine has multiple GPUs, Docker can target specific ones:

docker run --rm --gpus '"device=0"' nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Or limit by count:

docker run --rm --gpus 1 nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Picking the Right CUDA Base Image

NVIDIA publishes multiple CUDA image types depending on what you’re doing:

  • base → smallest runtime environment

  • runtime → runtime libraries included

  • devel → includes compilers and headers (best for building inside the container)

Example:

nvidia/cuda:12.4.1-devel-ubuntu22.04

You’re Ready 🚀

At this point, you’ve got a clean, scalable workflow:

  •  Host provides the NVIDIA driver
  •  Docker provides isolation and portability
  •  CUDA workloads run inside containers like any other service

Now you can start building your own Dockerfiles and ship GPU-powered apps without fighting dependency conflicts every time you update a library.





Comments

Post a Comment

Popular posts from this blog

I Built a Docker Expert Because I Was Tired of Searching Docs

  I Built a Docker Expert Because I Was Tired of Searching Docs I didn’t set out to build a “Docker expert.” I set out to stop breaking flow. If you’ve worked with Docker long enough, you know the feeling: you know the answer is in the docs, but you don’t know where . The information is correct, but fragmented. CLI flags in one place. Concepts in another. Edge cases buried three clicks deep. By the time you find what you need, the mental context is gone. So instead of reading Docker documentation, I asked a different question: What if I could  talk  to the Docker docs?  Not a chatbot that “knows Docker” in a vague, internet-trained way; but something grounded strictly in Docker’s own words; current, precise, and boringly correct. That’s what I built. The Idea: Treat Documentation as a Dataset Docker’s documentation is excellent. It’s also public, structured, and version-controlled on GitHub. That’s the key insight. Instead of scraping random websites or relying on a...
Post a Comment
Read more

The Centralization Trap in AI

  The Centralization Trap in AI. AI is everywhere—and the debate is intense. Enthusiasts call it a force for progress: multiplying productivity, creating new industries, and amplifying human capabilities. Critics warn of job loss, erosion of autonomy, environmental strain, and even existential risks. The real issue isn’t AI itself. It’s who controls it—and who pays the costs. Centralization Always Externalizes Harm Most AI lives in massive, centralized platforms. These data centers draw enormous amounts of electricity and water. Cooling alone can consume millions of gallons annually. High demand drives grid expansion and raises energy costs for local communities, many of whom see no benefit. Platforms also control access, dictate usage, and extract data without meaningful user oversight. Profit and influence concentrate, while environmental, economic, and social costs are externalized. The danger isn’t the technology. It’s the architecture. Why Architecture Matters Centralized AI r...
Post a Comment
Read more

MarvinOS Local AI Stack: Fully Self-Hosted AI Experimentation

   MarvinOS Local AI Stack: Fully Self-Hosted AI Experimentation I magine having a powerful, fully self-hosted AI experimentation platform that allows you to explore the possibilities of artificial intelligence without relying on cloud dependencies or compromising on security. That's exactly what we're excited to introduce today — the  MarvinOS Local AI Stack ! This innovative tool is designed for local AI experimentation, secure internal LLM access, GPU-accelerated inference, and even supports offline or air-gapped environments. It also includes  Retrieval-Augmented Generation (RAG)  with local document embeddings, enabling long-term knowledge bases without leaving your machine. Key Features & Benefits MarvinOS Local AI Stack offers a wide range of features that make it an ideal choice for those who value autonomy and control over their AI experiments: Fully local LLM inference : No cloud dependency means you're in complete control. RAG with Qdrant vector d...
Post a Comment
Read more