Week 8 · Lesson 5

Getting a GPU, or not needing one

You can do all of week 8 without a GPU: the grader's emulator runs your CUDA on the CPU and checks every bit. A real NVIDIA GPU adds one thing, the speed test. This lesson lists the ways to get one, from free to about a dollar for the whole week, and how to connect.

The paths

path	cost	what it grades	you need
no GPU: the emulator	free	every test's bits. The speed test only checks that your CUDA builds	`clang++` or `g++`
your own NVIDIA GPU, Linux	free	everything	the NVIDIA driver and the CUDA toolkit (`nvcc`)
your own NVIDIA GPU, Windows	free	everything	the NVIDIA driver and Docker Desktop
a free notebook: Colab or Kaggle	free, with limits	everything, if `nvcc` is there	a Google or Kaggle account
a rented GPU: Vast.ai or RunPod	about $0.15 to $0.70 an hour	everything	an account with a few dollars of credit, and SSH

A Mac has no NVIDIA GPU, so on a Mac use the emulator, a notebook, or a rented machine. A good plan for anyone: build the whole week against the emulator, then spend an hour on a real GPU at the end for the speed test.

No GPU: the emulator

With no nvcc on your PATH and no --cuda-docker, the grader uses its emulator automatically. Nothing to set up beyond the C++ compiler you likely have already (clang++ --version or g++ --version). It compiles your CUDA as C++, runs blocks one after another, and runs each block's threads as real threads with a real barrier. Wrong bits, missing barriers and unzeroed shared memory all fail there just as on a GPU. Only speed can't be measured, since a CPU running 1024 threads per block is nothing like a GPU.

Your own NVIDIA GPU

On Linux, install the NVIDIA driver and the CUDA toolkit (from NVIDIA's site, or sudo apt install nvidia-cuda-toolkit on Ubuntu, which is older but fine for this course). Then nvidia-smi should list your GPU and nvcc --version should work. The grader finds nvcc on its own.

On Windows, building CUDA natively needs Visual Studio's compiler. Docker is simpler, and it's how this course was tested. With a recent NVIDIA driver and Docker Desktop (WSL 2 backend), check that containers can see the GPU:

docker run --rm --gpus all nvidia/cuda:12.6.3-base-ubuntu22.04 nvidia-smi

If that prints your GPU, pass a CUDA image with the compiler in it to the grader. Your own compiler still runs on Windows. Only the CUDA build and run happen in the container:

python tools/test.py --compiler "C:/path/to/your.exe" --week 8 --cuda-docker nvidia/cuda:12.6.3-devel-ubuntu22.04

The -devel image takes about 11 GB of disk, a one-time download. Setting the environment variable UIR_CUDA_DOCKER to the image name does the same as the flag.

Free notebooks

Google Colab gives free GPU time, usually on an NVIDIA T4. Choose Runtime, then Change runtime type, then a GPU. Free GPUs aren't guaranteed, and sessions end when idle. Kaggle notebooks give about 30 GPU hours a week. Turn on the GPU accelerator and internet access in the notebook's settings, which asks you to verify a phone number once.

In both, a line starting with ! runs in a shell. Check for the compiler first:

!nvidia-smi
!nvcc --version
!git clone https://github.com/wilsonmike/ml-compilers-course

Then upload your compiler's source, build it there, and run the grader. A Python implementation runs as is. An OCaml one needs opam installed in the notebook, which takes a while every session, so for OCaml a rented machine is less hassle.

Renting a GPU

Marketplaces rent you a whole machine with a GPU by the hour, with a root shell over SSH. As of November 2026, an RTX 3090 or 4090 cost roughly $0.15 to $0.40 an hour on Vast.ai, where independent hosts list their machines, and about $0.35 to $0.70 on RunPod, depending on whether you pick its community or its secure cloud. Prices change weekly, so check before you start. This week needs an hour or two, so a dollar or so.

The steps are the same on both:

Make an SSH key on your machine, if you don't have one: ssh-keygen -t ed25519. Accept the defaults. The public half is ~/.ssh/id_ed25519.pub.
Make an account and add credit: a small amount, with automatic top-up off. That caps what a forgotten machine can cost.
Add your public key in the account's SSH keys settings. Paste the contents of the .pub file, never the private one.
Start a machine with any recent NVIDIA GPU, and an image that has nvcc: a CUDA -devel image such as nvidia/cuda:12.6.3-devel-ubuntu22.04, or the provider's PyTorch template.
Connect with the SSH command the provider shows. On RunPod, pick a pod with a public IP and TCP port 22 exposed, since its default proxied SSH doesn't support scp or VS Code. The command looks like ssh -p 41234 [email protected]. Run nvidia-smi and nvcc --version to check.
Copy your work in: git clone the course (first apt update && apt install -y git if the image lacks it), and copy your compiler from your machine with scp -P 41234 -r my-compiler [email protected]:~/. Install Python or OCaml as in week 0 (apt install -y python3, or apt install -y opam).
Run the grader as on your own machine. It finds nvcc by itself.
Stop or destroy the machine when you're done. A running machine bills by the hour whether you use it or not. A stopped one usually still bills a little for its disk. Destroy it when the week is over.

To edit on the rented machine as if it were local, VS Code's Remote-SSH extension works well. Add the machine to ~/.ssh/config:

Host gpu
  HostName 203.0.113.7
  Port 41234
  User root
  ForwardAgent no

Then ssh gpu connects, and in VS Code "Remote-SSH: Connect to Host" lists gpu. Before connecting to a machine you don't own, turn off two VS Code settings that are on by default, remote.SSH.enableAgentForwarding and git.terminalAuthenticationEnabled. Otherwise anyone with root there can use your SSH key and your Git login while you're connected.

A rented machine is someone else's computer

On a marketplace, the host can see what's on their machine. Bring only what the course needs: the course repo and your compiler. Never copy your private SSH key, a GitHub token, a password or an API key there. Clone public repos over HTTPS, copy your own code in with scp, and copy results back the same way.

Check yourself

You have no GPU. What does the week 8 suite check?

Read "No GPU: the emulator" again.

The emulator runs the same CUDA file with real threads and a real barrier, so it checks correctness fully. Timing needs the real hardware.

On Windows with an NVIDIA GPU, what does --cuda-docker run inside the container?

Where does your compiler run?

Your compiler runs on Windows and writes the CUDA. The grader then builds and runs it in the CUDA image, with --gpus all giving the container your GPU.

Which file do you paste into a GPU provider's SSH key settings?

Which one is safe to give away?

The public key lets the machine recognize you. The private key proves you're you, and it never leaves your computer.

You're done for the day on a rented machine. What stops the bill?

A machine bills while it runs.

Disconnecting leaves it running and billing. Stopping ends the hourly GPU charge, though most providers still charge a little for a stopped machine's disk.