All posts
Edit 한국어
AI 8 min read
AI assisted

Driving a Free Colab GPU from the CLI

Use google-colab-cli to drive free Colab GPUs from your laptop

#Colab #google-colab-cli #GPU #vLLM #cloudflared #T4

Colab is interactive at its core: paste a cell, run it, look at the output, move to the next. Recently, google-colab-cli shipped. What it does: it lets you drive that same interactive kernel from a local shell instead. colab new grabs a session, colab exec runs scripts against it, colab stop gives it back. Think of it as a free-tier GCE spot instance you drive from your shell — no card on file, no invoice, no GCP console. Personally I've been waiting for exactly this. The tradeoff mirrors a real spot VM: cheap-or-free, but preemptible — quota-capped, torn down on idle, hard-capped at 12 hours. Colab's awkward parts — the inbound firewall, quotas, session volatility — do not go away, but the CLI makes them things you can address in a shell. This post walks that setup end to end through one real session.

What the tool actually is

google-colab-cli is a Python-installed client (pip install google-colab-cli) that authenticates against your Google account and lets you drive Colab kernels remotely. It exposes a handful of verbs — new, sessions, exec, upload, download, stop — and each one talks to the same backend that the notebook UI does. The kernel that runs is the same kernel: same GPU, same disk, same idle timeout. What changes is that your editor, your version control, and your CI can now touch that kernel without a browser in the middle.

Two things make this useful beyond "notebooks without the notebook":

  • Scripts live in a repo, not in cells. You keep them in git, edit them locally, and exec them against the remote kernel. When the kernel dies you re-run the same file.
  • The CLI composes. exec returns stdout/stderr on completion. That lets a shell script chain steps, poll for status, or gate on output — the way you would with any other remote runner.

Sessions and quotas

The GPU is a slot, not a resource, and the tool makes the slot visible.

colab new --gpu t4        # returns a session id like e87c7c
colab sessions            # list active sessions
colab stop -s <SESS>      # release the slot

Two things worth remembering:

  • TooManyAssignmentsError (HTTP 412) is a quota, not a bug. Google's assigner refuses a second GPU slot when you already hold one. Idle sessions still hold the slot — sleep 1000 from an hour ago is enough to fail the next new. Fix: sessions to see what is alive, stop on the leftover, retry.
  • Sessions are volatile. Kernels get torn down after roughly 90 minutes of idle and hard-capped around 12 hours total. Anything you want to survive that has to leave the kernel — pushed to a repo, uploaded elsewhere, or captured through a tunnel.

That volatility is the reason for the workflow: keep the source of truth in your repo, treat the kernel as ephemeral compute.

Execution model

Almost every step in a session takes this shape:

colab exec -s <SESS> -f scripts/NN_step.py --timeout N

The script runs inside the VM. Its /content is the VM's /content, its localhost:8000 is the VM's localhost. That distinction is easy to miss when you first switch from a laptop workflow — a path or a port in the script only makes sense from the kernel's point of view, not from yours.

Two consequences show up in almost every non-trivial run:

  • Background processes need a nohup + log-file pattern. exec returns when the script returns. If you Popen a server and let the script exit, the server keeps running in the kernel; if you want to see its output you have to have written it to a log file the next exec can tail. This is what real Colab CLI setups look like — install/serve happen in one exec, poll happens in the next.
  • Environment does not leak backwards. Setting os.environ["X"] = "y" inside a script only affects that script. If a later step needs the variable, set it in the shell prefix of that step's command, not in a previous step. Same with LD_LIBRARY_PATH, PYTHONPATH, anything.

Files in and out

The Colab VM sits behind Google's edge network with no public IP and inbound filtered. scp, ssh, opening a port with iptables — none of these reach the VM, because the block is upstream. The CLI's answer is a pair of verbs that ride the same authenticated tunnel as exec:

colab upload  -s <SESS> local.png /content/local.png
colab download -s <SESS> /content/out.json ./out.json

That covers files. What it does not cover is arbitrary TCP — if a script on the kernel needs to serve a socket to the outside world, you need an outbound-initiated tunnel that the VM dials out on. cloudflared Quick Tunnel is the shortest working option (see below); ngrok and Tailscale funnel are equivalents.

Exposing a server: outbound tunnels

The reason for the pattern is worth stating cleanly:

  • The VM cannot accept a new inbound connection.
  • The VM can freely make outbound connections.
  • Both cloudflared, ngrok, tailscale funnel open a persistent outbound connection to their edge and accept public HTTPS requests on that edge, then forward down the tunnel.
  • So requests reach the VM through a return-path that was opened from the inside.

cloudflared's Quick Tunnel is zero-config:

curl -sL https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64 \
  -o /content/cloudflared && chmod +x /content/cloudflared
nohup /content/cloudflared tunnel --url http://localhost:8000 --no-autoupdate \
  >> /content/cf_tunnel.log 2>&1 &

After a few seconds the log carries https://<random>.trycloudflare.com. The URL is public and unauthenticated — Cloudflare will forward anyone's HTTPS request through the tunnel. If the demo lives longer than a demo, add an API key at the server (vllm serve ... --api-key <KEY>, or the equivalent for whatever you are serving). If you want a stable hostname, use a Cloudflare account + owned domain (free tier) via a Named Tunnel.

In practice: serving a 1B VLM on T4

The exercise that shook the workflow out: serve PaddleOCR-VL (~1B, ERNIE-4.5 + NaViT) on a free T4, and OCR an invoice image over the public URL from my laptop.

The CLI part is uneventful once the pattern above is clear:

colab exec -s $SESS -f scripts/03_start_install.py --timeout 60      # pip install vllm
colab exec -s $SESS -f scripts/04_poll_install.py  --timeout 200     # tail install log
colab exec -s $SESS -f scripts/08_start_server.py  --timeout 60      # nohup vllm serve
colab exec -s $SESS -f scripts/09_poll_server.py   --timeout 120     # /v1/models poll
colab exec -s $SESS -f scripts/10_test_ocr.py      --timeout 180     # local OCR round-trip
colab exec -s $SESS -f scripts/11_start_tunnel.py  --timeout 200     # cloudflared
colab exec -s $SESS -f scripts/12_get_url.py       --timeout 60      # extract trycloudflare URL

There are three T4-specific traps that any real serving job on Colab has to clear, and they are not CLI problems — they are what you find once your script actually runs on a Turing card with vLLM's stable-libtorch ABI:

  • vLLM 0.24.0 links against CUDA 13 while torch keeps CUDA 12; add nvidia-cu13-* to LD_LIBRARY_PATH before serving.
  • T4 has no bf16 tensor cores; pass --dtype float16 and let vLLM cast the weights.
  • The 131072-token default KV window will OOM; --max-model-len 16384 --max-num-batched-tokens 16384 --gpu-memory-utilization 0.85 --enforce-eager bring the server up predictably.

FA2 and FlashInfer are not available on sm75 and vLLM falls back to TORCH_SDPA / TRITON_ATTN automatically — no override needed. The full traceback for each trap and the exact scripts are in the source repo; here they are just what the workflow trips over.

Once the server is up and the tunnel is out, the client is stdlib-only:

python client/ocr_client.py \
  --url https://<random>.trycloudflare.com \
  --image sample.png --task "OCR:"

Cleanup and the quota habit

Every session needs the same closing move:

colab exec -s $SESS -f scripts/14_cleanup.py --timeout 60   # kill tunnel + server, rm logs
colab stop -s $SESS                                         # release the GPU slot

The stop is the important one. Idle sessions still hold the slot, and if you burn the slot without stopping, Google is not always quick to give you a new one. Get in the habit of stopping when you walk away.

When this tool fits (and when it does not)

The CLI is well matched to:

  • Short, script-shaped GPU experiments where an editor beats a notebook.
  • One-off demos that need a public URL for an hour.
  • CI or scheduled jobs that need occasional GPU access without paying for a dedicated card.
  • Anything where the source of truth belongs in a repo, not in cells.

It is not the right tool for:

  • Persistent services — the 12-hour cap and the idle teardown are hard limits.
  • Anything sensitive on the public URL side without extra auth in front.
  • Workloads that need to hold GPU across a long-running training run — a rented card is cheaper in total.

In short: a paid spot GPU VM is gcloud compute instances create ... --provisioning-model=SPOT; the free-tier equivalent is colab new --gpu t4. Same preemptible profile, no invoice. This CLI is what turns the second one into a real dev target.

Full source, the fourteen stepwise scripts, and a reusable OCR client are at github.com/ysys143/test_colab_cli.