--- ai_assisted: true title: "Driving a Free Colab GPU from the CLI" description: "google-colab-cli drives a free Colab GPU from your laptop. Worked example: serving PaddleOCR-VL on a T4 and running OCR on an invoice over a public URL." pubDate: 2026-07-02 category: "AI" subcategory: "Infra" tags: ["Colab", "google-colab-cli", "GPU", "vLLM", "cloudflared", "T4"] serial: 33 draft: false published: true --- Colab is interactive at its core: paste a cell, run it, look at the output, move to the next. Recently, [`google-colab-cli`](https://github.com/google-colab/google-colab-cli) shipped. What it does: it lets you drive that same interactive kernel from a local shell instead. `colab new` grabs a session, `colab exec` runs scripts against it, `colab stop` gives it back. Think of it as a **free-tier GCE spot instance you drive from your shell** — no card on file, no invoice, no GCP console. Personally I've been waiting for exactly this. The tradeoff mirrors a real spot VM: cheap-or-free, but preemptible — quota-capped, torn down on idle, hard-capped at 12 hours. Colab's awkward parts — the inbound firewall, quotas, session volatility — do not go away, but the CLI makes them things you can address in a shell. This post walks that setup end to end through one real session. --- ## What the tool actually is `google-colab-cli` is a Python-installed client (`pip install google-colab-cli`) that authenticates against your Google account and lets you drive Colab kernels remotely. It exposes a handful of verbs — `new`, `sessions`, `exec`, `upload`, `download`, `stop` — and each one talks to the same backend that the notebook UI does. The kernel that runs is the same kernel: same GPU, same disk, same idle timeout. What changes is that your editor, your version control, and your CI can now touch that kernel without a browser in the middle. Two things make this useful beyond "notebooks without the notebook": - **Scripts live in a repo, not in cells.** You keep them in git, edit them locally, and `exec` them against the remote kernel. When the kernel dies you re-run the same file. - **The CLI composes.** `exec` returns stdout/stderr on completion. That lets a shell script chain steps, poll for status, or gate on output — the way you would with any other remote runner. ## Sessions and quotas The GPU is a slot, not a resource, and the tool makes the slot visible. ```bash colab new --gpu t4 # returns a session id like e87c7c colab sessions # list active sessions colab stop -s # release the slot ``` Two things worth remembering: - **`TooManyAssignmentsError` (HTTP 412) is a quota, not a bug.** Google's assigner refuses a second GPU slot when you already hold one. Idle sessions still hold the slot — `sleep 1000` from an hour ago is enough to fail the next `new`. Fix: `sessions` to see what is alive, `stop` on the leftover, retry. - **Sessions are volatile.** Kernels get torn down after roughly 90 minutes of idle and hard-capped around 12 hours total. Anything you want to survive that has to leave the kernel — pushed to a repo, uploaded elsewhere, or captured through a tunnel. That volatility is the reason for the workflow: keep the source of truth in your repo, treat the kernel as ephemeral compute. ## Execution model Almost every step in a session takes this shape: ```bash colab exec -s -f scripts/NN_step.py --timeout N ``` The script runs **inside** the VM. Its `/content` is the VM's `/content`, its `localhost:8000` is the VM's localhost. That distinction is easy to miss when you first switch from a laptop workflow — a path or a port in the script only makes sense from the kernel's point of view, not from yours. Two consequences show up in almost every non-trivial run: - **Background processes need a nohup + log-file pattern.** `exec` returns when the script returns. If you `Popen` a server and let the script exit, the server keeps running in the kernel; if you want to see its output you have to have written it to a log file the next `exec` can `tail`. This is what real Colab CLI setups look like — install/serve happen in one exec, poll happens in the next. - **Environment does not leak backwards.** Setting `os.environ["X"] = "y"` inside a script only affects that script. If a later step needs the variable, set it in the shell prefix of that step's command, not in a previous step. Same with `LD_LIBRARY_PATH`, `PYTHONPATH`, anything. ## Files in and out The Colab VM sits behind Google's edge network with no public IP and inbound filtered. `scp`, `ssh`, opening a port with `iptables` — none of these reach the VM, because the block is upstream. The CLI's answer is a pair of verbs that ride the same authenticated tunnel as `exec`: ```bash colab upload -s local.png /content/local.png colab download -s /content/out.json ./out.json ``` That covers files. What it does not cover is arbitrary TCP — if a script on the kernel needs to serve a socket to the outside world, you need an **outbound-initiated** tunnel that the VM dials out on. cloudflared Quick Tunnel is the shortest working option (see below); ngrok and Tailscale funnel are equivalents. ## Exposing a server: outbound tunnels The reason for the pattern is worth stating cleanly: - The VM cannot accept a new inbound connection. - The VM can freely make outbound connections. - Both `cloudflared`, `ngrok`, `tailscale funnel` open a persistent outbound connection to their edge and accept public HTTPS requests on that edge, then forward down the tunnel. - So requests reach the VM through a return-path that was opened from the inside. cloudflared's Quick Tunnel is zero-config: ```bash curl -sL https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64 \ -o /content/cloudflared && chmod +x /content/cloudflared nohup /content/cloudflared tunnel --url http://localhost:8000 --no-autoupdate \ >> /content/cf_tunnel.log 2>&1 & ``` After a few seconds the log carries `https://.trycloudflare.com`. The URL is public and unauthenticated — Cloudflare will forward anyone's HTTPS request through the tunnel. If the demo lives longer than a demo, add an API key at the server (`vllm serve ... --api-key `, or the equivalent for whatever you are serving). If you want a stable hostname, use a Cloudflare account + owned domain (free tier) via a Named Tunnel. ## In practice: serving a 1B VLM on T4 The exercise that shook the workflow out: serve [PaddleOCR-VL](https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.6) (~1B, ERNIE-4.5 + NaViT) on a free T4, and OCR an invoice image over the public URL from my laptop. The CLI part is uneventful once the pattern above is clear: ```bash colab exec -s $SESS -f scripts/03_start_install.py --timeout 60 # pip install vllm colab exec -s $SESS -f scripts/04_poll_install.py --timeout 200 # tail install log colab exec -s $SESS -f scripts/08_start_server.py --timeout 60 # nohup vllm serve colab exec -s $SESS -f scripts/09_poll_server.py --timeout 120 # /v1/models poll colab exec -s $SESS -f scripts/10_test_ocr.py --timeout 180 # local OCR round-trip colab exec -s $SESS -f scripts/11_start_tunnel.py --timeout 200 # cloudflared colab exec -s $SESS -f scripts/12_get_url.py --timeout 60 # extract trycloudflare URL ``` There are three T4-specific traps that any real serving job on Colab has to clear, and they are not CLI problems — they are what you find once your script actually runs on a Turing card with vLLM's stable-libtorch ABI: - vLLM 0.24.0 links against CUDA 13 while torch keeps CUDA 12; add `nvidia-cu13-*` to `LD_LIBRARY_PATH` before serving. - T4 has no bf16 tensor cores; pass `--dtype float16` and let vLLM cast the weights. - The 131072-token default KV window will OOM; `--max-model-len 16384 --max-num-batched-tokens 16384 --gpu-memory-utilization 0.85 --enforce-eager` bring the server up predictably. FA2 and FlashInfer are not available on sm75 and vLLM falls back to `TORCH_SDPA` / `TRITON_ATTN` automatically — no override needed. The full traceback for each trap and the exact scripts are in the source repo; here they are just what the workflow trips over. Once the server is up and the tunnel is out, the client is stdlib-only: ```bash python client/ocr_client.py \ --url https://.trycloudflare.com \ --image sample.png --task "OCR:" ``` ## Cleanup and the quota habit Every session needs the same closing move: ```bash colab exec -s $SESS -f scripts/14_cleanup.py --timeout 60 # kill tunnel + server, rm logs colab stop -s $SESS # release the GPU slot ``` The `stop` is the important one. Idle sessions still hold the slot, and if you burn the slot without stopping, Google is not always quick to give you a new one. Get in the habit of stopping when you walk away. ## When this tool fits (and when it does not) The CLI is well matched to: - Short, script-shaped GPU experiments where an editor beats a notebook. - One-off demos that need a public URL for an hour. - CI or scheduled jobs that need occasional GPU access without paying for a dedicated card. - Anything where the source of truth belongs in a repo, not in cells. It is not the right tool for: - Persistent services — the 12-hour cap and the idle teardown are hard limits. - Anything sensitive on the public URL side without extra auth in front. - Workloads that need to hold GPU across a long-running training run — a rented card is cheaper in total. In short: a paid spot GPU VM is `gcloud compute instances create ... --provisioning-model=SPOT`; the free-tier equivalent is `colab new --gpu t4`. Same preemptible profile, no invoice. This CLI is what turns the second one into a real dev target. Full source, the fourteen stepwise scripts, and a reusable OCR client are at [github.com/ysys143/test_colab_cli](https://github.com/ysys143/test_colab_cli).