# YAAD GPU Lab: student guide

Your team gets a private workspace on one NVIDIA DGX Spark. Each Spark has a GB10 GPU, 20 CPU cores and about 119 GB of memory that the CPU and GPU share. It also has a 3.5 TB disk and full internet access.

You can also send batch jobs to the **compute pool**. They run on any Spark that has room, in parallel with other teams.

**The Sparks are ARM64 (aarch64).** Use images and wheels built for `linux/arm64`. On NGC, look for tags marked arm64 or "DGX Spark". To check an image before you pull it:

```bash
docker manifest inspect <image> | grep arm64
```

---

## 1. Log in and look around

```bash
ssh yt-<team>@<spark-ip>        # the exact line is on the website; connect the CSI VPN (FortiClient) first
```

| You get | Details |
|---|---|
| **root** in your own Ubuntu 24.04 + CUDA container | `id` shows `uid=0(root)`. `apt install …` and `pip install …` work. `sudo …` also works, so install scripts that call sudo run unchanged. |
| **Your files** in `/workspace` | Kept between sessions until the event ends. Copy files in and out with `scp` or `sftp` (they land in `/workspace`). |
| **The GPU** | `nvidia-smi` shows the GB10. Every container you start also gets it. |
| **`docker` and `docker compose`** | These run in **your team's own** container engine, so other teams can't see your containers and you can't see theirs. `--gpus all` works. So does `-v /workspace/data:/data`. |
| **Internet** | `apt`, `pip`, Hugging Face, NGC, GitHub and build.nvidia.com APIs all work. |
| **Limits** | Up to 100 GB memory per team. CPU is shared fairly with the other teams on your Spark (usually 2–8 teams). For long or heavy runs, use the compute pool (section 4). |

Each team also gets two values set for it:
- `$ROS_DOMAIN_ID` (your own ROS 2 domain)
- `$YAAD_PORT_BASE` (100 ports reserved for your team's web services)

Use them as described below.

## 1½. Long runs, coding agents, and when the website is down

**Your workspace does not need the website.** Once your team shows Ready, `ssh` goes straight to your Spark. If the website, the compute pool or the help desk is down, keep working over SSH: run scripts, `docker run --gpus all …`, train, and copy files with `scp`. Save your `ssh` line somewhere. You only need the website to create or join a team, change keys, or use the pool.

**Keep long runs alive when you disconnect.** Closing your laptop or losing the VPN ends your SSH session and anything running in it. Use `tmux`, or `nohup`:

```bash
apt install -y tmux
tmux new -s train            # start your run here; Ctrl-b then d to detach
tmux attach -t train         # come back later
nohup python train.py > /workspace/train.log 2>&1 &     # or this, then: tail -f /workspace/train.log
```

Save checkpoints to `/workspace`. A workspace restart stops running processes but keeps your files.

**Install a coding agent in your workspace.** It runs as root in your team's container and only sees your team's files. Sign in with your own account.

```bash
curl -fsSL https://claude.ai/install.sh | bash        # Claude Code; then run: claude  (it prints a sign-in link to open on your laptop)
curl -fsSL https://deb.nodesource.com/setup_22.x | bash - && apt install -y nodejs && npm i -g @openai/codex   # Codex CLI
```

If a browser sign-in can't reach the Spark, use an API key instead (`ANTHROPIC_API_KEY` or `OPENAI_API_KEY`). Your teammates can read files in `/workspace` and `/root`, so keep keys out of shared folders.

## 2. Web UIs: JupyterLab, Gradio, Isaac Sim streaming, TensorBoard

Your workspace shares the Spark's network. Start a service on a port in **your** range: `$YAAD_PORT_BASE` to `$YAAD_PORT_BASE+99`.

```bash
echo $YAAD_PORT_BASE                       # e.g. 20200
jupyter lab --no-browser --ip 0.0.0.0 --port $YAAD_PORT_BASE      # keep Jupyter's token on
```

Then open it from your laptop in one of two ways:
- **Direct:** `http://<spark-ip>:<port>`, on the CSI network or the CSI VPN (FortiClient).
- **Through SSH** (works anywhere you can SSH):
  ```bash
  ssh -L 8888:localhost:<port> yt-<team>@<spark-ip>
  ```
  then open `http://localhost:8888`.

Other teams on the same Spark could reach an open port. Keep logins and tokens on (Jupyter does this by default).

## 2½. Isaac Sim quick start

- **arm64 images:** `nvcr.io/nvidia/isaac-sim:6.0.0` and `:5.1.0`, `nvcr.io/nvidia/isaac-lab:2.3.0`. Isaac Sim 4.5 / 5.0 and Isaac Lab 2.2 are x86 only ("exec format error").
- The image stops unless `ACCEPT_EULA=Y` and `PRIVACY_CONSENT=Y` are set. The first pull is about 20 GB.

**Batch / synthetic data (headless): a pool job.** Your script is in `/workspace/sdg/`:
```json
{ "image": "nvcr.io/nvidia/isaac-sim:5.1.0",
  "env": { "ACCEPT_EULA": "Y", "PRIVACY_CONSENT": "Y" },
  "input_path": "sdg", "memory_gb": 48, "cpus": 8, "max_minutes": 120,
  "command": "/isaac-sim/python.sh /job/in/generate.py --out /job/out" }
```

Pool jobs always run your command as a bash script (the image's own entrypoint is skipped). Tested recipes:
- **Isaac Lab RL:** `nvcr.io/nvidia/isaac-lab:2.3.0`, 56 GB / 12 CPUs, env `ACCEPT_EULA=Y PRIVACY_CONSENT=Y OMNI_KIT_ACCEPT_EULA=YES`, command `cd /workspace/isaaclab && ./isaaclab.sh -p scripts/reinforcement_learning/rsl_rl/train.py --task Isaac-Cartpole-v0 --headless --num_envs 512 --max_iterations 30 && cp -r logs /job/out/`
- **ROS 2 Jazzy:** `docker.io/library/ros:jazzy`, 8 GB / 4 CPUs (ROS is already sourced).
- **Cosmos Reason 7B:** `nvcr.io/nvidia/pytorch:25.01-py3`, 40 GB / 8 CPUs, `pip install transformers==4.51.3 qwen-vl-utils`, bf16.

**Interactive, with the WebRTC streaming client: your workspace.** Signalling on `$YAAD_PORT_BASE`, media on the next port:
```bash
docker run -d --name isaac --gpus all -u 0 --network host \
  -e ACCEPT_EULA=Y -e PRIVACY_CONSENT=Y -v /workspace:/workspace \
  nvcr.io/nvidia/isaac-sim:6.0.0 /isaac-sim/runheadless.sh \
  --/exts/omni.kit.livestream.app/primaryStream/signalPort=$YAAD_PORT_BASE \
  --/exts/omni.kit.livestream.app/primaryStream/streamPort=$((YAAD_PORT_BASE+1)) \
  --/exts/omni.kit.livestream.app/primaryStream/publicIp=<spark-ip>
```
The Isaac Sim WebRTC Streaming Client always uses signalling port 49100. With the CSI VPN on, tunnel it and point the client at `127.0.0.1`:
```bash
ssh -N -o ServerAliveInterval=30 -L 49100:127.0.0.1:<YAAD_PORT_BASE> yt-<team>@<spark-ip>
```
- Forward to `127.0.0.1`. A forward to the Spark's IP is refused ("administratively prohibited").
- Video is UDP and can't go through ssh. It goes straight to `<spark-ip>:<YAAD_PORT_BASE+1>` over the VPN.
- Stream won't reconnect after a network drop (`NVST_R_BUSY` in the log)? Run `docker restart isaac`. It clears what is loaded, so **save the stage to `/workspace` often** and reopen it.
- A workspace session holds the GPU outside the pool. `docker stop isaac` when you're done; use pool jobs for long unattended runs.

## 3. What you can run, by task

| Task (from the Physical AI task list) | How to run it on YAAD |
|---|---|
| **Agent orchestration (NemoClaw)** | Install it in your workspace (`pip` or the vendor's container). It can call build.nvidia.com APIs with your free API key, and models served on the Spark. |
| **In-context reasoning: Cosmos Reason (AOI, defect, RCA)** | Run the Cosmos Reason arm64 container or Hugging Face weights in your workspace (interactive) or as a pool job. Or call the free build.nvidia.com Reasoner API with your key. Cache models in `/workspace` (`export HF_HOME=/workspace/hf`). |
| **Automated inspection models: TAO, VisualChangeNet, DEFT AOI** | Use the TAO/PyTorch NGC containers (arm64 tags). Train as a pool job (`image` = the NGC image, inputs = your dataset folder) and run inference in your workspace. |
| **Synthetic data generation: GenAI (FLUX.2, Cosmos T2I/I2V/V2V)** | Inference fits on the GB10 (128 GB unified memory). Cosmos fine-tuning on the Spark: Edge with SFT, Nano with **LoRA only**. Bigger runs: Brev or the build.nvidia.com APIs. |
| **Synthetic data generation from a digital twin: Isaac Sim / Omniverse** | Run **headless** with the arm64 Isaac Sim container, and write images and labels to `/workspace`. To watch it, use Isaac Sim's livestream (WebRTC) on ports in your range. Coordinate with your Spark's other teams, because Isaac Sim's default streaming ports are fixed. **Emulate3D** is Windows desktop software, so use the CSI GPU workstation for it. |
| **Robot policy training: Isaac Lab / Isaac Lab Arena** | Use the Isaac Lab container headless. Long training runs go to the compute pool with `image` set to the Isaac Lab image. Logs and checkpoints go to `/job/out`. |
| **ROS 2** | `docker run --rm -it --network host -e ROS_DOMAIN_ID ros:jazzy` (arm64 images exist). Your `ROS_DOMAIN_ID` keeps your topics separate from other teams on the same Spark. ROS discovery does not cross the campus network, so to talk to a robot or laptop elsewhere use a ROS bridge (rosbridge / Zenoh) on a port in your range. |
| **3D reconstruction: NuRec / Omniverse** | Run from the NVIDIA containers or scripts in your workspace. For large scenes, Brev is the documented option. |
| **Time-series forecasting: NVIDIA Kumo-TS** | Use Python in your workspace, or a pool job for training. |

NGC images that need a login:
```bash
docker login nvcr.io -u '$oauthtoken'
```
and paste your free NGC API key (from ngc.nvidia.com) as the password. Never put keys in shared files.

## 3½. Downloading models (Hugging Face and more)

Your workspace and your pool jobs have internet access: Hugging Face, NGC, GitHub, Ollama and Kaggle all work. Model weights work on the ARM64 Sparks like anywhere else; only containers and Python wheels have to be ARM64.

```bash
export HF_HOME=/workspace/hf                 # Hugging Face cache that survives logins
pip install -U huggingface_hub
huggingface-cli download Qwen/Qwen2.5-7B-Instruct --local-dir /workspace/models/qwen7b
```
- Gated models (Llama, some NVIDIA): accept the licence on the model page, then `huggingface-cli login` with your own token.
- Pool jobs start fresh every run: download once into `/workspace/models` and use `input_path: "models/qwen7b"`. A token in a job's `env` (`HF_TOKEN`) is stored with the job, so use a read-only one.
- Everything is deleted at the end of the event.

**Will it fit?** What counts is the memory while it runs (GB10: ~119 GB shared by GPU and CPU):
- Weights = billions of parameters × bytes: bf16 = 2, fp8 = 1, 4-bit ≈ 0.5 (8B: ~16 / ~8 / ~4–5 GB).
- Inference ≈ weights × 1.2 + 6 GB (more for contexts over 32k). LoRA ≈ weights × 1.25 + 0.4 GB per billion + 10 GB. Full fine-tuning ≈ 16 bytes per parameter (8B → ~140 GB: use LoRA).
- Limits: pool job 80 GB, workspace 100 GB, whole Spark ~119 GB, `nodes: 2` = one Spark per half (needs FSDP/DeepSpeed/tensor-parallel).

| Running a model | bf16 | fp8 | 4-bit |
|---|---|---|---|
| 7–8B | ~26 GB | ~16 GB | ~11 GB |
| 13–14B | ~40 GB | ~23 GB | ~15 GB |
| 32B | ~83 GB (workspace) | ~45 GB | ~26 GB |
| 70B | too big | ~90 GB (workspace) | ~48 GB |
| 100B+ | 4-bit, 2 Sparks, or hosted (build.nvidia.com) | | |

**scope_job does this for you.** Your agent calls it with the task, model size and precision, e.g. `{"task":"fine_tune_lora","model_params_b":8,"precision":"bf16"}`. It returns the memory/CPU/time estimate, where it fits (pool, workspace only, whole Spark, too big) and what to change if not. It also gives a ready `submit_job` request and whether the job would start now or queue. These are estimates: if a job runs out of memory, resubmit with more `memory_gb`.

## 4. The compute pool (batch jobs, any Spark, auto-queued)

Use the website (**Compute pool → Submit a job**) or your own AI agent over MCP (website → **Agent access** gives you a token and the setup line). Your job runs as root in a fresh container with the GPU:

| Field | Meaning |
|---|---|
| `command` | A bash script. It runs in `/job/in`; write results to `/job/out`. |
| `image` | Optional, e.g. `nvcr.io/nvidia/pytorch:<arm64-tag>`. Blank means Ubuntu 24.04 + CUDA. The first use of a big image takes minutes while it downloads. |
| `nodes` | `1` (default) or `2`: one rank on each of two Sparks joined by a 200G ConnectX-7 cable. `memory_gb`/`cpus` are per Spark. Each rank gets `RANK`, `WORLD_SIZE=2`, `MASTER_ADDR`, `MASTER_PORT`; run e.g. `torchrun --nnodes 2 --nproc-per-node 1 --node-rank $RANK --master-addr $MASTER_ADDR --master-port $MASTER_PORT train.py` with `nvcr.io/nvidia/pytorch:25.10-py3`. Rank 1's results land in `results/<job>/rank1/`. |
| `env` | Optional environment variables, e.g. `{"ACCEPT_EULA":"Y","PRIVACY_CONSENT":"Y"}` for Isaac Sim. Up to 20. |
| `input_path` | A folder in your `/workspace`, copied in as `/job/in`. |
| `memory_gb`, `cpus`, `max_minutes` | Reserved for you and enforced. |

If no Spark has room, the job waits in the queue and starts by itself when one frees up. When it ends, `out/` and `job.log` appear in `/workspace/results/<job_id>/`. Download them with:
```bash
scp -r yt-<team>@<spark-ip>:results/<job_id> .
```

## 5. What is not available (and what to do instead)

- **Desktop GUI apps on the Spark** (no monitor/X session for teams). Use web UIs, Isaac Sim livestream, or Jupyter.
- **`docker run --privileged` or host devices** (USB cameras, CAN, serial) are not available inside the shared Spark. For hardware-in-the-loop, use the CSI lab machines.
- **Another team's files, containers or processes** are not visible to you, and yours are not visible to them.
- Everything is **deleted at the end of the event**. Download what you want to keep.