Student guide

★All YAAD tools at a glance

On the website (Lab page)

In your workspace (SSH terminal on the Spark)

On your computer

Your AI agent's tools (MCP), see the agent guide

1Before you start

2Get your SSH key

You make a key pair on your device. The private key never leaves the device. The public key (one line starting with ssh-ed25519) is what you paste on the website. Pick your device:

$ ssh-keygen -t ed25519 -C "you@uwm.edu"     # press Enter to accept the defaults
$ cat ~/.ssh/id_ed25519.pub                   # copy this one line

3Register on the website

1. Open the Lab page and sign in with your UWM email and the emailed code.
2. One teammate creates the team: type a team name, paste their key, and click Create team. The page shows an 8-character team code.
3. Everyone else joins: enter the team code, paste their own key, and click Join team. Each person uses their own key.
4. More devices? In your team panel, paste one key per line (up to 3: laptop, desktop, iPad) under Replace my SSH keys.
5. After about a minute the status shows Ready, with your login line, for example ssh yt-yourteam@172.30.102.9X.

4Connect

PS> ssh yt-yourteam@172.30.102.9X        # exact line is on the website

$ ssh yt-yourteam@172.30.102.9X

# once, in PowerShell "Run as administrator"
Get-Service ssh-agent | Set-Service -StartupType Automatic
Start-Service ssh-agent
# then in a normal PowerShell (asks for the passphrase once)
ssh-add $env:USERPROFILE\.ssh\id_ed25519

$ ssh-add --apple-use-keychain ~/.ssh/id_ed25519

$ eval "$(ssh-agent -s)" && ssh-add ~/.ssh/id_ed25519

Copy files

$ scp data.zip yt-yourteam@172.30.102.9X:          # into /workspace
$ scp -r yt-yourteam@172.30.102.9X:results/<job_id> .   # bring results home

5Your workspace

5½Long runs, coding agents, and when the website is down

Keep long runs alive when you disconnect. Closing your laptop or losing the VPN ends your SSH session and anything running in it.

$ apt install -y tmux
$ tmux new -s train            # start your run here; Ctrl-b then d to detach
$ tmux attach -t train         # come back later
$ nohup python train.py > /workspace/train.log 2>&1 &   # or this, then: tail -f /workspace/train.log

Save checkpoints to /workspace. A workspace restart stops running processes but keeps your files.

Install a coding agent in your workspace. It runs as root in your team's container and only sees your team's files. Sign in with your own account.

$ curl -fsSL https://claude.ai/install.sh | bash      # Claude Code; run: claude (it prints a sign-in link)
$ curl -fsSL https://deb.nodesource.com/setup_22.x | bash - && apt install -y nodejs && npm i -g @openai/codex   # Codex CLI

If a browser sign-in can't reach the Spark, use an API key instead (ANTHROPIC_API_KEY or OPENAI_API_KEY). Your teammates can read files in /workspace and /root, so keep keys out of shared folders.

6Web UIs: Jupyter, Gradio, Isaac Sim stream, TensorBoard

$ echo $YAAD_PORT_BASE                                   # e.g. 20200; your range is +0..+99
$ jupyter lab --no-browser --ip 0.0.0.0 --port $YAAD_PORT_BASE

Open http://<spark-ip>:<port> (CSI network or CSI VPN), or tunnel: ssh -L 8888:localhost:<port> yt-yourteam@<spark-ip> → http://localhost:8888. Keep tokens and passwords on: other teams on the same Spark can reach open ports.

6½Isaac Sim quick start

A. Batch / synthetic data (headless): a pool job

Put your script in a folder of /workspace (e.g. sdg/) and submit it from the Lab page or your agent:

{ "image": "nvcr.io/nvidia/isaac-sim:5.1.0",
  "env": { "ACCEPT_EULA": "Y", "PRIVACY_CONSENT": "Y" },
  "input_path": "sdg", "memory_gb": 48, "cpus": 8, "max_minutes": 120,
  "command": "/isaac-sim/python.sh /job/in/generate.py --out /job/out" }

The output lands in /workspace/results/<job>. Pool jobs run as root, so writing to /job/out works. Your command always runs as a bash script; the image's own startup (entrypoint) is skipped.

Tested recipes (2026-10-15, as pool jobs):

B. Interactive with the streaming client: your workspace

Run it in your workspace with docker. Use your ports: signalling on $YAAD_PORT_BASE, media on the next port.

$ docker run -d --name isaac --gpus all -u 0 --network host \
    -e ACCEPT_EULA=Y -e PRIVACY_CONSENT=Y -v /workspace:/workspace \
    nvcr.io/nvidia/isaac-sim:6.0.0 /isaac-sim/runheadless.sh \
    --/exts/omni.kit.livestream.app/primaryStream/signalPort=$YAAD_PORT_BASE \
    --/exts/omni.kit.livestream.app/primaryStream/streamPort=$((YAAD_PORT_BASE+1)) \
    --/exts/omni.kit.livestream.app/primaryStream/publicIp=<spark-ip>
$ docker logs -f isaac        # wait for "app ready" (first start: a few minutes of shader compiling)

On your computer (CSI VPN on), the Isaac Sim WebRTC Streaming Client always uses signalling port 49100. Tunnel it to your port and point the client at 127.0.0.1:

PS> ssh -N -o ServerAliveInterval=30 -L 49100:127.0.0.1:<YAAD_PORT_BASE> yt-yourteam@<spark-ip>

• Forward to 127.0.0.1. A forward to the Spark's IP is refused ("administratively prohibited").
• The video (UDP) can't go through ssh. It goes straight to <spark-ip>:<YAAD_PORT_BASE+1> over the VPN, which works.
• A passphrase key? Load it into ssh-agent first (step 4).

Files: scp from your computer works (step 4). Inside the workspace you are root, so apt-get install -y python3 openssh-client adds missing tools.

7What you can run

NGC images that need a login: docker login nvcr.io -u '$oauthtoken', then paste your NGC API key (from ngc.nvidia.com) as the password.

7½Downloading models (Hugging Face and more)

Your workspace and your pool jobs have internet access, so you can download open models from Hugging Face, NGC, GitHub, Ollama, Kaggle and similar sites. Model weights work on the ARM64 Sparks like anywhere else; only containers and Python wheels have to be ARM64.

Download once, into your workspace

$ export HF_HOME=/workspace/hf                       # Hugging Face cache that survives logins
$ pip install -U huggingface_hub
$ huggingface-cli download Qwen/Qwen2.5-7B-Instruct --local-dir /workspace/models/qwen7b
$ du -sh /workspace/models/qwen7b                     # how big it really is

• Gated models (Llama, some NVIDIA models): accept the licence on the model's page, then run huggingface-cli login with your own token.
• Pool jobs start in a fresh container, so a download inside the command happens every run. For big models, download once into /workspace/models and give the job input_path: "models/qwen7b". It's copied in as /job/in.
• A token in a job's env (HF_TOKEN) is stored with the job record. Use a read-only token there.
• Ollama, git clone, wget and docker pull work too. Everything is deleted at the end of the event, so download what you want to keep.

Will it fit? Size in memory, not on disk

The Spark's GPU and CPU share one pool of memory (GB10, about 119 GB). What has to fit is the model while it runs:

Your limits: a pool job up to 80 GB · your workspace up to 100 GB · a whole Spark ~119 GB · a 2-Spark job (nodes: 2) gives each half its own Spark, if your code splits the model (FSDP, DeepSpeed, tensor/pipeline parallel).

Let scope_job do the math

Ask your AI agent, e.g. "Can I fine-tune Llama 3.1 8B with LoRA on YAAD?". It calls scope_job with the task, the model size and the precision. You get back:

• an estimate of memory, CPUs and time (the formulas above);
• where it fits: a pool job, your workspace only, a whole Spark, or too big for one Spark, with what to change (lower precision, LoRA/QLoRA, a smaller model, nodes: 2, hosted);
• a ready submit_job request (memory, CPUs, minutes, image, env) and whether it would start now or queue.

scope_job { "task": "fine_tune_lora", "model_params_b": 8, "precision": "bf16" }
→ estimate ~34 GB, 8 CPUs · fits: pool_or_workspace · suggested_submit_job { memory_gb: 34, cpus: 8, ... } · availability_now: 6 Sparks

Estimates are rules of thumb. Ask for enough memory not to be killed for running out, but not so much that you block other teams. If a job dies with "out of memory", resubmit with a higher memory_gb.

8The compute pool

Submit from the Lab page or let your AI agent do it (agent guide). If nothing has room, the job waits and starts by itself.

Two Sparks over the direct cable

Each pair of Sparks is joined by a 200G ConnectX-7 cable (RDMA, measured ~195 Gb/s). Set nodes: 2 (or tick 2 Sparks on the Lab page) to run one rank on each end: memory_gb and cpus are per Spark. Each rank gets RANK, WORLD_SIZE=2, MASTER_ADDR, MASTER_PORT and the NCCL network settings:

{ "nodes": 2, "image": "nvcr.io/nvidia/pytorch:25.10-py3", "memory_gb": 100, "cpus": 16, "input_path": "project",
  "command": "torchrun --nnodes 2 --nproc-per-node 1 --node-rank $RANK --master-addr $MASTER_ADDR --master-port $MASTER_PORT train.py" }

Rank 0's results land in /workspace/results/<job>/, rank 1's in results/<job>/rank1/. If one rank fails, YAAD stops the other. The job waits until both ends of a pair have room (list_resources → two_spark_pairs). Use a recent image (PyTorch 25.10 or later) so NCCL knows the GB10.

9Stuck? Get help

Some fixes need the operator, for example more memory. Those show Waiting for operator approval, and you don't have to ask again.

10Not available, and what to do instead

• Desktop GUI apps on the Spark: use web UIs, the Isaac Sim livestream or Jupyter.
• --privileged, USB cameras, serial/CAN: not on shared Sparks. Use the CSI lab machines for hardware-in-the-loop.
• Other teams' files, containers or processes: not visible to you, and yours are not visible to them.
• Everything is deleted at the end of the event. Download what you want to keep.