# YAAD compute pool: guide for AI agents

YAAD pools CSI's NVIDIA DGX Sparks into one scheduler. Each Spark has a GB10 GPU, 20 cores and about 119 GB of unified memory.

You do not pick a machine. You ask for capability: memory, CPUs, GPU and a time limit. YAAD then picks the Spark:
- It runs your job on the team's home Spark if that Spark has room.
- Otherwise it runs the job on any other Spark that fits, in parallel with other teams.
- If nothing fits right now, the job waits in the queue and starts by itself when capacity frees up.

You act as the signed-in user. Everything you see and change belongs to that user's team. Other teams are never visible to you.

## Setup: instructions for the AI agent
If your user asked you to set them up for YAAD, follow these steps in order. Steps marked **HUMAN STEP** need your user.

Rules:
- Do every step you can yourself. At a HUMAN STEP, stop and tell the user exactly what to open, click or type. Wait until they say it's done, then check it worked before you go on.
- Tell the user what each command does before you run it. Ask before you install software or change settings on their computer.
- Never ask for or show the private SSH key (the file without `.pub`). Don't repeat a `yaad_pat_` token back in chat, and don't save it in a project folder or a shared file.
- Anything that needs administrator rights on the user's own computer is a HUMAN STEP: PowerShell opened with "Run as administrator", or `sudo` on Mac/Linux.
- If something fails and you can't fix it, say so. Once step 5 is done, use `report_issue` and then `wait_for_answer`.

1. **Check what's already done.** If you already have YAAD tools, call `whoami`. If it shows an `active` team, skip to step 6. Otherwise find out the user's OS and which agent you are.
2. **SSH key.** Check for `~/.ssh/id_ed25519.pub` (Windows: `%USERPROFILE%\.ssh\id_ed25519.pub`) and that `ssh -V` works. If there's no key, ask: "Do you want a passphrase on your SSH key? (safer; you'll unlock it once per session)".
   - No passphrase: run `ssh-keygen -t ed25519 -N "" -f ~/.ssh/id_ed25519 -C "<their UWM email>"` (Windows PowerShell: `ssh-keygen -t ed25519 -N '""' -f $env:USERPROFILE\.ssh\id_ed25519`).
   - With a passphrase (HUMAN STEP): they run `ssh-keygen -t ed25519` themselves, press Enter for the file, and type the passphrase twice.

   Then show the public key (one line starting with `ssh-ed25519`) and copy it to the clipboard (Windows: `Get-Content $env:USERPROFILE\.ssh\id_ed25519.pub | Set-Clipboard`; Mac: `pbcopy < ~/.ssh/id_ed25519.pub`).
3. **HUMAN STEP: sign in and join a team.** Tell the user: "Open https://www.yaadops.com and sign in with your @uwm.edu email (only UWM addresses work) and the code you get by email. Then either create a team (team name, paste your SSH key, Create team) or join one (your teammate's 8-character team code, paste your SSH key, Join team). Wait until the status says Ready (about a minute), then tell me."
4. **HUMAN STEP: create an agent token.** Tell the user: "On the same page, open Agent access, type a name and click Create token. The page shows your token once and a ready-made setup command." Then they choose:
   - (a) They run the setup command shown on the tab for your agent in a terminal. You never see the token.
   - (b) They save it as an environment variable. Windows: `[Environment]::SetEnvironmentVariable('YAAD_TOKEN','<token>','User')`, then open a new terminal. Mac/Linux: add `export YAAD_TOKEN=<token>` to `~/.zshrc` or `~/.bashrc`.
5. **Connect YAAD** (skip if they did 4a). It's MCP over Streamable HTTP: URL `https://www.yaadops.com/api/mcp`, header `Authorization: Bearer $YAAD_TOKEN`.
   - Claude Code: `claude mcp add --transport http yaad https://www.yaadops.com/api/mcp --header "Authorization: Bearer $YAAD_TOKEN"` (PowerShell: `$env:YAAD_TOKEN`). Then HUMAN STEP: "Type /mcp and reconnect yaad, or restart Claude Code, so I can see the YAAD tools."
   - Codex CLI: in `~/.codex/config.toml` add `[mcp_servers.yaad]` with `url = "https://www.yaadops.com/api/mcp"` and `bearer_token_env_var = "YAAD_TOKEN"`; the user restarts Codex.
   - OpenAI Responses API / Agents SDK: see "Connect" below and read the token from `YAAD_TOKEN`.
   - No MCP: use the HTTP API below with the same bearer token.
6. **Check it.** Call `whoami` and `list_resources`. Tell the user their team, home Spark and SSH login line.
7. **Check SSH.** Run `ssh -o BatchMode=yes -o StrictHostKeyChecking=accept-new -o ConnectTimeout=10 <login> yaad-diag`.
   - It times out → HUMAN STEP: "Connect the CSI VPN in FortiClient (unless you're plugged into the CSI network), then tell me." Campus Wi-Fi (eduroam) and home networks need the VPN.
   - "Permission denied (publickey)" with a passphrase key → unlock it once. On Windows that is a HUMAN STEP in an administrator PowerShell: `Get-Service ssh-agent | Set-Service -StartupType Automatic; Start-Service ssh-agent`. Then the user runs `ssh-add $env:USERPROFILE\.ssh\id_ed25519` in a normal terminal. Mac: `ssh-add --apple-use-keychain ~/.ssh/id_ed25519`.
   - "Permission denied" without a passphrase → the key on the website doesn't match this computer. The user pastes this computer's public key under "Replace my SSH keys", waits a minute, and you try again.
8. **Done.** Summarise: what's set up, the login line, and what they can ask next (e.g. "scope a LoRA fine-tune of an 8B model", "what's free on YAAD?", "run train.py from my project folder and bring back the results"). Student guide: https://www.yaadops.com/guide

## Connect (MCP, recommended)
1. The user creates a personal token on the website, under **Agent access**. It looks like `yaad_pat_…` and expires after 14 days.
2. The user adds the MCP server to their agent:
   ```bash
   claude mcp add --transport http yaad https://<this-site>/api/mcp --header "Authorization: Bearer yaad_pat_..."
   ```

It is a standard MCP server over Streamable HTTP: `https://www.yaadops.com/api/mcp` with header `Authorization: Bearer <token>`.

**OpenAI Responses API** (hosted MCP tool):
```python
client.responses.create(model="<model>", input="...", tools=[{"type": "mcp", "server_label": "yaad",
    "server_url": "https://www.yaadops.com/api/mcp", "headers": {"Authorization": "Bearer yaad_pat_..."},
    "require_approval": "never"}])
```

**OpenAI Agents SDK:** `MCPServerStreamableHttp(params={"url": ".../api/mcp", "headers": {"Authorization": "Bearer ..."}})`, passed in `Agent(mcp_servers=[...])`.

**Codex CLI** (`~/.codex/config.toml`): `[mcp_servers.yaad]` with `url = "https://www.yaadops.com/api/mcp"` and `bearer_token_env_var = "YAAD_TOKEN"`.

If an agent needs higher permission on the user's own computer (installing software, starting ssh-agent), have the user run its terminal as administrator (Windows) or with sudo (Mac/Linux) for that step only.

The server exposes these tools:

| Tool | What it does |
|---|---|
| `scope_job` | **Call first.** From the task (inference, LoRA, full fine-tune, Isaac Sim SDG, Isaac Lab RL, ROS, ...), model size and precision: estimated memory/CPUs/time, whether it fits one Spark (and what to change if not), a ready `submit_job` request, and whether it would start now or queue |
| `list_resources` | Free memory, CPUs and job slots per Spark; queue length; per-job and per-team limits |
| `submit_job` | `{command, memory_gb=16, cpus=4, gpu=true, max_minutes=60, input_path, name, image, env, nodes=1}` → a job id. `nodes: 2` runs one rank on each of two Sparks joined by a 200G ConnectX-7 cable (per-Spark memory/CPUs; RANK, WORLD_SIZE, MASTER_ADDR, MASTER_PORT are set; rank 1's results in `results/<job>/rank1/`). `image` is optional (e.g. `nvcr.io/nvidia/pytorch:<arm64-tag>`); the Sparks are ARM64. `env` sets container variables, e.g. `{"ACCEPT_EULA":"Y","PRIVACY_CONSENT":"Y"}` for Isaac Sim |
| `get_job` | Status, queue position, exit code, last output lines, where the results are |
| `list_jobs` | The team's recent jobs |
| `cancel_job` | Cancel a queued job, or stop a running one |
| `fetch_results` | The `scp` command that copies results from the Spark to the user's computer |
| `open_session` | The SSH login for interactive work, as root in the team's workspace container |
| `report_issue` | Ask the CSI host for help: title + the exact error, logs and `yaad-diag` output. Team, Spark and recent jobs are attached automatically |
| `my_issues` | Your tickets and the host's answers (`open`, `answered`, `waiting_admin` = needs operator approval, `closed`). `new_answers` counts replies you haven't seen |
| `wait_for_answer` | `{ticket_id}`: waits up to ~50 s for the host's reply and returns it. Call it again until `answered=true`, then **tell the user** |
| `whoami` | The user, their team and its home Spark |

## How a job runs
- `command` is a bash script. It runs as **root** in a fresh Ubuntu 24.04 + CUDA container with the GPU, using the memory and CPUs you asked for. Those limits are enforced.
- The container has outbound internet. Install what you need inside the command, for example:
  ```bash
  apt-get update -qq && apt-get install -y -qq python3-pip
  pip install --break-system-packages torch
  ```
- **Inputs:** `input_path` names a folder in the team's `/workspace`. It is copied to `/job/in`, which is the working directory. Put code and data there first, over SSH or scp.
- **Outputs:** write to `/job/out`. When the job ends, `out/` and `job.log` are delivered to the team workspace at `results/<job_id>/`.
- **Status flow:** `queued` → `staging` → `running`, then one of `succeeded`, `failed`, `cancelled` or `timed_out`. Poll `get_job` every 10–30 s.
- **Getting results:** `fetch_results` gives a command such as `scp -r yt-team@<spark>:results/<job_id> .`. Run it on the user's computer, which holds their SSH key. Connect the CSI VPN (FortiClient) first unless on the CSI network.

## HTTP API (same thing without MCP)
Send `Authorization: Bearer yaad_pat_…` on every request.

```bash
curl -s https://<site>/api/yaad/v1/pool -H "authorization: Bearer $YAAD_TOKEN"                      # capacity + jobs
curl -s -X POST https://<site>/api/yaad/v1/pool -H "authorization: Bearer $YAAD_TOKEN" -H "content-type: application/json" \
  -d '{"action":"submit","command":"nvidia-smi > /job/out/gpu.txt","memory_gb":8,"cpus":2}'
curl -s "https://<site>/api/yaad/v1/pool?job=<job_id>" -H "authorization: Bearer $YAAD_TOKEN"        # one job
curl -s -X POST https://<site>/api/yaad/v1/pool -H "authorization: Bearer $YAAD_TOKEN" -H "content-type: application/json" \
  -d '{"action":"cancel","job_id":"<job_id>"}'
```

## Errors
| Code | Meaning |
|---|---|
| `401` | Token missing, expired or revoked |
| `403 not_allowlisted` | The account is not part of the event |
| `409 no_team` | The user must create or join a team on the website first |
| `409 team_not_ready` | The team workspace is still being set up |
| `400 over_limits` | The request is bigger than the pool limits; check `list_resources` |
| `429 team_queue_full` | The team has too many jobs waiting |
| `503 pool_disabled` | The operator has not opened the pool |

## The original fixed capability
`gpu.smoke.v1`, the operator-approved GPU visibility test, is still available. Send `POST /api/yaad/v1/jobs` with exactly `{"capability":"gpu.smoke.v1"}`.

Workload how-tos (Cosmos, TAO, Isaac Sim/Lab, ROS 2, Jupyter, ports): [/guide](/guide).

## When something breaks
1. Try the fix you can do yourself (logs, `yaad-diag` in the workspace, the guide).
2. If you are stuck, call `report_issue` with the exact error text and context. The host answers automatically, usually within a few minutes. It can run diagnostics inside your team's workspace and restart it.
3. Call `wait_for_answer` with the ticket id until it returns the answer, then tell the user what it says. If the status is `waiting_admin`, the fix needs the operator's approval; keep waiting.
4. A failed pool job opens a ticket by itself. Check `my_issues` for `new_answers` at the start of each session.