YAAD GPU LAB

Agent guide

Your AI agent talks to YAAD for you. It asks for what the job needs (memory, CPUs, GPU, time) and YAAD decides where it runs: your team's Spark first, otherwise any of the 8 Sparks with room. If nothing has room, the job queues and starts by itself. The agent acts only as you and your team.

scope_job→submit_job→get_job→fetch_results·stuck? report_issue → wait_for_answer
Fastest way to start: paste this to your AI agent (Claude Code, Codex, ChatGPT with tools, …): 
Set me up for YAAD by following https://www.yaadops.com/agents#setup
It does every step it can by itself and stops to tell you exactly what to click whenever a step needs you.

0Agent-led setup: instructions for the AI agent

If you are an AI agent and your user asked you to set them up for YAAD, follow these steps in order. A plain-text copy is at /agents.md. Steps marked HUMAN STEP need your user.

Rules for the agent
  • Do every step you can yourself. At a HUMAN STEP, stop and tell the user exactly what to open, click or type (use the wording on this page), wait until they say it's done, then check it worked before you go on.
  • Tell the user what each command does before you run it. Ask before you install software or change settings on their computer.
  • Never ask for or show the private SSH key (the file without .pub). Don't repeat a yaad_pat_ token back in chat, and don't save it in a project folder or a shared file.
  • A step that needs administrator rights on the user's own computer (e.g. starting the Windows ssh-agent service) is a HUMAN STEP: ask them to open PowerShell with Run as administrator, or to run it with sudo on Mac/Linux.
  • If something fails and you can't fix it, say so, and once step 5 is done use report_issue and wait_for_answer.
  1. Check what's already done. If you already have YAAD tools (e.g. whoami), call whoami. If it shows a team that is active, skip to step 6. Otherwise find out the user's OS and which agent you are (Claude Code, Codex, OpenAI SDK, other).
  2. SSH key on this computer. Check for ~/.ssh/id_ed25519.pub (Windows: %USERPROFILE%\.ssh\id_ed25519.pub) and that ssh -V works. If there's no key, ask the user: "Do you want a passphrase on your SSH key? (safer; you'll unlock it once per session)".
    • No passphrase: you run ssh-keygen -t ed25519 -N "" -f ~/.ssh/id_ed25519 -C "<their UWM email>" (Windows PowerShell: ssh-keygen -t ed25519 -N '""' -f $env:USERPROFILE\.ssh\id_ed25519).
    • With a passphrase (HUMAN STEP): they run ssh-keygen -t ed25519 themselves, press Enter for the file, and type the passphrase twice.
    Then show the public key (the one line starting with ssh-ed25519) and copy it to the clipboard if you can (Windows: Get-Content $env:USERPROFILE\.ssh\id_ed25519.pub | Set-Clipboard, Mac: pbcopy < ~/.ssh/id_ed25519.pub).
  3. HUMAN STEP: sign in and join a team. Tell the user: "Open https://www.yaadops.com and sign in with your @uwm.edu email (only UWM addresses work) and the code you get by email. Then either create a team (type a team name, paste your SSH key, click Create team) or join one (type your teammate's 8-character team code, paste your SSH key, click Join team). Wait until the status says Ready (about a minute), then tell me."
  4. HUMAN STEP: create an agent token. Tell the user: "On the same page, open Agent access, type a name (e.g. 'my laptop agent') and click Create token. The page shows your token once and a ready-made setup command for your agent." Then let them choose one of these:
    • (a) They run the setup command themselves. They copy it from the tab for your agent (e.g. Claude Code) and paste it into a terminal. You never see the token.
    • (b) They save it as an environment variable and you do the rest. Windows: [Environment]::SetEnvironmentVariable('YAAD_TOKEN','<token>','User'), then open a new terminal. Mac/Linux: add export YAAD_TOKEN=<token> to ~/.zshrc or ~/.bashrc.
  5. Connect YAAD (skip if they did 4a). MCP over Streamable HTTP: URL https://www.yaadops.com/api/mcp, header Authorization: Bearer $YAAD_TOKEN.
    • Claude Code: claude mcp add --transport http yaad https://www.yaadops.com/api/mcp --header "Authorization: Bearer $YAAD_TOKEN" (PowerShell: $env:YAAD_TOKEN). Then a HUMAN STEP: "Type /mcp and reconnect yaad, or restart Claude Code, so I can see the YAAD tools."
    • Codex CLI: add [mcp_servers.yaad] with url = "https://www.yaadops.com/api/mcp" and bearer_token_env_var = "YAAD_TOKEN" to ~/.codex/config.toml, then the user restarts Codex.
    • OpenAI Responses API / Agents SDK: use the code in section 1 with the token read from YAAD_TOKEN.
    • No MCP support: use the HTTP API with the same bearer token.
  6. Check it. Call whoami and list_resources. Tell the user their team, their home Spark and their SSH login line (e.g. ssh yt-team@172.30.102.9X).
  7. Check SSH to the Spark. Run ssh -o BatchMode=yes -o StrictHostKeyChecking=accept-new -o ConnectTimeout=10 <login> yaad-diag.
    • It times out: HUMAN STEP: "Connect the CSI VPN in FortiClient (unless you're plugged into the CSI network), then tell me." Campus Wi-Fi (eduroam) and home networks need the VPN. Then try again.
    • "Permission denied (publickey)" and the key has a passphrase: the key has to be unlocked once. Windows: a HUMAN STEP in an administrator PowerShell: Get-Service ssh-agent | Set-Service -StartupType Automatic; Start-Service ssh-agent. Then, in a normal terminal, the user runs ssh-add $env:USERPROFILE\.ssh\id_ed25519 (it asks for the passphrase). Mac: ssh-add --apple-use-keychain ~/.ssh/id_ed25519.
    • "Permission denied" without a passphrase: the key on the website doesn't match this computer. Have the user paste this computer's public key under Replace my SSH keys, wait a minute, and try again.
  8. Done. Summarise for the user: what's set up, their login line, and what they can ask next, e.g. "scope a LoRA fine-tune of an 8B model", "what's free on YAAD right now?", "run train.py from my project folder and bring back the results". Point them to the student guide for Isaac Sim, models and the compute pool.

1Connect (MCP)

  1. On the Lab page, sign in → Agent access → Create token. Copy the yaad_pat_… token. It's shown once and lasts 14 days.
  2. Add YAAD to your agent. It's a standard MCP server over Streamable HTTP: URL https://www.yaadops.com/api/mcp, header Authorization: Bearer <token>. 
    $ claude mcp add --transport http yaad https://www.yaadops.com/api/mcp --header "Authorization: Bearer yaad_pat_..."

    Check with /mcp (it lists yaad), then ask: "What GPU capacity is free on YAAD?"

  3. Check it by asking: "What GPU capacity is free on YAAD?" The agent should call list_resources.
Higher permission on your own computer: if your agent has to install software, start the ssh-agent service or change system settings, run its terminal as administrator for that step only. Windows: right-click PowerShell/Terminal → Run as administrator. Mac/Linux: run the step with sudo. Nothing on the Spark needs this: the workspace is already root.
Network: /api/mcp is reachable from anywhere. Logging in to the Spark with SSH or scp (for open_session or fetch_results) needs the CSI VPN in FortiClient, unless the computer is on the CSI network.
What a token can't do: create, join or delete teams, change SSH keys, make more tokens, or touch operator settings. Those need a website sign-in. Revoke a token any time under Agent access.

2Tools

scope_jobCall first. Give it the task (inference, LoRA, full fine-tune, Isaac Sim SDG, Isaac Lab RL, ROS, …), model size and precision. You get memory/CPUs/time, whether it fits one Spark (and what to change if not), an image hint, a ready submit_job request, and whether it would start now or queue.
list_resourcesFree memory, CPUs and job slots per Spark, queue length, per-job and per-team limits
submit_job{command, memory_gb=16, cpus=4, gpu=true, max_minutes=60, input_path, image, env, nodes=1, name} → job id. env: container variables (Isaac Sim: {"ACCEPT_EULA":"Y","PRIVACY_CONSENT":"Y"}). nodes: 2: one rank on each of two Sparks joined by a 200G cable (per-Spark memory/CPUs; RANK, WORLD_SIZE, MASTER_ADDR, MASTER_PORT set).
get_job · list_jobsStatus, queue position, exit code, the last lines of output, where the results are
cancel_jobCancels a queued job, or stops a running one
fetch_resultsThe scp command that copies results from the Spark to this computer
open_sessionThe SSH login for interactive work (root in the team workspace)
report_issue · my_issuesAsk the CSI host for help, then read the answers (new_answers = replies you haven't seen)
wait_for_answerWaits up to ~50 s for the reply to a ticket and returns it. Call again until answered is true, then tell the user.
whoamiThe user, their team and its home Spark

3A good workflow

  1. Scope: scope_job {task:"fine_tune_lora", model_params_b:8, precision:"bf16", expected_hours:2}
  2. Put code and data in the workspace (scp, or an open_session login), e.g. /workspace/project.
  3. Submit the suggested request with your command, input_path:"project" and an arm64 image.
  4. Poll get_job every 10–30 s: queued → staging → running → succeeded | failed | cancelled | timed_out.
  5. Fetch: run the fetch_results command on the user's computer (it uses their SSH key; the CSI VPN in FortiClient must be connected unless they are on the CSI network).

Prompts that work well:

"Scope and run a LoRA fine-tune of an 8B model on YAAD from my workspace folder project, then download the checkpoints."

"Generate 500 synthetic images with Isaac Sim headless on YAAD and bring the dataset back here."

"My job failed. Read the log, fix the command, resubmit. If you're stuck, report it to YAAD."

4How a job runs

WhereA fresh container, root, with the GPU and the memory/CPUs you asked for (enforced). It has outbound internet.
ImageDefault is Ubuntu 24.04 + CUDA. Or a public registry image (nvcr.io, docker.io, ghcr.io, quay.io). Must be arm64.
Inputsinput_path (a folder of /workspace) is copied in as /job/in, which is the working directory
OutputsWrite to /job/out. Afterwards, out/ and job.log are in the team workspace at results/<job_id>/.
Too big?scope_job says so and suggests fp8/fp4, LoRA/QLoRA, a smaller model or hosted options
# a typical job command
apt-get update -qq && apt-get install -y -qq python3-pip
pip install --break-system-packages -r requirements.txt
python3 train.py --data /job/in/data --out /job/out

5When something breaks

  1. Try the fix yourself first: read log_tail from get_job, check the student guide, retry with more memory_gb after an out-of-memory error.
  2. Still stuck? Call report_issue with the exact error text, what you ran, and yaad-diag output if you have a workspace session. Team, Spark and recent jobs are attached automatically.
  3. Call wait_for_answer with the ticket id until it returns the answer, then tell the user. The host answers automatically, usually within a few minutes. It can run diagnostics inside the team workspace and restart it.
  4. A status of waiting_admin means the fix needs the operator's approval. Keep waiting; don't open duplicates.
  5. A failed pool job opens a ticket by itself. At the start of a session, check my_issues for new_answers.

6HTTP API (same thing without MCP)

# capacity + your team's jobs
curl -s https://www.yaadops.com/api/yaad/v1/pool -H "authorization: Bearer $YAAD_TOKEN"
# submit
curl -s -X POST https://www.yaadops.com/api/yaad/v1/pool -H "authorization: Bearer $YAAD_TOKEN" -H "content-type: application/json" \
  -d '{"action":"submit","command":"nvidia-smi > /job/out/gpu.txt","memory_gb":8,"cpus":2}'
# one job
curl -s "https://www.yaadops.com/api/yaad/v1/pool?job=<job_id>" -H "authorization: Bearer $YAAD_TOKEN"

7Errors

401The token is missing, expired or revoked
403 not_allowlisted · token_not_allowedThe account isn't in the event · this action needs a website sign-in
409 no_team · team_not_readyThe user must create or join a team · the workspace is still being set up
400 over_limits · bad_imageThe request is bigger than the pool limits (check scope_job) · the image is not from a public registry
429 team_queue_fullToo many of the team's jobs are waiting
503 pool_disabledThe operator hasn't opened the pool