Shoal User Manual

Shoal is the sharing platform for the lab's GPU machine. You get a container of your own, your own disk space, and a fixed SSH port. Only one person's container runs at a time; who gets which hours is decided by the slot table on the booking page.

There is only one thing the administrator has to do for you: provision you with your public key. You do not have to install anything to do.

1. What to prepare before you are provisioned

1.1 Generate an SSH key that belongs only to you

On your own computer:

ssh-keygen -t ed25519 -C "yourname@lab"

Press Enter through the prompts (a passphrase is recommended but may be left empty). This produces two files:

• ~/.ssh/id_ed25519　the private key; never send it to anyone
• ~/.ssh/id_ed25519.pub　the public key; this is the one the administrator needs

Not sure what is inside the .pub file? Print it out and copy the whole line to send to the administrator:

cat ~/.ssh/id_ed25519.pub

(On Windows PowerShell, use type $env:USERPROFILE\.ssh\id_ed25519.pub instead.)

That line looks like ssh-ed25519 AAAAC3Nza... yourname@lab. Copy the whole line, including the leading ssh-ed25519 and the trailing name, with no missing characters and no extra line breaks. The next step is pasting it into the system.

You may not share a key with anyone else. The system enforces this: provisioning fails outright if the same key is already assigned to someone. A shared key means there is no separation of identity, nobody can tell who did what, and revoking one person means revoking everyone.

1.2 Sign in and paste your key

The administrator first adds your account name and Gmail to the list. Then:

1. Open the booking page and click "Sign in with Google", choosing the Gmail the administrator registered.
2. After signing in you will see "Account settings". Paste the single line you printed with cat into "Your SSH public key" and submit.
3. The status changes to "waiting for the administrator". Once provisioned, come back to this page: your port number appears and the booking area opens.

To replace your key later, paste the new one in the same place; the old one stops working immediately.

Fallback: you may also send that .pub line directly to the administrator, who provisions you by hand.

2. Connecting

After provisioning, the administrator gives you a port number. It is different for every person, and once assigned it never changes.

Put this in your ~/.ssh/config (replacing the name and port):

Host yourname-gpu
  HostName 43.213.210.65
  Port yourport
  User dev

Then:

ssh yourname-gpu

Inside, you are dev, with no sudo. The container is itself the isolation boundary, so just pip install whatever you need.

Logging in prints how much of your workspace is in use and what the GPU is doing. Type quota to see it again.

3. What survives and what does not

Location	After the container is rebuilt
/workspace	Survives
/workspace/.venv	Survives
Home directory, anything installed with apt, anywhere else in the container	All gone

Keep everything of yours under /workspace. The container may be rebuilt for maintenance, and everything outside /workspace goes back to zero when that happens.

/workspace/.venv is a Python environment created automatically on first start. It inherits torch from the image, so you do not have to reinstall several gigabytes. It is on the PATH, an interactive login activates it automatically, and non-interactive use such as ssh yourname-gpu "python train.py" picks it up too. Install anything you intend to keep into it, so that it travels with the workspace.

Disk

200G by default. There are two ceilings and whichever you hit first stops you: one counts only your files, the other counts snapshots as well.

Deleting files does not give the space back immediately. Older snapshots still pin that data until they rotate out. So "I deleted a lot and still cannot write" is expected, not a fault.

Recovering files from a snapshot

If you delete something by accident you can get it back yourself, without asking anyone:

ls /workspace/.zfs/snapshot/
cp -r /workspace/.zfs/snapshot/<some-snapshot>/the-file-you-want /workspace/

ls /workspace does not show the .zfs directory, but you can enter it by typing the full path. Snapshot names are in UTC, eight hours behind Taiwan time, so do not mistake a recent one for an old one.

4. Opening the booking page

If you are off campus and did not install Tailscale, you cannot open the booking page yourself. It only listens on the machine’s internal address, on purpose. Tell the administrator what you want to run and they will book the slot for you.

The booking page is not exposed; it only listens on an address inside the machine. The key you already have is the ticket. Use it to tunnel the page to your own computer:

ssh -N -L 8080:100.120.158.75:8080 shoal-web@100.120.158.75

That command does not return to a prompt. That is normal, it has to stay open. Leave the window alone, open a browser and go to http://localhost:8080/. Close the window when you are done and the tunnel goes with it.

This account can do nothing else: it has no shell, you cannot log in with it, and it may only forward to the booking page. Even if the key leaks, what it buys is a booking page.

If the command is too long to retype, put it in ~/.ssh/config:

Host shoal-booking
  HostName 100.120.158.75
  User shoal-web
  LocalForward 8080 100.120.158.75:8080
  RequestTTY no

Then ssh -N shoal-booking is enough.

5. Booking

The booking page is the one you came here from. The slot table is at the top and the booking form below it. Five fields to fill in: user, project, date, start time, length, plus one sentence saying what you are going to run.

Rule	Value
Length of one slot	2 hours, on fixed boundaries (00:00, 02:00, 04:00 and so on)
Most slots in one booking	12, that is 24 hours
How far ahead you may book	14 days
Minimum length of the purpose	3 characters
Usage cap	None

The absence of a usage cap is deliberate. How long you ran, which project it was on, and whether the purpose you stated matches what actually ran on the GPU are all recorded and go into the weekly report. The design makes overuse visible and open to discussion afterwards, rather than blocking you beforehand.

The "what will you run" field is printed alongside the programs that actually ran on the GPU during that slot, so make it specific ("color model finetune round 3", not "experiments").

When booking is refused

• "Pick a registered project": an empty dropdown means nobody has registered one yet. You can create one yourself in the collapsible Register-a-project box under the booking form (it opens automatically when empty); it only asks for a name and an owner.
• "Clashes with #N ...": that slot is taken. Pick another.
• "Say in one sentence what you are going to run": the purpose was shorter than 3 characters. Write something you will recognise later, not "experiment".

6. What happens when your slot arrives

Your slot starting does not mean the previous person's experiment gets killed. This is the most important rule in the system.

If your slot arrives while the previous person's GPU work is still running:

1. The system tells you the slot is being deferred, and tells them that someone is waiting and they should wrap up
2. The GPU has to stay idle for 5 consecutive minutes before their experiment counts as finished
3. Then a further 10 minutes of notice
4. Only then is their container stopped, and with docker stop -t 120, giving scripts two minutes to write a checkpoint

Someone running over is never killed automatically. After 120 minutes of overrun the system only raises an alert asking a human to step in.

The same protection applies to you: a training run of yours that overruns will not be killed by the system.

Rules that can cost you a slot

• No connection within 30 minutes of the slot starting counts as a no-show
• Nobody connected and no GPU work for 30 consecutive minutes releases the slot
• You get a reminder 15 minutes before the slot ends

7. When you are finished

Hand the slot back yourself. While your slot is running, the booking page shows a "Done, hand it over" button. Pressing it releases the slot immediately so the next person does not have to wait.

If GPU work is still running, the first press changes the button to "Press again to confirm stop", to make sure it was not a misclick. The point is that one stray click should not kill your own half-finished run.

If you are off campus and cannot open the booking page, run release inside the container to hand the slot back (add release --force if the GPU is still busy). It takes effect within about a minute; on success the container is stopped and your SSH session drops — that is the confirmation. If it cannot release (for example the GPU is still busy and you did not pass --force) it prints why. There is no full booker command inside the container, only release.

8. How the system reaches you

When the system has something to tell you it uses two channels: wall inside the container (which you see if a terminal is open), and a line appended to /workspace/.booker-notice. If you missed the live message:

tail /workspace/.booker-notice

9. When something is wrong

Three things to check yourself first:

1. Is the tunnel command still running? It has to stay open, and closing the terminal closes it
2. Does the booking page load? If it does not, the problem is the scheduler or the machine as a whole
3. If ssh yourname-gpu fails but the booking page loads, most likely your container is not running right now. Containers only start during your slot; this is by design, not a fault. Check the booking page to see whose slot it currently is

If none of those explain it, go to the administrator, and bring both the command you ran and the complete error message.

REMOTE HOST IDENTIFICATION HAS CHANGED on connecting

This should not happen under normal circumstances. The container's host key is stored outside the container and does not change when the container is rebuilt. If you really see this warning, ask the administrator before doing anything; do not just delete known_hosts to make it go away.