# Project Operating Notes

## Remote experiment hosts

- Experiment machines: `dash1`, `dash2`, `dash3`, and `dash4`.
- Do not use or probe `dash0`; it is reserved for other users.
- Hardware expectation: 8 NVIDIA H20 GPUs per host.
- Before scheduling, probe only `dash1`--`dash4` and confirm all eight GPUs are idle and healthy.
- Remote project path: `/home/admin/cpfs/wjh/aituner/aituner`.
- If remote downloads are slow or fail, start the proxy from the remote `wjh`
  home directory with `./auto_proxy.sh`, then run downloads in a shell where
  `proxyOn` from `~/.bashrc` has been applied. If `autossh` is unavailable,
  `ssh -Nf proxy` provides the same local `127.0.0.1:11235` tunnel.

## Local/remote sync workflow

- Treat this local repository and the `dash1`--`dash4` repositories as the same project checkout.
- Synchronize code through Git using `commit`, `push`, and `pull`.
- For remote experiments, commit local changes, push to `origin`, then pull on each assigned host in `/home/admin/cpfs/wjh/aituner/aituner` before running.
- Up to four independent 8-GPU experiment groups may run in parallel, one group per host; pin every job explicitly to one of `dash1`--`dash4`.
- Do not ask for the remote host or project path again unless the user explicitly changes them.