22 lines
1.3 KiB
Markdown
22 lines
1.3 KiB
Markdown
# Project Operating Notes
|
|
|
|
## Remote experiment hosts
|
|
|
|
- Experiment machines: `dash1`, `dash2`, `dash3`, and `dash4`.
|
|
- Do not use or probe `dash0`; it is reserved for other users.
|
|
- Hardware expectation: 8 NVIDIA H20 GPUs per host.
|
|
- Before scheduling, probe only `dash1`--`dash4` and confirm all eight GPUs are idle and healthy.
|
|
- Remote project path: `/home/admin/cpfs/wjh/aituner/aituner`.
|
|
- If remote downloads are slow or fail, start the proxy from the remote `wjh`
|
|
home directory with `./auto_proxy.sh`, then run downloads in a shell where
|
|
`proxyOn` from `~/.bashrc` has been applied. If `autossh` is unavailable,
|
|
`ssh -Nf proxy` provides the same local `127.0.0.1:11235` tunnel.
|
|
|
|
## Local/remote sync workflow
|
|
|
|
- Treat this local repository and the `dash1`--`dash4` repositories as the same project checkout.
|
|
- Synchronize code through Git using `commit`, `push`, and `pull`.
|
|
- For remote experiments, commit local changes, push to `origin`, then pull on each assigned host in `/home/admin/cpfs/wjh/aituner/aituner` before running.
|
|
- Up to four independent 8-GPU experiment groups may run in parallel, one group per host; pin every job explicitly to one of `dash1`--`dash4`.
|
|
- Do not ask for the remote host or project path again unless the user explicitly changes them.
|