Project Operating Notes

Remote experiment hosts

Experiment machines: dash1, dash2, dash3, and dash4.
Do not use or probe dash0; it is reserved for other users.
Hardware expectation: 8 NVIDIA H20 GPUs per host.
Before scheduling, probe only dash1--dash4 and confirm all eight GPUs are idle and healthy.
Remote project path: /home/admin/cpfs/wjh/aituner/aituner.
If remote downloads are slow or fail, start the proxy from the remote wjh home directory with ./auto_proxy.sh, then run downloads in a shell where proxyOn from ~/.bashrc has been applied. If autossh is unavailable, ssh -Nf proxy provides the same local 127.0.0.1:11235 tunnel.

Treat this local repository and the dash1--dash4 repositories as the same project checkout.
Synchronize code through Git using commit, push, and pull.
For remote experiments, commit local changes, push to origin, then pull on each assigned host in /home/admin/cpfs/wjh/aituner/aituner before running.
Up to four independent 8-GPU experiment groups may run in parallel, one group per host; pin every job explicitly to one of dash1--dash4.
Do not ask for the remote host or project path again unless the user explicitly changes them.