Description
The REmotely-managed Power Aware Computing Systems and Services (REPACSS) resource is a high-performance computing (HPC) cluster supported by multiple forms of energy developed to support research into advanced data center control for running scalable scientific workflows and data-intensive research in remotely managed settings. The focus of the project is on improvements to data center and infrastructure control to provide adaptability to emergent conditions and ability to adjust workloads to match data center load conditions including the availability and cost of electrical power. The GPU nodes feature dual-socket Intel Xeon Gold 6448Y processors, 512GB RAM, and 4 H100 GPUs connected as two H100-NVL pairs per node. The GPU nodes are interconnected with the rest of the cluster and with storage by NVIDIA ConnectX-7 network NDR Infiniband adapters running at 200 Gbps per card with two Infiniband cards per node. The Hammerspace storage provides nearly 3PB of combined NVMe and HDD storage, supporting large-scale data throughput. All nodes are also controlled and provisioned through high-bandwidth Dell PowerSwitch S5248-ON and S5232-ON Ethernet switches at 25 Gbps per node. The cluster supports intelligent workload placement and adaptive scheduling tools to align computational activity with the goal to match as much of the workload as possible to low-cost energy availability. REPACSS also features advanced remote management capabilities and automation tools to manage scientific workflows that are specifically targeted to be adopted at scale by other resource facilities and industry.
RP Description
REPACSS GPU is the GPU partition of REPACSS, a compute cluster with 8 nodes, each with 4 NVIDIA H100 NVL GPUs (94 GB), dual Intel Xeon Gold 6448Y CPUs, and 512 GB RAM. The cluster is power-aware and runs on variable energy. It is well suited to deep-learning training and inference and other GPU-offloaded code.
Jobs Information
You can run jobs at different sizes and durations on REPACSS GPU. The following lists the different queues that you can submit to, describing how many nodes you get, how long you can run, the type of resources you get, and the average wait time.
Jobs run through Slurm; the login node is for editing, compiling, and staging only. GPU jobs go to the h100 partition and must request GPUs explicitly, e.g. sbatch -p h100 --gres=gpu:nvidia_h100_nvl:2 job.sh or interactive -p h100 --gres=… for an interactive session (REPACSS recommends the interactive wrapper over salloc). Load CUDA only after you land on a GPU node; module load cuda fails on the login node. Max wall time isn't published; run sinfo to check current limits.
For more detail, please visit the [Job Basics] page.