The Delta GPU resource comprises 5 different node configurations intended to support accelerated computation across a broad range of domains including traditional simulation and AI/ML work. Delta is designed to support the transition of applications from CPU-only to using the GPU or hybrid CPU-GPU models. Delta GPU resource capacity is predominately provided by 200 single-socket nodes, each configured with 1 AMD EPYC 7763 (“Milan”) processors with 64-cores/socket (64-cores/node) at 2.55GHz and 256GB of DDR4-3200 RAM. Half of these single-socket GPU nodes (100 nodes) are configured with 4 NVIDIA A100 GPUs with 40GB HBM2 RAM and NVLink (400 total A100 GPUs); the remaining half (100 nodes) are configured with 4 NVIDIA A40 GPUs with 48GB GDDR6 RAM and PCIe 4.0 (400 total A40 GPUs). Rounding out the GPU resource is 14 additional “dense” GPU nodes, containing 8 GPUs each, in a dual-socket CPU configuration (128-cores per node) and 2TB of DDR4-3200 RAM but otherwise configured similarly to the single-socket GPU nodes. Within the “dense” GPU nodes, 5 nodes employ NVIDIA A100 GPUs (40 total A100 GPUs in “dense” configuration) and 1 node employs AMD MI100 GPUs (8 total MI100 GPUs) with 32GB HBM2 RAM. A 1.6TB, NVMe solid-state disk is available for use as local scratch space during job execution on each GPU node type. Finally 8 of the dense GPU nodes have NVIDIA H200 GPUs with 141GB of HBM each. All Delta GPU compute nodes are interconnected to each other and to the Delta storage resource by a 200 Gb/sec HPE Slingshot network fabric. One Delta GPU SU is equal to one A100 GPU hour in the standard quad A100 partition. Other node types have charge factors that reflect their relative cost with H200s costing 3x an A100 GPU hour.
Delta GPU is a GPU-accelerated compute cluster consisting of NVIDIA A40, A100, and H200 nodes and one mixed AMD MI100/MI210 node, with four to nine GPUs, 256 GB to 2 TB of RAM, and 1.5 TB to 2 TB of local storage per node. It is particularly well suited for GPU-accelerated and hybrid CPU-GPU workloads, and is often used for AI and machine learning training and inference, data analytics, and scientific simulations. It includes a great deal of GPU-optimized AI, scientific computing, and parallel programming software.
Delta GPU provides Home, Projects, Work HDD, Work NVMe, and Node-local storage. The work file systems are shared with DeltaAI and should be used for job I/O; home directories are separate, and home-directory snapshots are not backups. For details, see [Delta File Systems].
You can run jobs at different sizes and durations on Delta GPU. The following lists the different queues that you can submit to, describing how many nodes you get, how long you can run, the type of resources you get, and the average wait time.
Delta uses the Slurm scheduler. If you do not request a wall-clock time or memory amount, Delta defaults to 30 minutes and 1,000 MB of memory per core. Submit batch jobs with sbatch; use srun to run a command or start an interactive job; and use salloc to reserve resources for multiple commands. For instructions and Delta-specific examples, see [Delta Batch Jobs] for sbatch and [Delta Interactive Jobs] for srun and salloc.
Compute resources are also available through [Delta Open OnDemand] applications such as JupyterLab, VS Code Code Server, and noVNC Desktop.
Track your work with the usual Slurm commands: squeue -u $USER shows the status of your queued and running jobs, scontrol show job followed by a job ID shows the detail for one job, sinfo -a summarizes all Delta partitions and their node states, and scancel followed by a job ID cancels a queued job or terminates a running one.
Use gpuA100x4 as the default production partition. Use gpuA40x4 for workloads that benefit from A40 GPU memory or ray-tracing support, gpuA100x8 for large multi-GPU A100 workloads, gpuH200x8 for memory-intensive or FP8/FP16 workloads, and gpuMI100x8 for AMD GPU, ROCm, or large-memory workloads. Use the interactive partitions for short development, testing, and debugging sessions. Use the A100 and A40 preemptible partitions only for checkpointing-capable jobs that can tolerate interruption and loss of unsaved progress.
Wall-time limits for each partition are in the table below. Node sharing is the default. To request exclusive use of a GPU node, request all consumable resources for the node or use --exclusive --mem=0. CPU and GPU allocations use separate local charge accounts, so submit GPU jobs with the appropriate Delta GPU account.
For complete job guidance, see [Delta Running Jobs]. GPU, PyTorch, job-array, and other example scripts are available in [Delta Sample Scripts].