Description

The Delta GPU resource comprises 5 different node configurations intended to support accelerated computation across a broad range of domains including traditional simulation and AI/ML work. Delta is designed to support the transition of applications from CPU-only to using the GPU or hybrid CPU-GPU models. Delta GPU resource capacity is predominately provided by 200 single-socket nodes, each configured with 1 AMD EPYC 7763 (“Milan”) processors with 64-cores/socket (64-cores/node) at 2.55GHz and 256GB of DDR4-3200 RAM. Half of these single-socket GPU nodes (100 nodes) are configured with 4 NVIDIA A100 GPUs with 40GB HBM2 RAM and NVLink (400 total A100 GPUs); the remaining half (100 nodes) are configured with 4 NVIDIA A40 GPUs with 48GB GDDR6 RAM and PCIe 4.0 (400 total A40 GPUs). Rounding out the GPU resource is 14 additional “dense” GPU nodes, containing 8 GPUs each, in a dual-socket CPU configuration (128-cores per node) and 2TB of DDR4-3200 RAM but otherwise configured similarly to the single-socket GPU nodes. Within the “dense” GPU nodes, 5 nodes employ NVIDIA A100 GPUs (40 total A100 GPUs in “dense” configuration) and 1 node employs AMD MI100 GPUs (8 total MI100 GPUs) with 32GB HBM2 RAM. A 1.6TB, NVMe solid-state disk is available for use as local scratch space during job execution on each GPU node type. Finally 8 of the dense GPU nodes have NVIDIA H200 GPUs with 141GB of HBM each. All Delta GPU compute nodes are interconnected to each other and to the Delta storage resource by a 200 Gb/sec HPE Slingshot network fabric. One Delta GPU SU is equal to one A100 GPU hour in the standard quad A100 partition. Other node types have charge factors that reflect their relative cost with H200s costing 3x an A100 GPU hour.

Resource ID
742
Global Resource ID
delta-gpu.ncsa.access-ci.org
Resource Type
Compute
Latest Status
production
Latest Status Begin
Latest Status End
Project Affiliation
ACCESS
Organization Name
National Center for Supercomputing Applications
RP Description

Delta GPU is a GPU-accelerated compute cluster consisting of NVIDIA A40, A100, and H200 nodes and one mixed AMD MI100/MI210 node, with four to nine GPUs, 256 GB to 2 TB of RAM, and 1.5 TB to 2 TB of local storage per node. It is particularly well suited for GPU-accelerated and hybrid CPU-GPU workloads, and is often used for AI and machine learning training and inference, data analytics, and scientific simulations. It includes a great deal of GPU-optimized AI, scientific computing, and parallel programming software.

MFA Required
On
Storage Text

Delta GPU provides Home, Projects, Work HDD, Work NVMe, and Node-local storage. The work file systems are shared with DeltaAI and should be used for job I/O; home directories are separate, and home-directory snapshots are not backups. For details, see [Delta File Systems].

Jobs Information

You can run jobs at different sizes and durations on Delta GPU. The following lists the different queues that you can submit to, describing how many nodes you get, how long you can run, the type of resources you get, and the average wait time.

Delta uses the Slurm scheduler. If you do not request a wall-clock time or memory amount, Delta defaults to 30 minutes and 1,000 MB of memory per core. Submit batch jobs with sbatch; use srun to run a command or start an interactive job; and use salloc to reserve resources for multiple commands. For instructions and Delta-specific examples, see [Delta Batch Jobs] for sbatch and [Delta Interactive Jobs] for srun and salloc.

Compute resources are also available through [Delta Open OnDemand] applications such as JupyterLab, VS Code Code Server, and noVNC Desktop.

Track your work with the usual Slurm commands: squeue -u $USER shows the status of your queued and running jobs, scontrol show job followed by a job ID shows the detail for one job, sinfo -a summarizes all Delta partitions and their node states, and scancel followed by a job ID cancels a queued job or terminates a running one.

Use gpuA100x4 as the default production partition. Use gpuA40x4 for workloads that benefit from A40 GPU memory or ray-tracing support, gpuA100x8 for large multi-GPU A100 workloads, gpuH200x8 for memory-intensive or FP8/FP16 workloads, and gpuMI100x8 for AMD GPU, ROCm, or large-memory workloads. Use the interactive partitions for short development, testing, and debugging sessions. Use the A100 and A40 preemptible partitions only for checkpointing-capable jobs that can tolerate interruption and loss of unsaved progress.

Wall-time limits for each partition are in the table below. Node sharing is the default. To request exclusive use of a GPU node, request all consumable resources for the node or use --exclusive --mem=0. CPU and GPU allocations use separate local charge accounts, so submit GPU jobs with the appropriate Delta GPU account.

For complete job guidance, see [Delta Running Jobs]. GPU, PyTorch, job-array, and other example scripts are available in [Delta Sample Scripts].

Storage Filesystems
Directory
Home
File System Path
$HOME
Quota size
100
Quota inode amount
750000
Purge Policy
Never purged
Backup Policy
Snapshots only, retained 30 days
Notes
Software, scripts, and job files; not intended for job I/O. Snapshots live on the same hardware as the primary copy and are not a backup - back up anything you cannot lose.
Directory
Projects
File System Path
/projects/<project>
Quota size
500
Quota inode amount
300000
Purge Policy
Never purged
Backup Policy
Not backed up
Notes
Shared project data, software, and results. Up to 25 TB by allocation request.
Directory
Work HDD
File System Path
/work/hdd/<project>
Quota size
1000
Quota inode amount
500000
Purge Policy
Never purged
Backup Policy
Not backed up
Notes
Active computational data and job I/O. Up to 100 TB by allocation request. Shared with DeltaAI.
Directory
Work NVMe
File System Path
/work/nvme/<project>
Quota (Deprecated)
NVME space is available upon request.
Purge Policy
Never purged
Backup Policy
Not backed up
Notes
Available upon request. Use quota to check assigned capacity and inode limits. Best for frequent small-file I/O. Shared with DeltaAI.
Directory
Node-local
File System Path
/tmp
Purge Policy
Purged at job end
Backup Policy
Not backed up
Notes
1.5 TB on A40, A100, and AMD GPU nodes; 2 TB on H200 nodes; no enforced quota. Capacity may be shared by concurrent jobs. Use for fast small-file I/O and copy results before the job ends.
Queue Specifications
Queue Name
gpuA100x4
Purpose
Standard production GPU jobs; this is Delta’s default GPU partition
CPU Type
1x AMD EPYC 7763 "Milan"
GPU Type
NVIDIA A100
GPU Count
4
GPU vRAM
40
CPU Count
64
Node RAM
256
Queue Name
gpuA100x4-preempt
Purpose
Checkpointing-capable A100 jobs that can tolerate interruption and loss of unsaved progress
CPU Type
1x AMD EPYC 7763 "Milan"
GPU Type
NVIDIA A100
GPU Count
4
GPU vRAM
40
CPU Count
64
Node RAM
256
Queue Name
gpuA40x4
Purpose
Production jobs that benefit from A40 GPU memory or ray-tracing support
CPU Type
1x AMD EPYC 7763 "Milan"
GPU Type
NVIDIA A40
GPU Count
4
GPU vRAM
48
CPU Count
64
Node RAM
256
Queue Name
gpuA40x4-preempt
Purpose
Checkpointing-capable A40 jobs that can tolerate interruption and loss of unsaved progress
CPU Type
1x AMD EPYC 7763 "Milan"
GPU Type
NVIDIA A40
GPU Count
4
GPU vRAM
48
CPU Count
64
Node RAM
256
Queue Name
gpuH200x8
Purpose
Memory-intensive or FP8/FP16 production jobs; maximum 1 node per job
CPU Type
2x Intel Xeon Platinum 8558
GPU Type
NVIDIA H200
GPU Count
8
GPU vRAM
141
CPU Count
96
Node RAM
2048
Queue Name
gpuA100x8
Purpose
Large-scale production jobs that need up to eight A100 GPUs and 2 TB of node memory
CPU Type
2x AMD EPYC 7763 "Milan"
GPU Type
NVIDIA A100
GPU Count
8
GPU vRAM
40
CPU Count
128
Node RAM
2048
Queue Name
gpuMI100x8
Purpose
AMD GPU and ROCm workloads, or large-memory jobs that can use this single-node partition
CPU Type
2x AMD EPYC 7763 "Milan"
GPU Type
AMD MI100, AMD MI210
GPU Count
8
GPU vRAM
32
CPU Count
128
Node RAM
2048
Queue Name
gpuA100x4-interactive
Purpose
Short interactive sessions for development, testing, and debugging; maximum 4 nodes per job
CPU Type
1x AMD EPYC 7763 "Milan"
GPU Type
NVIDIA A100
GPU Count
4
GPU vRAM
40
CPU Count
64
Node RAM
256
Queue Name
gpuA40x4-interactive
Purpose
Short interactive sessions for development, testing, and debugging; maximum 4 nodes per job
CPU Type
1x AMD EPYC 7763 "Milan"
GPU Type
NVIDIA A40
GPU Count
4
GPU vRAM
48
CPU Count
64
Node RAM
256
Queue Name
gpuH200x8-interactive
Purpose
Short interactive H200 sessions for development, testing, and tuning; maximum 1 node per job
CPU Type
2x Intel Xeon Platinum 8558
GPU Type
NVIDIA H200
GPU Count
8
GPU vRAM
141
CPU Count
96
Node RAM
2048
Queue Name
gpuA100x8-interactive
Purpose
Short interactive multi-GPU sessions for development and testing; maximum 2 nodes per job
CPU Type
2x AMD EPYC 7763 "Milan"
GPU Type
NVIDIA A100
GPU Count
8
GPU vRAM
40
CPU Count
128
Node RAM
2048
Queue Name
gpuMI100x8-interactive
Purpose
Short interactive AMD GPU sessions for development and testing; maximum 1 node per job
CPU Type
2x AMD EPYC 7763 "Milan"
GPU Type
AMD MI100, AMD MI210
GPU Count
8
GPU vRAM
32
CPU Count
128
Node RAM
2048