Description

Expanse is a Dell integrated compute cluster, with AMD Rome processors, NVIDIA V100 GPUs, interconnected with Mellanox HDR InfiniBand in a hybrid fat-tree topology. The GPU component of Expanse features 52 GPU nodes, each containing four NVIDIA V100s (32 GB SMX2), connected via NVLINK, and dual 20-core Intel Xeon 6248 CPUs. They feature 1.6TB of NVMe storage and 256GB of DRAM per node. There is HDR100 connectivity to each node. The system also features 12PB of Lustre based performance storage (140GB/s aggregate), and 7PB of Ceph based object storage.

Resource ID
638
Global Resource ID
expanse-gpu.sdsc.access-ci.org
Resource Type
Compute
Latest Status
production
Latest Status Begin
Project Affiliation
ACCESS
Organization Name
San Diego Supercomputer Center
Organization URL
https://www.sdsc.edu
Organization Logo URL
https://cider.access-ci.org/public/resource_providers/371/logo
Detail View URL
https://www.sdsc.edu/support/user_guides/expanse.html
Raw Data URL
https://operations-api.access-ci.org/wh2/cider/v1/cider_resource_id/638/
RP Description

Expanse GPU has 56 nodes with four NVIDIA V100s (32 GB each) linked by NVLink. Suits multi-GPU AI/ML training and other workloads that scale across GPUs.

OnDemand URL
https://portal.expanse.sdsc.edu/
MFA Required
On
Jobs Information

 

GPU nodes are allocated as a separate resource. The GPU nodes can be accessed via either the "gpu" or the "gpu-shared" partitions.

#SBATCH -p gpu

or

#SBATCH -p gpu-shared

 

When users request 1 GPU, in gpu-shared partition, by default they will also receive, 1 CPU, and 1G memory.  

For more information and example job scripts see Expanse Running Jobs.

Storage Filesystems
Directory
Scratch Lustre
File System Path
/expanse/lustre/scratch
Quota (Deprecated)
10 TB
Quota size
10000
Purge Policy
90 days after allocation expiration.
Backup Policy
Not backed up​
Notes
This is not an archival file system, it is not backed up, and will be purged according to purge policy.
Directory
Scratch GPU Node
File System Path
/scratch/$USER/job_$SLURM_JOB_ID
Quota (Deprecated)
1.6TB
Quota size
1600
Notes
Users only have access to these SSDs during job execution at the local file system path to the gpu node.
Directory
Home
File System Path
/home
Quota (Deprecated)
100 GB
Quota size
100
Backup Policy
8 week rolling backup
Notes
The home directory is limited in space and should be used only for source code storage. Jobs should never be run from the home file system, as it is not set up for high performance throughput.
Queue Specifications
Queue Name
gpu
Purpose
Used for exclusive access to the GPU nodes
CPU Type
Xeon Gold 6248
GPU Type
NVIDIA V100 SXM2
RAM (Deprecated)
384 GB DDR4 DRAM
GPU Count
4
GPU vRAM
32
CPU Count
40
Node RAM
384
Queue Name
gpu-debug
Purpose
Priority access to gpu-shared nodes set aside for testing of jobs with short wall time and limited resources; max two gpus per job
CPU Type
Xeon Gold 6248
GPU Type
NVIDIA V100 SXM2
RAM (Deprecated)
384 GB DDR4 DRAM
GPU Count
4
GPU vRAM
32
CPU Count
40
Node RAM
384
Queue Name
gpu-shared
Purpose
Non-refundable discounted jobs to run on unallocated nodes that can be pre-empted by higher priority queues
CPU Type
Xeon Gold 6248
GPU Type
NVIDIA V100 SXM2
RAM (Deprecated)
384 GB DDR4 DRAM
GPU Count
4
GPU vRAM
32
CPU Count
40
Node RAM
384
Queue Name
nairr-gpu
Purpose
Expanse AI resource using compute with Nvidia GPUs
CPU Type
2 Intel Sapphire Road processors
GPU Type
Nvidia H100
RAM (Deprecated)
1 TB
GPU Count
4
GPU vRAM
80
CPU Count
36
Node RAM
1000
Datasets
Dataset Name
OceanTopography
Dataset Description

OpenTopography provides efficient, user-friendly access to high-resolution topography data, processing tools, and resources to advance understanding of the Earth's surface, vegetation, and built environment.

Dataset Name
OpenAltimetry
Dataset Description

OpenAltimetry is a web based data visualization and discovery tool for exploring surface elevation profiles over time using satellite altimetry data from NASA's ICESat and ICESat-2 missions.

Dataset Name
OpenForest4D
Dataset Description

OpenForest4D is a web-based platform that leverages multi-source remote sensing data and artificial intelligence to generate on-demand, research-grade estimates of forest structure and above-ground biomass in four dimensions for global forest monitoring.