Description

Expanse is a Dell integrated compute cluster, with AMD Rome processors, 128 cores per node, interconnected with Mellanox HDR InfiniBand in a hybrid fat-tree topology. The compute node section of Expanse has a peak performance of 3.373 PF. Full bisection bandwidth is available at rack level (56 compute nodes) with HDR100 connectivity to each node. HDR200 switches are used at the rack level and 3:1 oversubscription cross-rack. Compute nodes feature 1TB of NVMe storage and 256GB of DRAM per node. The system also features 12PB of Lustre based performance storage (140GB/s aggregate), and 7PB of Ceph based object storage.

Resource ID
637
Global Resource ID
expanse.sdsc.access-ci.org
Resource Type
Compute
Latest Status
production
Latest Status Begin
Project Affiliation
ACCESS
Organization Name
San Diego Supercomputer Center
Organization URL
https://www.sdsc.edu
Organization Logo URL
https://cider.access-ci.org/public/resource_providers/371/logo
Detail View URL
https://www.sdsc.edu/support/user_guides/expanse.html
Raw Data URL
https://operations-api.access-ci.org/wh2/cider/v1/cider_resource_id/637/
RP Description

Expanse CPU has 728 AMD EPYC Rome nodes with 128 cores and 256 GB RAM each. Suited for parallel CPU workloads that scale across many nodes.

OnDemand URL
https://portal.expanse.sdsc.edu/
MFA Required
On
Jobs Information

Expanse uses the Simple Linux Utility for Resource Management (SLURM) batch environment. When you run in the batch mode, you submit jobs to be run on the compute nodes using the sbatch command as described in Expanse Running Jobs.

Remember that computationally intensive jobs should be run only on the compute nodes and not the login nodes. 

Storage Filesystems
Directory
Scratch Lustre
File System Path
/expanse/lustre/scratch
Quota (Deprecated)
10 TB
Quota size
10000
Purge Policy
90 days after allocation expiration.
Backup Policy
Not backed up​
Notes
This is not an archival file system, it is not backed up, and will be purged according to purge policy.
Directory
Scratch Compute Node
File System Path
/scratch/$USER/job_$SLURM_JOB_ID
Quota (Deprecated)
1 TB
Quota size
1000
Notes
Users only have access to these SSDs during job execution at the local file system path to the compute node.
Directory
Home
File System Path
/home
Quota (Deprecated)
100 GB
Quota size
100
Purge Policy
N/A
Backup Policy
8 week rolling backup
Notes
The home directory is limited in space and should be used only for source code storage. Jobs should never be run from the home file system, as it is not set up for high performance throughput.
Queue Specifications
Queue Name
compute
Purpose
Compute Node Usage
CPU Type
AMD EPYC 7742
RAM (Deprecated)
256 GB DDR4 DRAM
CPU Count
128
Node RAM
256
Queue Name
debug
Purpose
Priority access to shared nodes set aside for testing of jobs with short wall time and limited resources
CPU Type
AMD EPYC 7742
RAM (Deprecated)
256 GB DDR4 DRAM
CPU Count
128
Node RAM
256
Queue Name
preempt
Purpose
Non-refundable discounted jobs to run on free nodes that can be pre-empted by jobs submitted to any other queue
CPU Type
AMD EPYC 7742
RAM (Deprecated)
256 GB DDR4 DRAM
CPU Count
128
Node RAM
256
Queue Name
large-shared
Purpose
Single-node jobs using large memory up to 2 TB (minimum memory required 256G)
CPU Type
AMD EPYC 7742
RAM (Deprecated)
256 GB DDR4 DRAM
CPU Count
128
Node RAM
2000
Queue Name
shared
Purpose
Jobs using part of a single 256 GB node, sharing the node with other jobs.
CPU Type
AMD EPYC 7742
RAM (Deprecated)
256 GB DDR4 DRAM
CPU Count
128
Node RAM
256
Datasets
Dataset Name
OceanTopography
Dataset Description

OpenTopography provides efficient, user-friendly access to high-resolution topography data, processing tools, and resources to advance understanding of the Earth's surface, vegetation, and built environment.

Dataset Name
OpenAltimetry
Dataset Description

OpenAltimetry is a web based data visualization and discovery tool for exploring surface elevation profiles over time using satellite altimetry data from NASA's ICESat and ICESat-2 missions.

Dataset Name
OpenForest4D
Dataset Description

OpenForest4D is a web-based platform that leverages multi-source remote sensing data and artificial intelligence to generate on-demand, research-grade estimates of forest structure and above-ground biomass in four dimensions for global forest monitoring. 

RP source documentation for Submitting Jobs
https://www.sdsc.edu/systems/expanse/user_guide.html#narrow-wysiwyg-7
RP source documentation for Software
https://www.sdsc.edu/systems/expanse/user_guide.html#narrow-wysiwyg-13