Description
Stampede3 is generously funded through the National Science Foundation and is designed to serve today's researchers as well as support the research community on an evolutionary path toward many-core processors and accelerated technologies. Stampede 3 maintains the familiar programming model for all of today's users, and thus will be broadly useful for traditional simulation users, users performing data intensive computations, and emerging classes of new users.
RP Description
Stampede3 is a compute cluster consisting of Intel Sapphire Rapids high-bandwidth-memory nodes, Ice Lake CPU nodes, large-memory Ice Lake NVDIMM nodes, NVIDIA H100 GPU nodes, Intel Data Center GPU Max nodes, and Skylake CPU nodes. It is particularly well suited for large-scale simulation, high-throughput computing, artificial intelligence, GPU-accelerated workloads, and large-memory applications, and is often used for groundbreaking open science research projects.
Login Text
Full instructions in [Accessing the System].
To login to Stampede3, users use standard ssh. A Stampede3 allocation though ACCESS, TACC account credentials, and multi-factor authentication are required to login.
TACC account password can be reset and multi-factor authentication can be set up in the TACC portal.
Note: If you receive error message "Corrupted MAC on input" when logging in, run:
$ssh -m hmac-sha2-512 <tacc-username>@stampede3.tacc.utexas.edu
External Storage
RANCH (Long-term Archival Tape Storage)
RANCH (Long-term Archival Tape Storage) is an allocated resource that is only available to users who have an allocation on a TACC system such as Stampede3. The default allocation for RANCH is 2 TB, and it is provided at no additional cost, meaning it does not consume ACCESS credits. It is intended for final data that is unlikely to change and does not require frequent access. RANCH can be accessed through the Globus endpoint Ranch3, and files can also be transferred using SCP with a command such as scp myfile ${ARCHIVER}:${ARCHIVE}/myfilepath.
Corral (Online Project Storage)
Corral (Online Project Storage) is TACC’s online project storage system. It is designed for users and projects that need more storage than $HOME and $WORK provide. Corral can be accessed from Stampede3 through supported TACC data transfer methods, including Globus.
For more information on managing your files, see this page.
File Transfer Text
Stampede3 supports two primary data transfer technologies: SSH-based tools (scp, sftp, rsync) and Globus. For more information about Data Transfer at TACC, see this page.
For smaller transfers less than ~200GB, ssh tools are typically easier to work with. See SSH-Based Data Transfer for more information.
For transfer larger than 200GB, Globus is the recommended transfer method. For more information on Globus on Stampede3, see Globus Data Transfer Guide at TACC.
Jobs Information
You can run jobs at different sizes and durations on Stampede3. The following lists the different queues that you can submit to, describing how many nodes you get, how long you can run, the type of resources you get, and the average wait time.
Stampede3 uses Slurm. Multi-node and MPI jobs are supported. Submit from a login node with sbatch, and monitor with squeue, scancel, and sinfo. Your job script needs --partition, --time, --account, and --ntasks set explicitly.
One unadjusted Service Unit (SU) represents a single compute node used for one hour (a node-hour). The queue charge rates are determined by the supply and demand for that particular queue or type of node used and are subject to change.
Stampede3 SUs billed = (# nodes) x max(job duration in wall clock hours,.25) x (charge rate per node-hour)
The Slurm scheduler tracks and charges for usage to a granularity of a few seconds of wall clock time. The system charges only for the resources you actually use, not those you request; however, there is a 15 minute minimum charge of queue time.
To display a summary of your TACC project balances and disk quotas at any time, execute:
login1$ /usr/local/etc/taccinfo
For submission commands, sample scripts, and node-sharing details, see Job Scripts, Running Jobs, Submitting Batch Jobs, Job Management, and Launching Applications.