Description
A virtual HTCondor pool made up of resources from the OSG Consortium
RP Description
The OSG provides technologies and services for supporting pools of distributed high-throughput computing (dHTC) and data capacity that serve a number of research communities, worldwide. Among these services, the OSG operates the Open Science Pool (OSPool), which includes capacity contributed from clusters at campuses and national labs across and beyond the US, organized as a single virtual cluster.
File Transfer Text
When using a command line application for performing file transfers, utilities such as scp can be used. Other GUI tools can be used for transferring files and folders. If some of the data or software the jobs depend on is available via the web, files can be transferred using the appropriate HTTP address.
Storage Text
Smaller job files should be stored in the /home directory and can be transferred using HTCondor. Larger job files should be stored in and transferred using OSDF.
Jobs Information
Jobs are submitted using the HTCondor software. This process involves preparing an executable script and a submit file to submit the job. From there, the status of jobs in the queue can be reviewed. Upon conclusion, the results can be examined. An example of this process can be found here. Additionally, a video can be found here that goes into more detail.
It is important to determine the amount of resources that need to be requested by starting with test job submissions before scaling up. A job duration category must also be specified upon job submission. Tips on submitting test jobs and determining resources can be found on this page. Additional information can be found in this video on high throughput computing with HTCondor.
HTCondor can submit multiple jobs using queue. Information on this process can be found on this page.
Information on converting workflow from slurm to HTCondor can be found here.