An introduction to High Performance Computing
Location: Online
Organisers
Trainers
- Renato Alves (Bio-IT -EMBL)
- Benjamin Wolff (ZB MED - Information Centre for Life Sciences)
- Alan O'Cais (University of Barcelona)
BioNT - BIO Network for Training - is an international consortium of academic entities and small and medium-sized enterprises (SMEs). BioNT is dedicated to providing a comprehensive training program and fostering a community for digital skills relevant to the biotechnology industry and biomedical sector. With a curriculum tailored for both beginners and advanced professionals, BioNT aims to equip individuals with the necessary expertise in handling, processing, and visualising biological data, as well as utilising computational biology tools. Leveraging the consortium's strong background in digital literacy training and extensive network of collaborations, BioNT is poised to professionalise life sciences data management, processing, and analysis skills.
This practical introduction to high performance computing (HPC) will cover all basic concepts needed to access an HPC cluster and to run applications and workflows there.
The course is built upon the Carpentries and HPC Carpentry online learning materials, with HPC Carpentry being a lesson program in incubation for the Carpentries. The Carpentries is a nonprofit organisation that teaches software engineering and data science skills to researchers in order to conduct efficient, open, and reproducible research. Their volunteer instructors (4287) have run 4000+ Workshops in 65 countries since 2012, with 450+ alone in 2022 [3]. All of their lesson materials are freely reusable under the Creative Commons - Attribution licence [4].
Here, we are offering a 3-day workshop, composed of a half-day and 2 full-day sessions, with the primary goal of introducing High-Performance Computing (HPC) to individuals who have limited or no prior experience with such computing resources. The first day is dedicated to guiding participants through the basics of file systems and the Unix shell. This foundation is a prerequisite for the other days which introduces HPC resources, the cluster management tool Slurm, and how to run applications and workflows on such resources.
Join this workshop if you are:
- Interested in learning what HPC is
- Interested in learning how to access and use HPC resources
- Interested in running applications and workflows on HPC resources
Learning Outcomes:
By the end of this workshop, you will be able to:
- Use the UNIX shell (also known as terminal or command line) to connect to a cluster
- Identify problems a cluster can help solve
- Transfer files onto a cluster
- Comfortably manipulate files and directories, locally and on a remote resource
- Submit and manage jobs on a cluster using a scheduler
- Observe the benefits and limitations of parallel execution
- Understand how to construct a reliable, scalable and reproducible scientific workflow as a series of chained rules
- Use a subset of features of the Snakemake workflow tool
- Run a Snakemake workflow on an HPC resource
Requirements:
- There is no need for programming or informatics skills but a basic understanding of what files and directories are is required
- PC/Laptop with an up-to-date browser. Chrome, Safari and Firefox browsers are all supported (some older browsers, including Internet Explorer version 9 and below, may not be)
- The lessons require you to have access to a terminal application with ssh capabilities. If it is unclear what this requirement means, please click here for guidance on how to make this available for your operating system
Recommendations for your setup and interacting during the workshop:
- To follow the workshop more efficiently, we recommend having a two-screen setup: for example, one to display the instructor’s shared screen and the collaborative pad, and another one for your own coding.
- To actively communicate during the workshop, please familiarise yourself with Markdown formatting by reviewing the HedgeDoc features document.
Useful links and videos
- https://carpentries.org/index.html
- https://software-carpentry.org/
- https://www.hpc-carpentry.org/
- https://carpentries-incubator.github.io/hpc-intro/setup.html
- https://docs.carpentries.org/topic_folders/policies/code-of-conduct.html
Interaction between participants, trainers and helpers
The workshop will be delivered in a Zoom webinar format, with participants’ visibility disabled to preserve their privacy. You, as a participant, will be able to see and learn from the trainers but a direct interaction (e.g. chat or voice) will not be possible during the sessions. Instead, a collaborative document, previously setup by the trainers, will be shared with you before the session. You will be expected to engage and interact anonymously with other participants as well as with the workshop helpers and trainers directly in this document.
Trainer Hubs
All BioNT workshops are offered at no cost, but there are a limited number of seats available. To make workshops more accessible for members of the same company we highly recommend organising what we refer to as "Training Hubs." In this arrangement, one person is formally registered for the workshop, but the knowledge sharing can be expanded to numerous colleagues within their company or SME through live-streaming the session.
Topics
Day |
Topic |
Tutorial |
Day 1 |
The Unix Shell |
2. Navigating Files and Directories |
Day 2 |
Introduction to High-Performance Computing |
2. Connecting to a remote HPC system 5. Accessing software via Modules |
Day 3 |
Introduction to High-Performance Computing HPC Workflow Management with Snakemake |
8. Using resources effectively 9. Using shared resources responsibly
1. Running commands with Snakemake 2. Running Snakemake on the cluster 3. Placeholders 4. MPI applications and Snakemake
|
How to register
The workshop is free of charge. To participate, please follow these steps:
To participate, please follow these steps:
- Click on the window “Participate” at the top of this page
- You will be redirected to the members.cecam.org page. If you already have an account on our platform, please proceed to step 5
- On the top-right corner click "Register" and complete the provided form. As indicated, completing this form does not register you to the workshop. Within 72 hours you will receive an email confirming your account has been activated. Due to this processing time, we advise you to register a few days before the registration deadline
- After receiving the account activation confirmation, visit the workshop page again and follow instructions starting from step 1
- You should now have an active account. After login in with your login details, you should be redirected to the workshop registration page
- In order to start your registration please follow the instructions of the linked pre-workshop survey until you will get your unique identifier
- To finalise your registration please use the unique identifier in the CECAM platform in the corresponding section and press “Send mail”
- Your application is now submitted for evaluation. If selected, you will be contacted later to confirm your attendance and provide instructions for installing the required software and participating in the online workshop.
Literature
[1] The Carpentries “Annual Report 2022”, 2022, https://carpentries.org/files/reports/AnnualReport2022.pdf
[2] Software Carpentry “About us”, retrieved 25.08.2023, https://software-carpentry.org/about
References
Javier Diaz (University of Barcelona) - Organiser
Alan O'Cais (University of Barcelona) - Organiser
Ignacio Pagonabarraga (University of Barcelona) - Organiser