The manual approach becomes quite inconvenient at production, so you will be much better off by using a higher-level framework.īasic cluster operations with Slurm, an open-source, modern job scheduler deployed on many HPC centers. If you feel that all those frameworks are too black-boxy, try building a cluster manually 2 to understand how multiple nodes are glued together. If you prefer a multi-platform, general-purpose tool, consider ElastiCluster, but expect a steeper learning curve and less up-to-date AWS features. Spin-up an HPC cluster with AWS ParallelCluster, AWS's official HPC framework. I would avoid diving into a particular scientific field, or into a niche AWS utility with no counterparts on other cloud platforms - those can be left to other posts, but do not belong to this guide. But you should be able to use a cloud-HPC cluster with confidence after going through this guide.įocus on common, general, transferrable cases. That being said, it can still take a while for new users to learn.

Minimize the learning curve for non-IT/non-CS people. To make people's lives a bit easier (and to provide a reference for the future me), I wrote this guide to show an easy-to-follow workflow of building a fully-fledged HPC cluster environment on AWS. There are very few documentations for this niche field, and users could face a pretty steep learning curve.

However, orchestrating a cloud-HPC cluster is by no means easy, especially considering that many HPC users are from science and engineering and are not trained with IT and system administration skills.

Cloud-HPC is growing rapidly 1, and the growth can only be faster with AWS's recent HPC-oriented services such as EC2 c5n, FSx for Lustre, and the soon-coming EFA.

How to connect to wps on s3