How to create AWS ParallelCluster with Slurm scheduler
Environment preparation phase within AWS Management Console
- Login the AWS IAM console:
- From the left pane, click on Policies -> click on Users -> Add User -> specify the name parallelcluster-user -> Access type: Programmatic access -> click Next: Permissions -> Set permissions -> select a group with “AdministratorAccess” role -> click Next: Tags -> click Next: Review -> click on Create user -> click on Download .csv and keep it in a secured location -> click on Close
- Follow the instructions below to create a key pair to access the cluster machines:
- Follow the instructions below to create S3 bucket (with unique name) for storing data to export and import data to/from the FSx Lustre storage:
- https://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html
- Note 1: Document the S3 bucket name for use inside the ParallelCluster config file
- Note 2: Create a folder called export (in small letters), inside the S3 bucket
- In-case you wish to create a dedicate VPC and subnet for the HPC cluster, follow the instructions below:
- Logoff the AWS console
Python installation phase on Linux (Debian / Ubuntu)
- Login to a Linux machine using SSH, and follow the instructions below to install Python 3:
- https://docs.aws.amazon.com/cli/latest/userguide/install-linux-python.html
- Note: In-case you already have Python 3 install, use the command below to upgrade to the latest build:
- sudo apt-get upgrade python3
- To install pip3, run the command below:
- sudo apt install python3-pip
Python installation phase on Windows
- Login to a Windows machine using privileged account, and follow the instructions below to install Python 3 and PIP: