How to create AWS ParallelCluster with Slurm scheduler - Revision history

Eyales at 13:17, 1 August 2019

2019-08-01T13:17:51Z

← Older revision		Revision as of 13:17, 1 August 2019
Line 177:		Line 177:
	* Scale HPC Workloads with Elastic Fabric Adapter and AWS ParallelCluster		* Scale HPC Workloads with Elastic Fabric Adapter and AWS ParallelCluster
	: https://idk.dev/scale-hpc-workloads-with-elastic-fabric-adapter-and-aws-parallelcluster/		: https://idk.dev/scale-hpc-workloads-with-elastic-fabric-adapter-and-aws-parallelcluster/
			* Best Practices for Running Ansys Fluent Using AWS ParallelCluster
			: https://aws.amazon.com/blogs/opensource/best-practices-running-ansys-fluent-aws-parallelcluster/
			* AWS ParallelCluster with AWS Directory Services Authentication
			: https://aws.amazon.com/blogs/opensource/aws-parallelcluster-aws-directory-services-authentication/
	* Adding support for FSx for Lustre:		* Adding support for FSx for Lustre:
	: https://aws-parallelcluster.readthedocs.io/en/develop/configuration.html#fsx-section		: https://aws-parallelcluster.readthedocs.io/en/develop/configuration.html#fsx-section

Eyales at 13:24, 29 July 2019

2019-07-29T13:24:31Z

← Older revision		Revision as of 13:24, 29 July 2019
Line 20:		Line 20:
	* To install pip3, run the command below:		* To install pip3, run the command below:
	: '''sudo apt install python3-pip'''		: '''sudo apt install python3-pip'''

			== Python installation phase on CentOS 7 ==
			* Login to the CentOS machine using SSH, and follow the instructions below to install Python3 and Python3-PIP:
			: https://www.rosehosting.com/blog/how-to-install-python-3-6-4-on-centos-7/

	== Python installation phase on Windows ==		== Python installation phase on Windows ==

Eyales at 12:16, 29 July 2019

2019-07-29T12:16:37Z

← Older revision		Revision as of 12:16, 29 July 2019
Line 92:		Line 92:
	* Wait for the cluster deployment to complete.		* Wait for the cluster deployment to complete.
	* Document the '''MasterPublicIP''' and '''ClusterUser'''.		* Document the '''MasterPublicIP''' and '''ClusterUser'''.

			== Increase EC2 service limit ==
			* In-case you need to increase the EC2 service limit (for example number of EC2 instances from a specific instance type), follow the instructions below:
			: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-resource-limits.html#request-increase

	== Connecting to the HPC cluster (from Linux Machine) ==		== Connecting to the HPC cluster (from Linux Machine) ==

Eyales at 12:02, 29 July 2019

2019-07-29T12:02:58Z

← Older revision		Revision as of 12:02, 29 July 2019
Line 139:		Line 139:
	: '''pcluster start mycluster'''		: '''pcluster start mycluster'''
	: Note: Replace '''mycluster''' with the previously create cluster name		: Note: Replace '''mycluster''' with the previously create cluster name
	* ~~Delete a~~ cluster:
			== Delete AWS ParallelCluster ==
			* In-case you wish to keep the AWS ParallelCluster master node static IP, login to the AWS console:
			: https://console.aws.amazon.com/ec2/
			* From the left pane, click on Elastic IPs -> select the public IP of the master node -> Actions -> Disassociate address
			* From command prompt (the same machine you used the pcluster commands), run the command below to delete the cluster:
	: '''pcluster delete mycluster'''		: '''pcluster delete mycluster'''
	: Note: Replace '''mycluster''' with the previously create cluster name		: Note: Replace '''mycluster''' with the previously create cluster name

Eyales at 06:52, 4 July 2019

2019-07-04T06:52:03Z

← Older revision		Revision as of 06:52, 4 July 2019
Line 99:		Line 99:
	: Note 2: Replace '''/path/to/keyfile.pem''' with the actual path and key file name		: Note 2: Replace '''/path/to/keyfile.pem''' with the actual path and key file name
	* Run the command below to verify the state of the cluster:		* Run the command below to verify the state of the cluster:
	: '''~~qinfo~~'''		: '''sinfo'''

	== Connecting to the HPC cluster (from Windows Machine) ==		== Connecting to the HPC cluster (from Windows Machine) ==
Line 118:		Line 118:
	* Click on Open		* Click on Open
	* Run the command below to verify the state of the cluster:		* Run the command below to verify the state of the cluster:
	: '''~~qinfo~~'''		: '''sinfo'''

	== Common actions to control the cluster ==		== Common actions to control the cluster ==

Eyales at 06:18, 4 July 2019

2019-07-04T06:18:35Z

← Older revision		Revision as of 06:18, 4 July 2019
Line 146:		Line 146:
	* Long term data must be stored inside S3 bucket		* Long term data must be stored inside S3 bucket
	* The Amazon FSx for Lustre storage (mount '''/fsx''') will be used for the duration of the compute job		* The Amazon FSx for Lustre storage (mount '''/fsx''') will be used for the duration of the compute job

			== References ==
			* Getting started with AWS ParallelCluster:
			: https://aws-parallelcluster.readthedocs.io/en/latest/getting_started.html#
			* Setting Up AWS ParallelCluster
			: https://docs.aws.amazon.com/parallelcluster/latest/ug/getting_started.html
			* Install AWS ParallelCluster in a Virtual Environment
			: https://docs.aws.amazon.com/parallelcluster/latest/ug/install-virtualenv.html
			* A Scientist's Guide to Cloud-HPC: Example with AWS ParallelCluster, Slurm, Spack, and WRF
			: https://jiaweizhuang.github.io/blog/aws-hpc-guide/
			* Launch your first sample HPC environment on AWS and review important concepts along the way
			: https://aws.amazon.com/getting-started/use-cases/hpc/
			* AWS ParallelCluster Wiki:
			: https://github.com/aws/aws-parallelcluster
			* Deploying an Elastic HPC Cluster
			: https://d1.awsstatic.com/Projects/P4114756/deploy-elastic-hpc-cluster_project.pdf
			* Scale HPC Workloads with Elastic Fabric Adapter and AWS ParallelCluster
			: https://idk.dev/scale-hpc-workloads-with-elastic-fabric-adapter-and-aws-parallelcluster/
			* Adding support for FSx for Lustre:
			: https://aws-parallelcluster.readthedocs.io/en/develop/configuration.html#fsx-section
			* Getting Started with Amazon FSx for Lustre
			: https://docs.aws.amazon.com/fsx/latest/LustreGuide/getting-started.html
			* Amazon FSx for Lustre Lustre User Guide
			: https://docs.aws.amazon.com/fsx/latest/LustreGuide/LustreGuide.pdf

Eyales at 06:17, 4 July 2019

2019-07-04T06:17:01Z

← Older revision		Revision as of 06:17, 4 July 2019
Line 119:		Line 119:
	* Run the command below to verify the state of the cluster:		* Run the command below to verify the state of the cluster:
	: '''qinfo'''		: '''qinfo'''

			== Common actions to control the cluster ==
			* Displays a list of stacks that are associated with AWS ParallelCluster:
			: '''pcluster list'''
			* Displays a list of all instances in a cluster:
			: '''pcluster instances mycluster'''
			: Note: Replace '''mycluster''' with the previously create cluster name
			* View the current status of the cluster:
			: '''pcluster status mycluster'''
			: Note: Replace '''mycluster''' with the previously create cluster name
			* Updates a running cluster by using the values in the configuration file:
			: '''pcluster update mycluster -c ~/.parallelcluster/config'''
			: Note 1: Replace '''mycluster''' with the previously create cluster name
			: Note 2: Replace '''~/.parallelcluster/config''' with the target config file location
			* Stops the compute fleet, leaving the master node running:
			: '''pcluster stop mycluster'''
			: Note: Replace '''mycluster''' with the previously create cluster name
			* Starts the compute fleet for a cluster that has been stopped:
			: '''pcluster start mycluster'''
			: Note: Replace '''mycluster''' with the previously create cluster name
			* Delete a cluster:
			: '''pcluster delete mycluster'''
			: Note: Replace '''mycluster''' with the previously create cluster name

			== Important notes regarding shared storage ==
			* Long term data must be stored inside S3 bucket
			* The Amazon FSx for Lustre storage (mount '''/fsx''') will be used for the duration of the compute job

Eyales at 06:14, 4 July 2019

2019-07-04T06:14:39Z

@@ Line 98: / Line 98: @@
 : Note 1: Replace '''mycluster''' with the previously create cluster name
 : Note 2: Replace '''/path/to/keyfile.pem''' with the actual path and key file name
 * Run the command below to verify the state of the cluster:
 : '''qinfo'''

Eyales at 06:12, 4 July 2019

2019-07-04T06:12:33Z

@@ Line 84: / Line 84: @@
 : '''weekly_maintenance_start_time = 1:00:00'''
 : Note 1: The storage_capacity is the size of the FSx Lustre storage in GB
-: Note 2: Replace the value of bucket with the previously S3 bucket name
+: Note 2: Replace the value of '''bucket''' with the previously S3 bucket name

Eyales at 06:09, 4 July 2019

2019-07-04T06:09:48Z

@@ Line 52: / Line 52: @@
 :* '''Master Subnet ID''': Specify here the name of the target subnet ID to deploy the HPC cluster into
 :: Note: The full list of subnets can be found within the AWS management console: https://console.aws.amazon.com/vpc
-* Edit the ParallelCluster config file:
+* Edit the ParallelCluster '''config''' file:
 :* Linux: The file is located inside '''~/.parallelcluster/config'''
 :* Windows: The file is located inside '''%UserProfile%\.parallelcluster\config'''