PUBLIC-WIKI - User contributions [en]

PUBLIC-WIKI:Privacy policy

2021-01-07T12:39:25Z

Eyales: Created page with "== Website Privacy Policy and Cookies Policy == This policy describes how we collect and use your personal data during your use of the IUCC Wiki website (https://public-wiki.i..."

== Website Privacy Policy and Cookies Policy ==
This policy describes how we collect and use your personal data during your use of the IUCC Wiki website (https://public-wiki.iucc.ac.il/) (the ‘website’) in accordance with the General Data Protection Regulation (GDPR) and related Israel data protection legislation. For further information about the structure of the IUCC Wiki‘s web presence, please see in the Website.

Inter-University Computation Center (IUCC) do not collect any information about customers of the IUCC Wiki website.

If you are still looking for more information than you can contact us through one of our preferred contact methods:
* Email: dpo@iucc.ac.il

How to run MATLAB Parallel Server on Azure

2020-11-16T06:30:45Z

Eyales:

== Pre-Requirements ==
* Azure subscription
* Subscription owner privilege

== MATLAB License ==
* Locate the institution MATLAB License Administrator, through the License Centre:
: https://www.mathworks.com/licensecenter/licenses
* Log into your account (if prompted)
* Click on license number
* Click on "Contact Administrators" tab
* Send an email to the MATLAB License Administrator, and ask the license administrator to add the researcher's institution email address, a license for "MATLAB Parallel Server"
* To verify that "MATLAB Parallel Server" license was assigned, go to the link below:
: https://www.mathworks.com/licensecenter/

== Deploy MATLAB Parallel Server on Azure ==
* Open a web browser
* Follow the deployment instructions in the link below:
: https://github.com/mathworks-ref-arch/matlab-parallel-server-on-azure/blob/master/releases/R2020b/README.md

== Delete an existing MATLAB Parallel Server ==
* Open a web browser
* Follow the instructions in the link below:
: https://github.com/mathworks-ref-arch/matlab-parallel-server-on-azure/blob/master/releases/R2020b/README.md#delete-your-cloud-resources

How to run MATLAB Parallel Server on AWS

2020-11-16T06:29:51Z

Eyales:

== Pre-Requirements ==
* AWS Account
* AWS Full Administrator IAM Role

== MATLAB License ==
* Locate the institution MATLAB License Administrator, through the License Centre:
: https://www.mathworks.com/licensecenter/licenses
* Log into your account (if prompted)
* Click on license number
* Click on "Contact Administrators" tab
* Send an email to the MATLAB License Administrator, and ask the license administrator to add the researcher's institution email address, a license for "MATLAB Parallel Server"
* To verify that "MATLAB Parallel Server" license was assigned, go to the link below:
: https://www.mathworks.com/licensecenter/

== Deploy MATLAB Parallel Server on AWS ==
* Open a web browser
* Follow the deployment instructions in the link below:
: https://github.com/mathworks-ref-arch/matlab-parallel-server-on-aws/blob/master/releases/R2020b/README.md

== Delete an existing MATLAB Parallel Server ==
* Open a web browser
* Follow the instructions in the link below:
: https://github.com/mathworks-ref-arch/matlab-parallel-server-on-aws/blob/master/releases/R2020b/README.md#delete-your-cloud-resources

How to run MATLAB Parallel Server on Azure

2020-08-31T13:14:49Z

Eyales: Created page with "== Pre-Requirements == * Azure subscription * Subscription owner privilege == MATLAB License == * Locate the institution MATLAB License Administrator, through the License Cen..."

== Pre-Requirements ==
* Azure subscription
* Subscription owner privilege

== MATLAB License ==
* Locate the institution MATLAB License Administrator, through the License Centre:
: https://www.mathworks.com/licensecenter/licenses
* Log into your account (if prompted)
* Click on license number
* Click on "Contact Administrators" tab
* Send an email to the MATLAB License Administrator, and ask the license administrator to add the researcher's institution email address, a license for "MATLAB Parallel Server"
* To verify that "MATLAB Parallel Server" license was assigned, go to the link below:
: https://www.mathworks.com/licensecenter/

== Deploy MATLAB Parallel Server on Azure ==
* Open a web browser
* Follow the deployment instructions in the link below:
: https://github.com/mathworks-ref-arch/matlab-parallel-server-on-azure/blob/master/releases/R2019b/README.md

== Delete an existing MATLAB Parallel Server ==
* Open a web browser
* Follow the instructions in the link below:
: https://github.com/mathworks-ref-arch/matlab-parallel-server-on-azure/blob/master/releases/R2019b/README.md#delete-your-cloud-resources

Azure - How To

2020-08-31T13:11:54Z

Eyales:

*[[Best practices for Azure cost optimization]]
*[[Best practices for securing Azure subscriptions]]
*[[How to configure MFA (Multi-Factor Authentication) for AD Azure account]]
*[[How to create a file share in Azure Files]]
*[[How to create Azure Virtual Machine and perform login using SSH]]
*[[How to create Azure Network Security Group]]
*[[How to create Azure Network Interface]]
*[[How to create Azure Resource Group]]
*[[How to create Azure Route Table]]
*[[How to create Azure Storage Account]]
*[[How to create Azure Virtual Network (VNet)]]
*[[How to create HPC Cluster based on Azure CycleCloud]]
*[[How to create Windows Virtual Machine and perform login using RDP]]
*[[How to install Wordpress server based on Azure Container Instances]]
*[[How to install Wordpress server based on Azure Web App]]
*[[How to mount Azure Blob Storage inside a Linux machine]]
*[[How to run MATLAB Parallel Server on Azure]]
*[[Using Azure CLI for managing Azure resources]]
*[[Using PowerShell for managing Azure resources]]

How to run MATLAB Parallel Server on AWS

2020-08-31T13:04:48Z

Eyales:

== Pre-Requirements ==
* AWS Account
* AWS Full Administrator IAM Role

== MATLAB License ==
* Locate the institution MATLAB License Administrator, through the License Centre:
: https://www.mathworks.com/licensecenter/licenses
* Log into your account (if prompted)
* Click on license number
* Click on "Contact Administrators" tab
* Send an email to the MATLAB License Administrator, and ask the license administrator to add the researcher's institution email address, a license for "MATLAB Parallel Server"
* To verify that "MATLAB Parallel Server" license was assigned, go to the link below:
: https://www.mathworks.com/licensecenter/

== Deploy MATLAB Parallel Server on AWS ==
* Open a web browser
* Follow the deployment instructions in the link below:
: https://github.com/mathworks-ref-arch/matlab-parallel-server-on-aws/blob/master/releases/R2019b/README.md

== Delete an existing MATLAB Parallel Server ==
* Open a web browser
* Follow the instructions in the link below:
: https://github.com/mathworks-ref-arch/matlab-parallel-server-on-aws/blob/master/releases/R2019b/README.md#delete-your-cloud-resources

How to run MATLAB Parallel Server on AWS

2020-08-31T13:03:07Z

Eyales:

== Pre-Requirement ==
* AWS Account
* AWS Full Administrator IAM Role

== MATLAB License ==
* Locate the institution MATLAB License Administrator, through the License Centre:
: https://www.mathworks.com/licensecenter/licenses
* Log into your account (if prompted)
* Click on license number
* Click on "Contact Administrators" tab
* Send an email to the MATLAB License Administrator, and ask the license administrator to add the researcher's institution email address, a license for "MATLAB Parallel Server"
* To verify that "MATLAB Parallel Server" license was assigned, go to the link below:
: https://www.mathworks.com/licensecenter/

== Deploy MATLAB Parallel Server on AWS ==
* Open a web browser
* Follow the deployment instructions in the link below:
: https://github.com/mathworks-ref-arch/matlab-parallel-server-on-aws/blob/master/releases/R2019b/README.md

== Delete an existing MATLAB Parallel Server ==
* Open a web browser
* Follow the instructions in the link below:
: https://github.com/mathworks-ref-arch/matlab-parallel-server-on-aws/blob/master/releases/R2019b/README.md#delete-your-cloud-resources

How to run MATLAB Parallel Server on AWS

2020-08-31T13:02:50Z

Eyales: Created page with "== How to run MATLAB Parallel Server on AWS == == Pre-Requirement == * AWS Account * AWS Full Administrator IAM Role == MATLAB License == * Locate the institution MATLAB Lic..."

== How to run MATLAB Parallel Server on AWS ==

== Pre-Requirement ==
* AWS Account
* AWS Full Administrator IAM Role

== MATLAB License ==
* Locate the institution MATLAB License Administrator, through the License Centre:
: https://www.mathworks.com/licensecenter/licenses
* Log into your account (if prompted)
* Click on license number
* Click on "Contact Administrators" tab
* Send an email to the MATLAB License Administrator, and ask the license administrator to add the researcher's institution email address, a license for "MATLAB Parallel Server"
* To verify that "MATLAB Parallel Server" license was assigned, go to the link below:
: https://www.mathworks.com/licensecenter/

== Deploy MATLAB Parallel Server on AWS ==
* Open a web browser
* Follow the deployment instructions in the link below:
: https://github.com/mathworks-ref-arch/matlab-parallel-server-on-aws/blob/master/releases/R2019b/README.md

== Delete an existing MATLAB Parallel Server ==
* Open a web browser
* Follow the instructions in the link below:
: https://github.com/mathworks-ref-arch/matlab-parallel-server-on-aws/blob/master/releases/R2019b/README.md#delete-your-cloud-resources

How To

2020-08-31T13:00:47Z

Eyales:

*[[Best practices for AWS cost optimization]]
*[[Best practices for managing AWS account]]
*[[Best practices for securing AWS account]]
*[[How to add permissions to AWS resources using AWS CLI on Windows client]]
*[[How to configure MFA (Multi-Factor Authentication) for AWS IAM User]]
*[[How to configure S3 bucket]]
*[[How to connect to S3 bucket using Windows client]]
*[[How to create a new security group]]
*[[How to create a route table]]
*[[How to create a subnet]]
*[[How to create a user account in AWS IAM]]
*[[How to create an EBS volume]]
*[[How to create an AWS Managed Microsoft AD directory]]
*[[How to create Amazon EC2 instance and perform login using SSH]]
*[[How to create Amazon Virtual Private Cloud (VPC)]]
*[[How to create AWS ParallelCluster with Slurm scheduler]]
*[[How to create budget and billing alerts]]
*[[How to create Windows based Amazon EC2 instance from the AWS Marketplace]]
*[[How to mount Amazon S3 Storage inside a Linux machine]]
*[[How to register for the AWSome Day]]
*[[How to run MATLAB Parallel Server on AWS]]
*[[How to set up Amazon FSx for Windows File Server]]
*[[How to sync files to Amazon Glacier]]
*[[Recommendations for configuring an AWS linked account]]
*[[Using AWS CLI for managing AWS Resources]]
*[[Using PowerShell for managing AWS resources]]

Using AWS CLI for managing AWS Resources

2020-06-24T05:14:47Z

Eyales:

== Installing AWS CLI ==
* Login to the machine using privileged account.
* Download the latest build of AWS CLI.
:* Windows download instruction and location:
:: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-windows.html
:* Linux download instruction and location:
:: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html

== How to configure AWS Account and Access Keys ==
* Login to the IAM Console:
: https://console.aws.amazon.com/iam/
* From the left pane, click on Users -> click on “Add user” -> specify the user name -> access type: “Programmatic access” -> do not select “AWS Management Console access” -> click “Next: Permissions”
* From the “add user to group”, either select existing group or click on “Create group” -> click “Next: Review” -> click on “Create user”
* Download the CSV file with the “Access key ID” and “Secret access key” and save the CSV file in a secure location
* Click Close

== Configuring the AWS CLI ==
* Run the command below in-order to configure AWS CLI:
: '''aws configure –profile <profile_name>'''
:* '''Profile name''' – set relevant target profile name
:* '''AWS Access Key ID''' - Specify the value from the CSV of the previously created IAM user
:* '''AWS Secret Access Key''' - Specify the value from the CSV of the previously created IAM user
:* '''Default region name''' - Specify a region such as '''eu-west-1'''
:: Full list: https://docs.aws.amazon.com/general/latest/gr/rande.html
:* '''Default output format''' - JSON
:: Note 1: By default, the credentials file is stored here:
::* On Windows: '''C:\Users\username\.aws\credentials'''
::* On Linux: '''~/.aws/credentials'''
:: Note 2: Reference about configuration and credential file settings can be found at:
:: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html

== Storage related commands ==
* Create a new S3 bucket:
: '''aws s3 mb <BucketName>'''
* Remove S3 bucket:
: '''aws s3 rb s3://<BucketName> --force'''
* List all S3 buckets:
: '''aws s3 ls'''
* List of S3 buckets using additional AWS IAM account:
: '''aws s3 ls --profile [profile_name]'''
* List the content of specific S3 bucket:
: '''aws s3 ls s3://<BucketName>'''
* Copy file from S3 bucket to local directory:
: '''aws s3 cp s3://<BucketName>/<FileName> /<LocalFolderName>'''

== References ==
* Installing the AWS Command Line Interface
: https://linuxacademy.com/blog/tutorials/installing-the-aws-command-line-interface/
* 28 Essential AWS S3 CLI Command Examples to Manage Buckets and Objects
: https://www.thegeekstuff.com/2019/04/aws-s3-cli-examples/

How To

2020-06-08T12:04:28Z

Eyales:

*[[Best practices for AWS cost optimization]]
*[[Best practices for managing AWS account]]
*[[Best practices for securing AWS account]]
*[[How to add permissions to AWS resources using AWS CLI on Windows client]]
*[[How to configure MFA (Multi-Factor Authentication) for AWS IAM User]]
*[[How to configure S3 bucket]]
*[[How to connect to S3 bucket using Windows client]]
*[[How to create a new security group]]
*[[How to create a route table]]
*[[How to create a subnet]]
*[[How to create a user account in AWS IAM]]
*[[How to create an EBS volume]]
*[[How to create an AWS Managed Microsoft AD directory]]
*[[How to create Amazon EC2 instance and perform login using SSH]]
*[[How to create Amazon Virtual Private Cloud (VPC)]]
*[[How to create AWS ParallelCluster with Slurm scheduler]]
*[[How to create budget and billing alerts]]
*[[How to create Windows based Amazon EC2 instance from the AWS Marketplace]]
*[[How to mount Amazon S3 Storage inside a Linux machine]]
*[[How to register for the AWSome Day]]
*[[How to set up Amazon FSx for Windows File Server]]
*[[How to sync files to Amazon Glacier]]
*[[Recommendations for configuring an AWS linked account]]
*[[Using AWS CLI for managing AWS Resources]]
*[[Using PowerShell for managing AWS resources]]

Using AWS CLI for managing AWS Resources

2020-06-08T11:57:36Z

Eyales:

== Installing AWS CLI ==
* Login to the machine using privileged account.
* Download the latest build of AWS CLI.
:* Windows download instruction and location:
:: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-windows.html
:* Linux download instruction and location:
:: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html

== How to configure AWS Account and Access Keys ==
* Login to the IAM Console:
: https://console.aws.amazon.com/iam/
* From the left pane, click on Users -> click on “Add user” -> specify the user name -> access type: “Programmatic access” -> do not select “AWS Management Console access” -> click “Next: Permissions”
* From the “add user to group”, either select existing group or click on “Create group” -> click “Next: Review” -> click on “Create user”
* Download the CSV file with the “Access key ID” and “Secret access key” and save the CSV file in a secure location
* Click Close

== Configuring the AWS CLI ==
* Run the command below in-order to configure AWS CLI:
: '''aws configure –profile <profile_name>'''
:* '''Profile name''' – set relevant target profile name
:* '''AWS Access Key ID''' - Specify the value from the CSV of the previously created IAM user
:* '''AWS Secret Access Key''' - Specify the value from the CSV of the previously created IAM user
:* '''Default region name''' - Specify a region such as '''eu-west-1'''
:: Full list: https://docs.aws.amazon.com/general/latest/gr/rande.html
:* '''Default output format''' - JSON
:: Note 1: By default, the credentials file is stored here:
::* On Windows: '''C:\Users\username.aws\credentials'''
::* On Linux: '''~/.aws/credentials'''
:: Note 2: Reference about configuration and credential file settings can be found at:
:: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html

== Storage related commands ==
* Create a new S3 bucket:
: '''aws s3 mb <BucketName>'''
* Remove S3 bucket:
: '''aws s3 rb s3://<BucketName> --force'''
* List all S3 buckets:
: '''aws s3 ls'''
* List of S3 buckets using additional AWS IAM account:
: '''aws s3 ls --profile [profile_name]'''
* List the content of specific S3 bucket:
: '''aws s3 ls s3://<BucketName>'''
* Copy file from S3 bucket to local directory:
: '''aws s3 cp s3://<BucketName>/<FileName> /<LocalFolderName>'''

== References ==
* Installing the AWS Command Line Interface
: https://linuxacademy.com/blog/tutorials/installing-the-aws-command-line-interface/
* 28 Essential AWS S3 CLI Command Examples to Manage Buckets and Objects
: https://www.thegeekstuff.com/2019/04/aws-s3-cli-examples/

Using AWS CLI for managing AWS Resources

2020-06-08T11:54:48Z

Eyales: Created page with "== Installing AWS CLI == * Login to the machine using privileged account. * Download the latest build of AWS CLI. :* Windows download instruction and location: :: https://docs..."

== Installing AWS CLI ==
* Login to the machine using privileged account.
* Download the latest build of AWS CLI.
:* Windows download instruction and location:
:: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-windows.html
:* Linux download instruction and location:
:: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html

== How to configure AWS Account and Access Keys ==
* Login to the IAM Console:
: https://console.aws.amazon.com/iam/
* From the left pane, click on Users -> click on “Add user” -> specify the user name -> access type: “Programmatic access” -> do not select “AWS Management Console access” -> click “Next: Permissions”
* From the “add user to group”, either select existing group or click on “Create group” -> click “Next: Review” -> click on “Create user”
* Download the CSV file with the “Access key ID” and “Secret access key” and save the CSV file in a secure location
* Click Close

== Configuring the AWS CLI ==
* Run the command below in-order to configure AWS CLI:
: '''aws configure –profile <profile_name>'''
:* '''Profile name''' – set relevant target profile name
:* '''AWS Access Key ID''' - Specify the value from the CSV of the previously created IAM user
:* '''AWS Secret Access Key''' - Specify the value from the CSV of the previously created IAM user
:* '''Default region name''' - Specify a region such as '''eu-west-1'''
:: Full list: https://docs.aws.amazon.com/general/latest/gr/rande.html
:* '''Default output format''' - JSON
:: Note 1: By default, the credentials file is stored here:
::* On Windows: '''C:\Users\username.aws\credentials'''
::* On Linux: '''~/.aws/credentials'''
:: Note 2: Reference about configuration and credential file settings can be found at:
:: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html

How To

2020-06-08T11:34:42Z

Eyales:

*[[AWS CLI Cheat Sheet]]
*[[Best practices for AWS cost optimization]]
*[[Best practices for managing AWS account]]
*[[Best practices for securing AWS account]]
*[[How to add permissions to AWS resources using AWS CLI on Windows client]]
*[[How to configure MFA (Multi-Factor Authentication) for AWS IAM User]]
*[[How to configure S3 bucket]]
*[[How to connect to S3 bucket using Windows client]]
*[[How to create a new security group]]
*[[How to create a route table]]
*[[How to create a subnet]]
*[[How to create a user account in AWS IAM]]
*[[How to create an EBS volume]]
*[[How to create an AWS Managed Microsoft AD directory]]
*[[How to create Amazon EC2 instance and perform login using SSH]]
*[[How to create Amazon Virtual Private Cloud (VPC)]]
*[[How to create AWS ParallelCluster with Slurm scheduler]]
*[[How to create budget and billing alerts]]
*[[How to create Windows based Amazon EC2 instance from the AWS Marketplace]]
*[[How to mount Amazon S3 Storage inside a Linux machine]]
*[[How to register for the AWSome Day]]
*[[How to set up Amazon FSx for Windows File Server]]
*[[How to sync files to Amazon Glacier]]
*[[Recommendations for configuring an AWS linked account]]
*[[Using AWS CLI for managing AWS Resources]]
*[[Using PowerShell for managing AWS resources]]

Using PowerShell to manage GCP resources

2020-06-06T13:13:57Z

Eyales:

== How to configure PowerShell for managing GCP resources (Windows platform) ==
* Login to the machine using privileged account.
* Follow the instructions below to install the latest build of PowerShell:
: https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell?view=powershell-7
* From command prompt, run the command below to invoke PowerShell:
: '''pwsh'''
: Note: You need to run cmd.exe or pwsh.exe as administrator.
* Run the command below to find out the current PowerShell version:
: '''$PSVersionTable.PSVersion'''
* In-case you currently have version older than 5.1, follow the article below to locate the download URL for upgrading to the latest version of PowerShell:
: https://docs.microsoft.com/en-us/powershell/scripting/install/migrating-from-windows-powershell-51-to-powershell-7?view=powershell-7
* Run the command below to install Cloud SDK tools for PowerShell:
: '''Install-Module -Name GoogleCloud -Force'''
* List all available cmdlets of Google Cloud PowerShell SDK:
: '''Get-Command -CommandType Cmdlet -Module GoogleCloud*'''
* Install Google Cloud SDK, as instructed below:
: https://cloud.google.com/sdk/docs/quickstart-windows
* Run the following from command prompt to initialize the Cloud SDK:
: '''gcloud init --console-only'''
* Select a GCP project from the list
* Select a default Compute region and zone

== How to configure PowerShell for managing GCP resources (RHEL/CentOS platform) ==
* Login to the machine using privileged account.
* Run the command below to register the RedHat 7 or CentOS 7 repository:
: '''curl https://packages.microsoft.com/config/rhel/7/prod.repo | sudo tee /etc/yum.repos.d/microsoft.repo'''
* Run the command below to install PowerShell:
: '''sudo yum install -y powershell'''
* From command prompt, run the command below to invoke PowerShell:
: '''sudo pwsh'''
* Run the command below to find out the current PowerShell version:
: '''$PSVersionTable.PSVersion'''
* Run the command below to install Cloud SDK tools for PowerShell:
: '''Install-Module -Name GoogleCloud -Force'''
* List all available cmdlets of Google Cloud PowerShell SDK:
: '''Get-Command -CommandType Cmdlet -Module GoogleCloud*'''
* Install Google Cloud SDK, as instructed below:
: https://cloud.google.com/sdk/docs/quickstart-redhat-centos
* Run the following from command prompt to initialize the Cloud SDK:
: '''gcloud init --console-only'''
* Select a GCP project from the list
* Select a default Compute region and zone

== How to configure PowerShell for managing GCP resources (Ubuntu 18.04 platform) ==
* Login to the machine using privileged account.
* Run the command below to register the Ubuntu 18.04 repository:
: '''wget -q https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb'''
: '''sudo dpkg -i packages-microsoft-prod.deb'''
: '''sudo apt-get update'''
: '''sudo add-apt-repository universe'''
* Run the command below to install PowerShell:
: '''sudo apt-get install -y powershell'''
* From command prompt, run the command below to invoke PowerShell:
: '''pwsh'''
* Run the command below to find out the current PowerShell version:
: '''$PSVersionTable.PSVersion'''
* Run the command below to install Cloud SDK tools for PowerShell:
: '''Install-Module -Name GoogleCloud -Force'''
* List all available cmdlets of Google Cloud PowerShell SDK:
: '''Get-Command -CommandType Cmdlet -Module GoogleCloud*'''
* Install Google Cloud SDK, as instructed below:
: https://cloud.google.com/sdk/docs/quickstart-debian-ubuntu
* Run the following from command prompt to initialize the Cloud SDK:
: '''gcloud init --console-only'''
* Select a GCP project from the list
* Select a default Compute region and zone

== Common PowerShell commands for GCP ==
* Login to Google Cloud Platform:
: '''gcloud auth login --no-launch-browser'''
* List all active GCP accounts:
: '''gcloud auth list'''
* Change the active account:
: '''gcloud config set account <Account_Name>'''
: Note: Replace <Account_Name> with the target GCP account
* Lists all available GCP projects:
: '''Get-GcpProject | select Name,ProjectId'''
* Change the GCP project:
: '''gcloud config set project “<Project_ID>”'''
: Note: Replace <Project_ID> with the target GCP project ID

== Network related commands ==
* List available networks:
: '''Get-GceNetwork | select Name,AutoCreateSubnetworks,IPv4Range,GatewayIPv4'''
* Create a new network inside a specific GCP project:
: '''New-GceNetwork -Name "my-network" -IPv4Range <CIDR_Block> -Project "my-project"'''
: Note 1: The above commands should be written as a single line
: Note 2: Replace '''my-network''' with the relevant network name
: Note 3: Replace '''<CIDR_Block>''' with the relevant value (such as 10.240.0.0/16)
: Note 4: Replace '''my-project''' with the target GCP project ID
* Delete a network inside a specific GCP project:
: '''Remove-GceNetwork -Network "my-network" -Project "my-project"'''
: Note 1: Replace '''my-network''' with the relevant network name
: Note 2: Replace '''my-project''' with the target GCP project ID

== Firewall rules related commands ==
* List all available Firewall rules inside a GCP project:
: '''Get-GceFirewall -Project "my-project" | select Name,Direction,Priority,Allowed,Denied'''
: Note: Replace '''my-project''' with the target GCP project ID
* List settings of a specific Firewall rule inside a GCP project:
: '''Get-GceFirewall "my-firewall" -Project "my-project"'''
: Note 1: Replace '''my-firewall''' with the specific firewall rule name
: Note 2: Replace '''my-project''' with the target GCP project ID
* Create a new Web allow firewall rule:
: '''New-GceFirewallProtocol tcp -Port 80 | Add-GceFirewall -Project "my-project" -Name "rule-name" -Network "my-network" -Description "Allow Web Traffic"'''
: Note 1: The above commands should be written as a single line
: Note 2: Replace '''80''' with the target port number
: Note 3: Replace '''my-project''' with the target GCP project ID
: Note 4: Replace '''rule-name''' with the target rule name
: Note 5: Replace '''my-network''' with the relevant network name
: Note 6: Replace '''"Allow Web Traffic"''' with a relevant rule description

== VM Instance related commands ==
* List all available VM instances in the current GCP project:
: '''Get-GceInstance'''
* List available VM instance image types, sorted by Name and Family:
: '''Get-GceImage | Format-Table Name,Family'''
* List all machine types:
: '''Get-GceMachineType | Format-Table Name,Zone,GuestCpus,MemoryMb,Deprecated'''
* Create a new CentOS 7 VM instance:
: '''$disk = Get-GceImage "centos-cloud" -Family "centos-7"'''
: '''$config = New-GceInstanceConfig "my-sample-vm" -MachineType "n1-standard-1" -Region europe-west2 -DiskImage $disk'''
: '''$config | Add-GceInstance -Project "my-project" -Zone "europe-west2-a"'''
: Note 1: The second command should be written as a single line
: Note 2: Replace '''centos-cloud''' with the relevant Image-project
: Note 3: Replace '''centos-7''' with the relevant Image-family
: Note 4: Replace '''my-sample-vm''' with the target VM instance hostname
: Note 5: Replace '''n1-standard-1''' with the relevant VM instance type, from the list:
: https://cloud.google.com/compute/docs/machine-types
: Note 6: Replace '''europe-west2''' with the target region
: Note 7: Replace '''my-project''' with the target GCP project ID
: Note 7: Replace '''europe-west2-a''' with the target zone
* Get information about specific VM instance in a specific zone:
: '''Get-GceInstance "my-instance" -Project "my-project" -Zone "europe-west2-a"'''
: Note 1: Replace '''my-instance''' with the target VM instance hostname
: Note 2: Replace '''my-project''' with the target GCP project ID
: Note 3: Replace '''europe-west2-a''' with the target zone
* Start VM instance in a specific zone:
: '''Start-GceInstance -Name "my-instance" -Project "my-project" -Zone "europe-west2-a"'''
: Note 1: Replace '''my-instance''' with the target VM instance hostname
: Note 2: Replace '''my-project''' with the target GCP project ID
: Note 3: Replace '''europe-west2-a''' with the target zone
* Restart VM instance in a specific zone:
: '''Restart-GceInstance -Name "my-instance" -Project "my-project" -Zone "europe-west2-a"'''
: Note 1: The second command should be written as a single line
: Note 2: Replace '''my-instance''' with the target VM instance hostname
: Note 3: Replace '''my-project''' with the target GCP project ID
: Note 4: Replace '''europe-west2-a''' with the target zone
* Stop VM instance in a specific zone:
: '''Stop-GceInstance -Name "my-instance" -Project "my-project" -Zone "europe-west2-a"'''
: Note 1: Replace '''my-instance''' with the target VM instance hostname
: Note 2: Replace '''my-project''' with the target GCP project ID
: Note 3: Replace '''europe-west2-a''' with the target zone

== Storage related commands ==
* List all Google Cloud Storage buckets inside the current GCP project:
: '''Get-GcsBucket -Project "<Project_ID>" | select Name'''
: Note: Replace '''<Project_ID>''' with the target GCP project ID
* Create a new Google Cloud Storage bucket:
: '''New-GcsBucket -Name "my-bucket" -Project "<Project_ID>"'''
: Note 1: Replace '''my-bucket''' with the target bucket name
: Note 2: Replace '''<Project_ID>''' with the target GCP project ID
* Remove Google Cloud Storage bucket:
: '''Remove-GcsBucket "my-bucket" -Force'''
: Note: Replace '''my-bucket''' with the target bucket name

Using Google Cloud SDK CLI Tools for managing GCP resources

2020-06-06T13:04:19Z

Eyales:

== How to install Google Cloud SDK (Windows Platform) ==
* Login to the machine using privileged account.
* Install Google Cloud SDK, as instructed below:
: https://cloud.google.com/sdk/docs/quickstart-windows
* Run the following from command prompt to initialize the Cloud SDK:
: '''gcloud init --console-only'''
* Select a GCP project from the list
* Select a default Compute region and zone

== How to install Google Cloud SDK (RHEL / CentOS Platform) ==
* Login to the machine using privileged account.
* Install Google Cloud SDK, as instructed below:
: https://cloud.google.com/sdk/docs/quickstart-redhat-centos
* Run the following command to initialize the Cloud SDK:
: '''gcloud init --console-only'''
* Select a GCP project from the list
* Select a default Compute region and zone

== How to install Google Cloud SDK (Ubuntu/Debian Platform) ==
* Login to the machine using privileged account.
* Install Google Cloud SDK, as instructed below:
: https://cloud.google.com/sdk/docs/quickstart-debian-ubuntu
* Run the following from command prompt to initialize the Cloud SDK:
: '''gcloud init --console-only'''
* Select a GCP project from the list
* Select a default Compute region and zone

== Common Google Cloud SDK CLI Commands ==
* Login to Google Cloud Platform:
: '''gcloud auth application-default login --no-launch-browser'''
* List all active GCP accounts:
: '''gcloud auth list'''
* Change the active account:
: '''gcloud config set account <Account_Name>'''
: Note: Replace '''<Account_Name>''' with the target GCP account
* Lists all available GCP projects:
: '''gcloud projects list'''
* Change the GCP project:
: '''gcloud config set project “<Project_ID>”'''
: Note: Replace '''<Project_ID>''' with the target GCP project ID

== Network related commands ==
* List available networks:
: '''gcloud compute networks list'''
* Create a new network:
: '''gcloud compute networks create my-network'''
: Note: Replace '''my-network''' with the relevant network name
* Delete a network:
: '''gcloud compute networks delete my-network'''
: Note: Replace '''my-network''' with the relevant network name
* List available subnets:
: '''gcloud compute networks subnets list'''
* Create a new subnet:
: '''gcloud compute networks subnets create my-subnet --network=my-network --range=<Subnet address prefix CIDR> --region=region-name'''
: Note 1: The above commands should be written as a single line
: Note 2: Replace '''my-subnet''' with your own subnet name (in lower-case)
: Note 3: Replace '''my-network''' with the relevant network name
: Note 4: Replace '''<Subnet address prefix CIDR>''' with the relevant value (such as 192.168.0.0/20)
: Note 5: Replace '''region-name''' with value from the list below:
: https://cloud.google.com/compute/docs/regions-zones/

== Firewall rules related commands ==
* List all available Firewall rules inside a GCP project:
: '''gcloud compute firewall-rules list'''
* List settings of a specific Firewall rule:
: '''gcloud compute firewall-rules describe default-allow-ssh'''
: Note: Replace '''default-allow-ssh''' with the target firewall rule name
* Create a new RDP allow firewall rule:
: '''gcloud compute firewall-rules create allow-rdp --allow tcp:3389 --description="Allow RDP"'''
: Note 1: The above commands should be written as a single line
: Note 2: Replace '''allow-rdp''' with your own rule name (in lower-case)
: Note 3: Replace '''3389''' with the target port number
: Note 4: Replace '''"Allow RDP"''' with your own rule description

== VM Instance related commands ==
* List all available VM instances in the current GCP project:
: '''gcloud compute instances list'''
* List available VM instance image types, sorted by Project and Family:
: '''gcloud compute images list'''
* List all machine types:
: '''gcloud compute machine-types list'''
* Create a new CentOS 7 VM instance:
: '''gcloud compute instances create my-sample-vm --image-family centos-7 --image-project centos-cloud --zone europe-west2-a'''
: Note 1: The above commands should be written as a single line
: Note 2: Replace '''my-sample-vm''' with the target VM instance hostname
: Note 3: Replace '''centos-7''' with the relevant Image-family
: Note 4: Replace '''centos-cloud''' with the relevant Image-project
: Note 5: Replace '''europe-west2-a''' with the target zone, from the list:
: https://cloud.google.com/compute/docs/regions-zones/
* Get information about specific VM instance in a specific zone:
: '''gcloud compute instances describe my-sample-vm --zone europe-west2-a'''
: Note 1: Replace '''my-sample-vm''' with the target VM instance hostname
: Note 2: Replace '''europe-west2-a''' with the target zone, from the list:
: https://cloud.google.com/compute/docs/regions-zones/
* Show VM instance power state:
: '''gcloud compute instances describe my-sample-vm --zone europe-west2-a --format='table(name,status)''''
: Note 1: The above commands should be written as a single line
: Note 2: Replace '''my-sample-vm''' with the target VM instance hostname
: Note 3: Replace '''europe-west2-a''' with the target zone, from the list:
: https://cloud.google.com/compute/docs/regions-zones/
* Start VM instance in a specific zone:
: '''gcloud compute instances start my-sample-vm --zone europe-west2-a'''
: Note 1: Replace '''my-sample-vm''' with the target VM instance hostname
: Note 2: Replace '''europe-west2-a''' with the target zone, from the list:
: https://cloud.google.com/compute/docs/regions-zones/
* Restart VM instance in a specific zone:
: '''gcloud compute instances reset my-sample-vm --zone europe-west2-a'''
: Note 1: Replace '''my-sample-vm''' with the target VM instance hostname
: Note 2: Replace '''europe-west2-a''' with the target zone, from the list:
: https://cloud.google.com/compute/docs/regions-zones/
* Stop VM instance in a specific zone:
: '''gcloud compute instances stop my-sample-vm --zone europe-west2-a'''
: Note 1: Replace '''my-sample-vm''' with the target VM instance hostname
: Note 2: Replace '''europe-west2-a''' with the target zone, from the list:
: https://cloud.google.com/compute/docs/regions-zones/

== Storage related commands ==
* List all Google Cloud Storage buckets inside the current GCP project:
: '''gsutil ls -p “<Project_ID>”'''
: Note: Replace '''<Project_ID>''' with the target GCP project ID
* Create a new Google Cloud Storage bucket:
: '''gsutil mb -c regional -l europe-west2 gs://my-bucket -p “<Project_ID>”'''
: Note 1: Replace '''europe-west2''' with the target region
: Note 2: Replace '''my-bucket''' with the target bucket name
: Note 3: Replace '''<Project_ID>''' with the target GCP project ID
* Remove Google Cloud Storage bucket:
: '''gsutil rm -r gs://my-bucket/'''
: Note: Replace '''my-bucket''' with the target bucket name

Using PowerShell for managing Azure resources

2020-06-06T13:02:06Z

Eyales:

== How to configure PowerShell for managing Azure resources (Windows platform) ==
* Login to the machine using privileged account.
* Follow the instructions below to install the latest build of PowerShell:
: https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell?view=powershell-7
* From command prompt, run the command below to invoke PowerShell:
: '''pwsh'''
: Note: You need to run cmd.exe or PowerShell.exe as administrator.
* Run the command below to find out the current PowerShell version:
: '''$PSVersionTable.PSVersion'''
* In-case you currently have version older than 5.1, follow the article below to locate the download URL for upgrading to the latest version of PowerShell:
: https://docs.microsoft.com/en-us/powershell/scripting/install/migrating-from-windows-powershell-51-to-powershell-7?view=powershell-7
* Run the command below to install Azure cmdlet for PowerShell:
: '''Install-Module -Name Az -AllowClobber -Force'''
* Run the commands below to update to the latest Azure cmdlet for PowerShell:
: '''Update-Module -Name Az -Force'''
* To view the installed versions of Az, run the command below:
: '''Get-InstalledModule -Name Az | select Name,Version'''

== How to configure PowerShell for managing Azure resources (RHEL/CentOS platform) ==
* Login to the machine using privileged account.
* Run the command below to register the RedHat 7 or CentOS 7 repository:
: '''curl https://packages.microsoft.com/config/rhel/7/prod.repo | sudo tee /etc/yum.repos.d/microsoft.repo'''
: Note: The above command should be written in a single line
* Run the command below to install PowerShell:
: '''sudo yum install -y powershell'''
* From command prompt, run the command below to invoke PowerShell:
: '''sudo pwsh'''
: Note: You need to run cmd.exe or PowerShell.exe as administrator.
* Run the command below to find out the current PowerShell version:
: '''$PSVersionTable.PSVersion'''
* Run the command below to install Azure cmdlet for PowerShell:
: '''Install-Module -Name Az -AllowClobber -Force'''
* Run the commands below to update to the latest Azure cmdlet for PowerShell:
: '''Update-Module -Name Az -Force'''
* To view the installed versions of Az, run the command below:
: '''Get-InstalledModule -Name Az | select Name,Version'''

== How to configure PowerShell for managing Azure resources (Ubuntu 18.04 platform) ==
* Login to the machine using privileged account.
* Run the command below to register the Ubuntu 18.04 repository:
: '''wget -q https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb'''
: '''sudo dpkg -i packages-microsoft-prod.deb'''
: '''sudo apt-get update'''
: '''sudo add-apt-repository universe'''
* Run the command below to install PowerShell:
: '''sudo apt-get install -y powershell'''
* From command prompt, run the command below to invoke PowerShell:
: '''pwsh'''
* Run the command below to find out the current PowerShell version:
: '''$PSVersionTable.PSVersion'''
* Run the command below to install Azure cmdlet for PowerShell:
: '''Install-Module -Name Az -AllowClobber -Force'''
* Run the commands below to update to the latest Azure cmdlet for PowerShell:
: '''Update-Module -Name Az -Force'''
* To view the installed versions of Az, run the command below:
: '''Get-InstalledModule -Name Az | select Name,Version'''

== How to configure PowerShell for managing Azure resources (Debian 10 platform) ==
* Login to the machine using privileged account.
* Run the command below to register the Debian 10 repository:
: '''wget https://packages.microsoft.com/config/debian/10/packages-microsoft-prod.deb'''
: '''sudo dpkg -i packages-microsoft-prod.deb'''
: '''sudo apt-get update'''
* Run the command below to install PowerShell:
: '''sudo apt-get install -y powershell'''
* From command prompt, run the command below to invoke PowerShell:
: '''pwsh'''
* Run the command below to find out the current PowerShell version:
: '''$PSVersionTable.PSVersion'''
* Run the command below to install Azure cmdlet for PowerShell:
: '''Install-Module -Name Az -AllowClobber -Force'''
* Run the commands below to update to the latest Azure cmdlet for PowerShell:
: '''Update-Module -Name Az -Force'''
* To view the installed versions of Az, run the command below:
: '''Get-InstalledModule -Name Az | select Name,Version'''

== Common PowerShell commands for Azure ==
* Login to an Azure account:
: '''Connect-AzAccount'''
* List available subscriptions:
: '''Get-AzSubscription'''
* Change the context to a specific Azure subscription:
: '''Set-AzContext -SubscriptionId <subscriptionid>'''
: Note: Replace <subscriptionid> with the relevant subscription ID
* Run the command below to suppress warning messages:
: '''Set-Item Env:\SuppressAzurePowerShellBreakingChangeWarnings "true"'''

== Resource group related commands ==
* List available resource groups:
: '''Get-AzResourceGroup'''
* Create a new Azure resource group:
: '''New-AzResourceGroup -Name <ResourceGroupName> -Location <Location>'''
: Note 1: Replace <ResourceGroupName> with your own relevant group name
: Note 2: Replace <Location> with the target location, from the list below:
: https://azure.microsoft.com/en-us/global-infrastructure/locations/
: Example:
: '''New-AzResourceGroup -Name RG01 -Location "West Europe"'''

== Networking related commands ==
* List available virtual networks:
: '''Get-AzVirtualNetwork'''
* List available subnets
: '''Get-AzVirtualNetwork -Name <Virtual Network Name> -ResourceGroupName <Resource Group Name> | Get-AzureRmVirtualNetworkSubnetConfig | Format-Table'''
: Note 1: The above command should be written in a single line
: Note 2: Replace <Virtual Network Name> with the relevant VNET
: Note 3: Replace <Resource Group Name> with the relevant resource group name
: Example:
: '''Get-AzVirtualNetwork -Name VNET01 -ResourceGroupName RG01 | Get-AzVirtualNetworkSubnetConfig | Format-Table'''
* Create a new virtual network and a new subnet:
: '''$subnetConfig = New-AzVirtualNetworkSubnetConfig -Name <SubnetName> -AddressPrefix <Subnet address prefix CIDR>'''

: '''New-AzVirtualNetwork -ResourceGroupName <Resource Group Name> -Location <Location> -Name <Virtual network name> -AddressPrefix <Virtual network address prefix CIDR> -Subnet $subnetConfig'''
: Note 1: The above commands should be written in a single line (for each command)
: Note 2: Replace <SubnetName> with a relevant subnet name
: Note 3: Replace <Subnet address prefix CIDR> with relevant value (see example below)
: Note 4: Replace <Resource Group Name> with relevant value (see example below)
: Note 5: Replace <Location> with the target location, from the list below:
: https://azure.microsoft.com/en-us/global-infrastructure/locations/
: Note 6: Replace <Virtual network name> with relevant value (see example below)
: Note 7: Replace <Virtual network address prefix CIDR> with relevant value (see example below)
: Example:
: '''$subnetConfig = New-AzVirtualNetworkSubnetConfig -Name mySubnet -AddressPrefix 192.168.1.0/24'''

: '''New-AzVirtualNetwork -ResourceGroupName RG01 -Location "UK West" -Name VNET01 -AddressPrefix 192.168.0.0/16 -Subnet $subnetConfig'''
* List all available network security groups:
: '''Get-AzNetworkSecurityGroup'''
* Create a new network security group:
: '''New-AzNetworkSecurityGroup -ResourceGroupName <Resource Group Name> -Location <Location> -Name <Network security group name>'''
: Note 1: The above command should be written in a single line
: Note 2: Replace <Resource Group Name> with relevant value (see example below)
: Note 3: Replace <Location> with the target location, from the list below:
: https://azure.microsoft.com/en-us/global-infrastructure/locations/
: Note 4: Replace <Network security group name> with relevant value (see example below)
: Example:
: '''New-AzNetworkSecurityGroup -ResourceGroupName RG01 -Location "UK West" -Name myNetworkSecurityGroup'''
* List all available rules inside a network security group:
: '''Get-AzNetworkSecurityGroup -ResourceGroupName <Resource Group Name> -Name <Network security group name> | Get-AzNetworkSecurityRuleConfig | Format-Table'''
: Note 1: The above command should be written in a single line
: Note 2: Replace <Resource Group Name> with relevant value (see example below)
: Note 3: Replace <Network security group name> with relevant value (see example below)
: Example:
: '''Get-AzNetworkSecurityGroup -ResourceGroupName RG01 -Name myNetworkSecurityGroup | Get-AzNetworkSecurityRuleConfig | Format-Table'''
* Create a new rule inside an existing network security group:
: '''$nsgRule = New-AzNetworkSecurityRuleConfig -Name <Security rule name> -Protocol Tcp -Direction Inbound -Priority 1000 -SourceAddressPrefix * -SourcePortRange * -DestinationAddressPrefix * -DestinationPortRange 3389 -Access Allow'''

: '''New-AzNetworkSecurityGroup -ResourceGroupName <Resource Group Name> -Location <Location> -Name <Network security group name> -SecurityRules $nsgRule -Force'''
: Note 1: The above commands should be written in a single line (for each command)
: Note 2: Replace <Security rule name> with relevant value (see example below)
: Note 3: Replace <Resource Group Name> with relevant value (see example below)
: Note 4: Replace <Location> with the target location, from the list below:
: https://azure.microsoft.com/en-us/global-infrastructure/locations/
: Note 5: Replace <Network security group name> with relevant value (see example below)
: Example:
: '''$nsgRule = New-AzNetworkSecurityRuleConfig -Name AllowRDP -Protocol Tcp -Direction Inbound -Priority 1000 -SourceAddressPrefix * -SourcePortRange * -DestinationAddressPrefix * -DestinationPortRange 3389 -Access Allow'''

: '''New-AzNetworkSecurityGroup -ResourceGroupName RG01 -Location "West Europe" -Name myNetworkSecurityGroup -SecurityRules $nsgRule –Force'''
* List available public IP addresses assigned to virtual machines:
: '''Get-AzPublicIpAddress'''

== Virtual machine related commands ==
* List available virtual machines in a subscription:
: '''Get-AzVM'''
* List available virtual machines in a specific resource group:
: '''Get-AzVM -ResourceGroupName <Resource Group Name>'''
: Note: Replace <Resource Group Name> with relevant value
* Get information about a specific virtual machine inside a resource group:
: '''Get-AzVM -ResourceGroupName <Resource group name> -Name <VM Name>'''
: Note 1: Replace <Resource Group Name> with relevant value
: Note 2: Replace <VM Name> with the relevant value
* Start a virtual machine inside a specific resource group:
: '''Start-AzVM -ResourceGroupName <Resource group name> -Name <VM Name>'''
: Note 1: Replace <Resource Group Name> with relevant value
: Note 2: Replace <VM Name> with the relevant value
* Restart a virtual machine inside a specific resource group:
: '''Restart-AzVM -ResourceGroupName <Resource group name> -Name <VM Name>'''
: Note 1: The above command should be written in a single line
: Note 2: Replace <Resource Group Name> with relevant value
: Note 3: Replace <VM Name> with the relevant value
* Stop a virtual machine inside a specific resource group:
: '''Stop-AzVM -ResourceGroupName <Resource group name> -Name <VM Name>'''
: Note 1: Replace <Resource Group Name> with relevant value
: Note 2: Replace <VM Name> with the relevant value
* List available virtual machine sizes for a region:
: '''Get-AzVMSize -Location <Location>'''
: Note: Replace <Location> with the target location, from the list below:
: https://azure.microsoft.com/en-us/global-infrastructure/locations/
* List available virtual machine image publishers for a region:
: '''Get-AzVMImagePublisher -Location <Location>'''
: Note: Replace <Location> with the target location, from the list below:
: https://azure.microsoft.com/en-us/global-infrastructure/locations/
* List available offer types for a publisher (such as Vendor)
: '''Get-AzVMImageOffer -Location <Location> -PublisherName <Publisher Name>'''
: Note 1: Replace <Location> with the target location, from the list below:
: https://azure.microsoft.com/en-us/global-infrastructure/locations/
: Note 2: Replace <Publisher Name> with a relevant value (such as MicrosoftWindowsServer or RedHat)
: Example:
: '''Get-AzVMImageOffer -Location "West Europe" -PublisherName MicrosoftWindowsServer'''
* List available virtual machine image SKU’s for a region:
: '''Get-AzVMImageSku -Location <Location> -PublisherName <Publisher Name> -Offer <Offer name>'''
: Note 1: The above command should be written in a single line
: Note 2: Replace <Location> with the target location, from the list below:
: https://azure.microsoft.com/en-us/global-infrastructure/locations/
: Note 3: Replace <Publisher Name> with the relevant value (see example below)
: Note 4: Replace <Offer Name> with the relevant value (see example below)
: Example:
: '''Get-AzVMImageSku -Location "West Europe" -PublisherName "MicrosoftWindowsServer" -Offer "WindowsServer"'''

== Storage related commands ==
* List available storage accounts:
: '''Get-AzStorageAccount | Select StorageAccountName, Location'''
* Create an empty new storage account:
: '''New-AzStorageAccount -ResourceGroupName <Resource Group Name> -AccountName <Storage account name> -Location <Location> -SkuName <Storage account type>'''
: Note 1: The above command should be written in a single line
: Note 2: Replace <Resource Group Name> with relevant value (see example below)
: Note 3: Replace <Storage account name> with a unique value between 3-24 characters (numbers and lower-case letters)
: Note 4: Replace <Location> with the target location, from the list below:
: https://azure.microsoft.com/en-us/global-infrastructure/locations/
: Note 5: Replace <Storage account type> with a value from the list below:
: https://docs.microsoft.com/en-us/powershell/module/azurerm.storage/set-azurermstorageaccount?view=azurermps-6.11.0
: Example:
: '''New-AzStorageAccount -ResourceGroupName RG01 -AccountName mystorageacct01 -Location “West Europe” -SkuName Standard_LRS'''
* Create a new storage account and a new blob storage container:
: '''$storageAccount = New-AzStorageAccount -ResourceGroupName <Resource Group Name> -Name <Storage account name> -SkuName <Storage account type> -Location <Location> -Kind Storage'''

: '''$ctx = $storageAccount.Context'''

: '''$containerName = <Container name>'''

: '''New-AzStorageContainer -Name $containerName -Context $ctx -Permission blob'''
: Note 1: The above commands should be written in a single line (for each command)
: Note 2: Replace <Resource Group Name> with relevant value (see example below)
: Note 3: Replace <Storage account name> with a unique value between 3-24 characters (numbers and lower-case letters)
: Note 4: Replace <Storage account type> with a value from the list below:
: https://docs.microsoft.com/en-us/powershell/module/azurerm.storage/set-azurermstorageaccount?view=azurermps-6.11.0
: Note 5: Replace <Location> with the target location, from the list below:
: https://azure.microsoft.com/en-us/global-infrastructure/locations/
: Note 6: Replace <Container name> with a relevant and unique value
: Example:
: '''$storageAccount = New-AzStorageAccount -ResourceGroupName RG01 -Name mystorageacct01 -SkuName Standard_LRS -Location “West Europe” -Kind Storage'''

: '''$ctx = $storageAccount.Context'''

: '''$containerName = "quickstartblobs"'''

: '''New-AzStorageContainer -Name $containerName -Context $ctx -Permission blob'''
* Remove a storage account:
: '''Remove-AzStorageAccount -ResourceGroupName <Resource Group Name> -AccountName <Storage account name> –Force'''
: Note 1: The above command should be written in a single line
: Note 2: Replace <Resource Group Name> with relevant value (see example below)
: Note 3: Replace <Storage account name> with the relevant value
: Example:
: '''Remove-AzStorageAccount -ResourceGroupName RG01 -AccountName mystorageacct01 –Force'''

Using Azure CLI for managing Azure resources

2020-06-06T12:54:12Z

Eyales:

== Installing Azure CLI ==
* Login to the machine using privileged account.
* Download the latest build of Azure CLI.
:* Windows download instruction and location:
:: https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-windows?view=azure-cli-latest
:* RHEL/CentOS download instruction and location:
:: https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-yum?view=azure-cli-latest
:* Ubuntu/Debian download instruction and location:
:: https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-apt?view=azure-cli-latest

== Common Azure CLI commands ==
* Login to an Azure account, from command prompt:
: '''az login'''
* List available subscriptions:
: '''az account list --output table'''
* Change the context to a specific Azure subscription:
: '''az account set --subscription "My Subscription"'''
: Note: Replace '''“My Subscription”''' with the relevant subscription name
* Run the command below to verify the currently selected Azure subscription:
: '''az account show'''

== Resource group related commands ==
* Create a new Azure resource group:
: '''az group create --name MyResourceGroup --location MyLocation'''
: Note 1: Replace '''MyResourceGroup''' with your own relevant group name
: Note 2: Replace '''MyLocation''' with the target location, from the list below:
: https://azure.microsoft.com/en-us/global-infrastructure/locations/
* List information about a resource group:
: '''az group show --name MyResourceGroup --output table'''
: Note: Replace '''MyResourceGroup''' with your own relevant group name

== Networking related commands ==
* List available virtual networks:
: '''az network vnet list --output table'''
* List available subnets (Run the command as a single line):
: '''az network vnet subnet list --resource-group MyResourceGroup --vnet-name MyVNet --output table'''
: Note 1: Replace '''MyResourceGroup''' with your own relevant group name
: Note 2: Replace '''MyVNet''' with the relevant VNET name
* Create a new virtual network and a new subnet (Run the command as a single line):
: '''az network vnet create --resource-group MyResourceGroup -n MyVnet --address-prefix <Virtual network address prefix CIDR> --subnet-name MySubnet --subnet-prefix <Subnet address prefix CIDR>'''
: Note 1: Replace '''MyResourceGroup''' with your own relevant group name
: Note 2: Replace '''MyVNet''' with the relevant VNET name
: Note 3: Replace '''MySubnet''' with the target subnet name
: Note 4: Replace '''<Virtual network address prefix CIDR>''' with relevant value (see example below)
: Note 5: Replace '''<Subnet address prefix CIDR>''' with relevant value (see example below)
: Example:
: '''az network vnet create --resource-group MyResourceGroup -n MyVnet --address-prefix 10.0.0.0/16 --subnet-name MySubnet --subnet-prefix 10.0.0.0/24'''
* List all available network security groups:
: '''az network nsg list --output table'''
* Create a new network security group:
: '''az network nsg create --resource-group MyResourceGroup -n MyNsg'''
: Note 1: Replace '''MyResourceGroup''' with your own relevant group name
: Note 2: Replace '''MyNsg''' with the target network security group
* List all default available rules inside a network security group (Run the command as a single line):
: '''az network nsg show --resource-group MyResourceGroup -n MyNsg --query "defaultSecurityRules[]" --output table'''
: Note 1: Replace '''MyResourceGroup''' with your own relevant group name
: Note 2: Replace '''MyNsg''' with the target network security group
* List all available rules inside a network security group (Run the command as a single line):
: '''az network nsg rule list --resource-group MyResourceGroup --nsg-name MyNsg --output table'''
: Note 1: Replace '''MyResourceGroup''' with your own relevant group name
: Note 2: Replace '''MyNsg''' with the target network security group
* Create a new RDP rule inside an existing network security group (Run the command as a single line)
: '''az network nsg rule create --resource-group MyResourceGroup --nsg-name MyNsg -n AllowRDP --priority 500 --source-address-prefixes Internet --destination-port-ranges 3389 --access Allow --protocol Tcp --description "Allow RDP"'''
: Note 1: Replace '''MyResourceGroup''' with your own relevant group name
: Note 2: Replace '''MyNsg''' with the target network security group
: Note 3: Replace '''AllowRDP''' with the relevant rule name
: Note 4: Replace '''3389''' with the relevant port number
: Note 5: Replace '''"Allow RDP"''' with the relevant rule description
* List available public IP addresses assigned to virtual machines:
: '''az network public-ip list --output table'''

== Virtual machine related commands ==
* List available virtual machines in a subscription:
: '''az vm list --output table'''
* List available virtual machines in a specific resource group:
: '''az vm list --resource-group MyResourceGroup --output table'''
: Note: Replace '''MyResourceGroup''' with your own relevant group name
* Create a Linux VM:
: '''az vm create -n MyVm --resource-group MyResourceGroup --image Centos --data-disk-sizes-gb 10 20 --size Standard_DS2_v2 --vnet-name MyVnet --subnet MySubnet --admin-username myusername --generate-ssh-keys'''
: Note 1: Replace '''MyResourceGroup''' with your own relevant group name
: Note 2: Replace '''MyVm''' with the target virtual machine hostname
: Note 3: Replace '''MyVNet''' with the relevant VNET name
: Note 4: Replace '''MySubnet''' with the target subnet name
: Note 5: Replace '''myusername''' with the relevant value
* Get information about a specific virtual machine inside a resource group:
: '''az vm show --resource-group MyResourceGroup -n MyVm'''
: Note 1: Replace '''MyResourceGroup''' with your own relevant group name
: Note 2: Replace '''MyVm''' with the target virtual machine hostname
* Show VM power state:
: '''az vm show --resource-group MyResourceGroup -n MyVm -d --query "powerState"'''
: Note 1: Replace '''MyResourceGroup''' with your own relevant group name
: Note 2: Replace '''MyVm''' with the target virtual machine hostname
* Start a virtual machine inside a specific resource group:
: '''az vm start --resource-group MyResourceGroup -n MyVm'''
: Note 1: Replace '''MyResourceGroup''' with your own relevant group name
: Note 2: Replace '''MyVm''' with the target virtual machine hostname
* Restart a virtual machine inside a specific resource group:
: '''az vm restart --resource-group MyResourceGroup -n MyVm'''
: Note 1: Replace '''MyResourceGroup''' with your own relevant group name
: Note 2: Replace '''MyVm''' with the target virtual machine hostname
* Stop a virtual machine inside a specific resource group:
: '''az vm stop --resource-group MyResourceGroup -n MyVm'''
: Note 1: Replace '''MyResourceGroup''' with your own relevant group name
: Note 2: Replace '''MyVm''' with the target virtual machine hostname
* List available virtual machine sizes for a region:
: '''az vm list-sizes -l MyLocation --output table'''
: Note: Replace '''MyLocation''' with the target location, from the list below:
: https://azure.microsoft.com/en-us/global-infrastructure/locations/
* List available virtual machine image publishers for a region:
: '''az vm image list-publishers -l MyLocation --output table'''
: Note: Replace '''MyLocation''' with the target location, from the list below:
: https://azure.microsoft.com/en-us/global-infrastructure/locations/
* List available virtual machine image SKU’s for a region:
: '''az vm list-skus -l MyLocation --output table'''
: Note: Replace '''MyLocation''' with the target location, from the list below:
: https://azure.microsoft.com/en-us/global-infrastructure/locations/

== Storage related commands ==
* List available storage accounts:
: '''az storage account list'''
* Create an empty new storage account:
: '''az storage account create -n mystorageaccount01111 --resource-group MyResourceGroup -l MyLocation --sku Standard_LRS'''
: Note 1: Replace '''MyResourceGroup''' with your own relevant group name
: Note 2: Replace '''MyLocation''' with the target location, from the list below:
: https://azure.microsoft.com/en-us/global-infrastructure/locations/
: Note 3: Replace '''mystorageaccount01111''' with a unique storage account name between 3-24 characters (numbers and lower-case letters)
* Remove a storage account (Run as a single line):
: '''az storage account delete -n mystorageaccount01111 --resource-group MyResourceGroup'''
: Note 1: Replace '''MyResourceGroup''' with your own relevant group name
: Note 2: Replace '''mystorageaccount01111''' with the relevant storage account name

Using PowerShell for managing AWS resources

2020-06-06T12:52:06Z

Eyales:

== How to configure PowerShell for managing AWS resources (Windows platform) ==
* Login to the machine using privileged account.
* Follow the instructions below to install the latest build of PowerShell:
: https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell?view=powershell-7
* From command prompt, run the command below to invoke PowerShell:
: '''pwsh'''
: Note: You need to run cmd.exe or pwsh.exe as administrator.
* Run the command below to find out the current PowerShell version:
: '''$PSVersionTable.PSVersion'''
* In-case you currently have version older than 5.1, follow the article below to locate the download URL for upgrading to the latest version of PowerShell:
: https://docs.microsoft.com/en-us/powershell/scripting/install/migrating-from-windows-powershell-51-to-powershell-7?view=powershell-7
* Run the commands below to install AWS tools for PowerShell:
: '''Install-Module -Name AWS.Tools.Common -AllowClobber -Force'''
: '''Install-Module -Name AWS.Tools.Installer -Force'''
* Run the commands below to update to the latest AWS PowerShell module:
: '''Update-Module -Name AWS.Tools.Common -Force'''
: '''Update-Module -Name AWS.Tools.Installer -Force'''
* To view the installed versions of AWS PowerShell module, run the command below:
: '''Get-Module -Name AWS.Tools.* -List | select Name,Version'''

== How to configure PowerShell for managing AWS resources (RHEL/CentOS platform) ==
* Login to the machine using privileged account.
* Run the command below to register the RedHat 7 or CentOS 7 repository:
: '''curl https://packages.microsoft.com/config/rhel/7/prod.repo | sudo tee /etc/yum.repos.d/microsoft.repo'''
* Run the command below to install PowerShell:
: '''sudo yum install -y powershell'''
* From command prompt, run the command below to invoke PowerShell:
: '''sudo pwsh'''
* Run the command below to find out the current PowerShell version:
: '''$PSVersionTable.PSVersion'''
* Run the commands below to install AWS tools for PowerShell:
: '''Install-Module -Name AWS.Tools.Common -AllowClobber -Force'''
: '''Install-Module -Name AWS.Tools.Installer -Force'''
* Run the commands below to update to the latest AWS PowerShell module:
: '''Update-Module -Name AWS.Tools.Common -Force'''
: '''Update-Module -Name AWS.Tools.Installer -Force'''
* To view the installed versions of AWS PowerShell module, run the command below:
: '''Get-Module -Name AWS.Tools.* -List | select Name,Version'''

== How to configure PowerShell for managing AWS resources (Ubuntu 18.04 platform) ==
* Login to the machine using privileged account.
* Run the command below to register the Ubuntu 18.04 repository:
: '''wget -q https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb'''
: '''sudo dpkg -i packages-microsoft-prod.deb'''
: '''sudo apt-get update'''
: '''sudo add-apt-repository universe'''
* Run the command below to install PowerShell:
: '''sudo apt-get install -y powershell'''
* From command prompt, run the command below to invoke PowerShell:
: '''pwsh'''
* Run the command below to find out the current PowerShell version:
: '''$PSVersionTable.PSVersion'''
* Run the commands below to install AWS tools for PowerShell:
: '''Install-Module -Name AWS.Tools.Common -AllowClobber -Force'''
: '''Install-Module -Name AWS.Tools.Installer -Force'''
* Run the commands below to update to the latest AWS PowerShell module:
: '''Update-Module -Name AWS.Tools.Common -Force'''
: '''Update-Module -Name AWS.Tools.Installer -Force'''
* To view the installed versions of AWS PowerShell module, run the command below:
: '''Get-Module -Name AWS.Tools.* -List | select Name,Version'''

== How to configure PowerShell for managing AWS resources (Debian 10 platform) ==
* Login to the machine using privileged account.
* Run the command below to register the Debian 10 repository:
: '''wget https://packages.microsoft.com/config/debian/10/packages-microsoft-prod.deb'''
: '''sudo dpkg -i packages-microsoft-prod.deb'''
: '''sudo apt-get update'''
* Run the command below to install PowerShell:
: '''sudo apt-get install -y powershell'''
* From command prompt, run the command below to invoke PowerShell:
: '''pwsh'''
* Run the command below to find out the current PowerShell version:
: '''$PSVersionTable.PSVersion'''
* Run the commands below to install AWS tools for PowerShell:
: '''Install-Module -Name AWS.Tools.Common -AllowClobber -Force'''
: '''Install-Module -Name AWS.Tools.Installer -Force'''
* Run the commands below to update to the latest AWS PowerShell module:
: '''Update-Module -Name AWS.Tools.Common -Force'''
: '''Update-Module -Name AWS.Tools.Installer -Force'''
* To view the installed versions of AWS PowerShell module, run the command below:
: '''Get-Module -Name AWS.Tools.* -List | select Name,Version'''

== How to configure AWS Account and Access Keys ==
* Login to the IAM Console:
: https://console.aws.amazon.com/iam/
* From the left pane, click on Users -> click on “Add user” -> specify the user name -> access type: “Programmatic access” -> do not select “AWS Management Console access” -> click “Next: Permissions”
* From the “add user to group”, either select existing group or click on “Create group” -> click “Next: Review” -> click on “Create user”
* Download the CSV file with the “Access key ID” and “Secret access key” and save the CSV file in a secure location
* Click Close

== Managing Profiles ==
* Login to the machine using privileged account.
* From command prompt, run the command below to invoke PowerShell (Windows platform)
: '''pwsh'''
* From command prompt, run the command below to invoke PowerShell (Linux platform)
: '''pwsh'''
* Run the command below to import the AWSPowerShell.NetCore module:
: '''import-Module AWSPowerShell.NetCore'''
* Run the command below to add a new profile:
: '''Set-AWSCredential -AccessKey <AWS_Access_Key> -SecretKey <AWS_Secret_Key> -StoreAs <Profile_Name>'''
: Note 1: Replace '''<AWS_Access_Key>''' with the relevant value from the CSV file created above.
: Note 2: Replace '''<AWS_Secret_Key>''' with the relevant value from the CSV file created above.
: Note 3: Replace '''<Profile_Name>''' with your own profile name
* List all available profiles:
: '''Get-AWSCredential -ListProfileDetail'''

Reference:
:* https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html

== VPC related commands ==
* List all available VPC’s in a specific region:
: '''Get-EC2VPC -Region <Region_Name> -ProfileName <Profile_Name>'''
: Note 1: Replace '''<Region_Name>''' with the target region, from the list below:
: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html
: Note 2: Replace '''<Profile_Name>''' with your own profile name
: Example:
: '''Get-EC2VPC -Region us-east-1 -ProfileName MyProfile'''
* Create a new VPC inside a specific region:
: '''New-EC2VPC -CidrBlock <CIDR_Block> -Region <Region_Name> -ProfileName <Profile_Name>'''
: Note 1: The above command should be written in a single line
: Note 2: Replace '''<CIDR_Block>''' with the IPv4 network range for the VPC, in CIDR notation. For example, 10.0.0.0/16.
: Note 2: Replace '''<Region_Name>''' with the target region, from the list below:
: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html
: Note 3: Replace '''<Profile_Name>''' with your own profile name
: Example:
: '''New-EC2VPC -CidrBlock 10.0.0.0/16 -Region us-east-1 -ProfileName MyProfile'''

== Subnet related commands ==
* List all available subnets inside a specific region:
: '''Get-EC2Subnet -Region <Region_Name> -ProfileName <Profile_Name>'''
: Note 1: Replace '''<Region_Name>''' with the target region, from the list below:
: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html
: Note 2: Replace '''<Profile_Name>''' with your own profile name
: Example:
: '''Get-EC2Subnet -Region us-east-1 -ProfileName MyProfile'''
* Get information about specific subnet:
: '''Get-EC2Subnet -SubnetId <Subnet_ID> -Region <Region_Name> -ProfileName <Profile_Name>'''
: Note 1: The above command should be written in a single line
: Note 2: Replace '''<Subnet_ID>''' with the relevant value
: Note 3: Replace '''<Region_Name>''' with the target region, from the list below:
: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html
: Note 4: Replace '''<Profile_Name>''' with your own profile name
: Example:
: '''Get-EC2Subnet -SubnetId subnet-101ad84c -Region us-east-1 -ProfileName MyProfile'''

== Security Group related commands ==
* List all security groups in a specific region:
: '''Get-EC2SecurityGroup -Region <Region_Name> -ProfileName <Profile_Name>'''
: Note 1: Replace '''<Region_Name>''' with the target region, from the list below:
: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html
: Note 2: Replace '''<Profile_Name>''' with your own profile name
: Example:
: '''Get-EC2SecurityGroup -Region us-east-1 -ProfileName MyProfile'''
* Create a new security group inside a specific VPC:
: '''$groupid = New-EC2SecurityGroup -VpcId <VPC_Name> -GroupName <Security_Group_Name> -GroupDescription <Group_Description> -Region <Region_Name> -ProfileName <Profile_Name>'''
: Note 1: The above command should be written a single line
: Note 2: Replace '''<VPC_Name>''' with the relevant value
: Note 3: Replace '''<Security_Group_Name>''' with a unique value (up to 255 characters), as mentioned below:
: https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html
: Note 4: Replace '''<Group Description>''' with relevant value, as mentioned below:
: https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html
: Note 5: Replace '''<Region_Name>''' with the target region, from the list below:
: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html
: Note 6: Replace '''<Profile_Name>''' with your own profile name
: Example:
: '''$groupid = New-EC2SecurityGroup -VpcId "vpc-64c0c61f" -GroupName "myPSSecurityGroup" -GroupDescription "EC2-VPC from PowerShell" -Region us-east-1 -ProfileName cliuser'''

Reference:
:* https://4sysops.com/archives/create-and-view-ec2-security-groups-with-powershell/

== Key Pair related commands ==
* Create a new key pair:
: '''$myPSKeyPair = New-EC2KeyPair -KeyName <Key_Pair_Name> -Region <Region_Name> -ProfileName <Profile_Name>'''
: Note 1: The above command must be written as a single line
: Note 2: Replace '''<Key_Pair_Name>''' with your own key pair name
: Note 3: Replace '''<Region_Name>''' with the target region, from the list below:
: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html
: Note 4: Replace '''<Profile_Name>''' with your own profile name
: Example:
: '''$myPSKeyPair = New-EC2KeyPair -KeyName MyKeyPair01 -Region us-east-1 -ProfileName MyProfile'''
* View a key pair fingerprint:
: '''Get-EC2KeyPair -KeyName <Key_Pair_Name> -Region <Region_Name> -ProfileName <Profile_Name> | format-list KeyName, KeyFingerprint'''
: Note 1: The above command must be written as a single line
: Note 2: Replace '''<Key_Pair_Name>''' with your own key pair name
: Note 3: Replace '''<Region_Name>''' with the target region, from the list below:
: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html
: Note 4: Replace '''<Profile_Name>''' with your own profile name
: Example:
: '''Get-EC2KeyPair -KeyName MyKeyPair01 -Region us-east-1 -ProfileName MyProfile | format-list KeyName, KeyFingerprint'''
* Storing a private key to a file:
: '''$myPSKeyPair.KeyMaterial | Out-File -Encoding ascii <Private_Key_Name>.pem'''
: Note: Replace '''<Private_Key_Name>.pem''' with your own value

Reference:
:* https://docs.aws.amazon.com/powershell/latest/userguide/pstools-ec2-keypairs.html
:* https://docs.aws.amazon.com/powershell/latest/userguide/pstools-ec2.html

== Storage related commands ==
* List all S3 buckets inside a specific AWS account:
: '''Get-S3Bucket -ProfileName <Profile_Name>'''
: Note: Replace '''<Profile_Name>''' with your own profile name
* Get S3 bucket location:
: '''Get-S3BucketLocation -BucketName <S3_Bucket_Name> -ProfileName <Profile_Name>'''
: Note 1: Replace '''<S3_Bucket_Name>''' with the relevant S3 bucket name
: Note 2: Replace '''<Profile_Name>''' with your own profile name
* Create a new S3 bucket:
: '''New-S3Bucket -BucketName <S3_Bucket_Name> -Region <Region_Name> -ProfileName <Profile_Name>'''
: Note 1: The above command must be written as a single line
: Note 2: Replace '''<S3_Bucket_Name>''' with a unique value between 3-63 characters (numbers and lower-case letters)
: Note 3: Replace '''<Region_Name>''' with the target region, from the list below:
: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html
: Note 4: Replace '''<Profile_Name>''' with your own profile name
: Example:
: '''New-S3Bucket -BucketName mys3bucket001 -Region us-east-1 -ProfileName MyProfile'''
* Remove S3 bucket:
: '''Remove-S3Bucket -BucketName <S3_Bucket_Name> -ProfileName <Profile_Name>'''
: Note 1: Replace '''<S3_Bucket_Name>''' with the relevant S3 bucket name
: Note 2: Replace '''<Profile_Name>''' with your own profile name

Best practices for GCP Projects cost optimization

2020-04-13T17:35:01Z

Eyales: Created page with "== Unused Compute Engine Disks == * Open the GCP Management console: : https://console.cloud.google.com/compute/disks * From the upper pane, select an existing GCP Project * R..."

== Unused Compute Engine Disks ==
* Open the GCP Management console:
: https://console.cloud.google.com/compute/disks
* From the upper pane, select an existing GCP Project
* Review the list of existing Disks
* Check each Disk under "In use by" field
* In case a disk is not in use by any VM, select the disk and click Delete
* Log off GCP Management console.

== Unused Virtual machines ==
* Open the GCP Management console:
: https://console.cloud.google.com/compute/instances
* From the upper pane, select an existing GCP Project
* Review the list of existing VM instances
* In case a VM instance is shutdown, review if the VM instance is needed.
* If a VM instance is not needed, select the VM and click Delete
* Log off GCP Management console.

== Unassociated External IP Addresses ==
* Open the GCP Management console:
: https://console.cloud.google.com/networking/addresses/list
* From the upper pane, select an existing GCP Project
* Review the list of existing External IP addresses
* In case an External IP address is not needed or not assigned to any resource, check the box next to the IP address to release.
* Click Release IP address.
* Log off GCP Management console.

GCP How To

2020-04-13T17:33:29Z

Eyales:

*[[Best practices for GCP Projects cost optimization]]
*[[Best practices for securing GCP Projects]]
*[[How to configure Google Cloud Storage bucket]]
*[[How to configure MFA (Multi-Factor Authentication) for Google G Suite or Gmail account]]
*[[How to create a GCP firewall rule]]
*[[How to create a GCP subnet]]
*[[How to create a GCP route]]
*[[How to create a GCP VM instance and perform login using SSH]]
*[[How to create GCP budget and billing alerts]]
*[[How to create Google Virtual Private Cloud (VPC) Network]]
*[[How to create HPC Cluster with Slurm scheduler on Google Cloud Platform]]
*[[How to grant Google Cloud IAM role to a new member]]
*[[How to mount Google Cloud Storage inside a Linux machine]]
*[[How to sync files to Google Coldline Storage]]
*[[Using Google Cloud SDK CLI Tools for managing GCP resources]]
*[[Using PowerShell to manage GCP resources]]

Best practices for securing GCP Projects

2020-04-13T17:22:39Z

Eyales:

== Configure MFA (Multi-Factor Authentication) for any account with owner privileges ==
In-order to avoid potential compromise of credentials, it is recommended to configure multi-factor authentication for any account with project owner privilege.
* Install Google Authenticator, as instructed on:
: https://support.google.com/accounts/answer/1066447
: https://apps.apple.com/us/app/google-authenticator/id388497605
* Login to the Google Account console:
: https://myaccount.google.com/
* From the left pane, click on Security
* Under "Signing in to Google", click on 2-Step Verification
* Click on "Get started"
* Enter your Google G Suite or Gmail password
* Under "Authenticator app", click on Set Up
* Choose which phone you have and click Next
* From your mobile device, click Scan a barcode
* Scan the barcode
* Click Next
* Enter the code shown on the Google Authenticator app on your phone
* Click Verify
* Click Done

== Limit number of inbound ports ==
Allowing large number of inbound ports access GCP resources increase the chance of network breach.
Limit the number of inbound ports to required ports only and to specific resources or specific subnets.
* Login to the Firewall rules page:
: https://console.cloud.google.com//networking/firewalls/list
* From the upper pane, select an existing GCP Project
* Review the list of existing Firewall rules, specifically rules with filter "IP ranges: 0.0.0.0/0"
: Note: It is highly recommended that inbound access on SSH (port 22TCP) or RDP (port 3389TCP) will be limited to specific IP address or IP range from known source location.
* Update Firewall rules as needed.
* Log off the GCP Management console.

== Google cloud storage permissions ==
Allowing public access to Google cloud storage buckets increase the chance of data breach.
Make sure no Google cloud storage bucket is publicly accessible.
* Open the Cloud Storage browser:
: https://console.cloud.google.com/storage/browser
* From the upper pane, select an existing GCP Project
* Review the list of existing Cloud storage buckets
* Check each bucket under "Public access" field, and make sure no bucket is configured as "Public to internet"
* Configure buckets as needed.
* Log off GCP Management console.

Best practices for securing GCP Projects

2020-04-13T17:21:24Z

Eyales: Created page with "== Configure MFA (Multi-Factor Authentication) for any account with owner privileges == In-order to avoid potential compromise of credentials, it is recommended to configure m..."

GCP How To

2020-04-13T17:20:29Z

Eyales:

*[[Best practices for securing GCP Projects]]
*[[How to configure Google Cloud Storage bucket]]
*[[How to configure MFA (Multi-Factor Authentication) for Google G Suite or Gmail account]]
*[[How to create a GCP firewall rule]]
*[[How to create a GCP subnet]]
*[[How to create a GCP route]]
*[[How to create a GCP VM instance and perform login using SSH]]
*[[How to create GCP budget and billing alerts]]
*[[How to create Google Virtual Private Cloud (VPC) Network]]
*[[How to create HPC Cluster with Slurm scheduler on Google Cloud Platform]]
*[[How to grant Google Cloud IAM role to a new member]]
*[[How to mount Google Cloud Storage inside a Linux machine]]
*[[How to sync files to Google Coldline Storage]]
*[[Using Google Cloud SDK CLI Tools for managing GCP resources]]
*[[Using PowerShell to manage GCP resources]]

Best practices for Azure cost optimization

2020-04-13T16:58:20Z

Eyales: Created page with "== Low Utilization virtual machines == * Login to the Azure Portal: : https://portal.azure.com/ * From the upper search pane, write "Virtual machines" * Select an existing vir..."

== Low Utilization virtual machines ==
* Login to the Azure Portal:
: https://portal.azure.com/
* From the upper search pane, write "Virtual machines"
* Select an existing virtual machine
* From the overview pane, change “Show data for last” to 7 days
* If the CPU utilization is less than 3% and network utilization is less than 2%, the selected virtual machine qualifies as candidate for an idle instance.
* Decide if you wish to delete the virtual machine.
* To delete the virtual machine -> from the upper pane, click on Delete and confirm the action.
* Log off the Azure portal

== Unassociated public IP Addresses ==
* Login to the Azure Portal:
: https://portal.azure.com/
* From the upper search pane, write "Public IP addresses"
* Select an existing public IP address from the list
* From the overview pane, check if the “Associate to” field is empty.
* If the IP is not associated to any resource, from the upper pane, click on Delete and confirm the action.
* Log off the Azure portal

Azure - How To

2020-04-13T16:57:18Z

Eyales:

*[[Best practices for Azure cost optimization]]
*[[Best practices for securing Azure subscriptions]]
*[[How to configure MFA (Multi-Factor Authentication) for AD Azure account]]
*[[How to create a file share in Azure Files]]
*[[How to create Azure Virtual Machine and perform login using SSH]]
*[[How to create Azure Network Security Group]]
*[[How to create Azure Network Interface]]
*[[How to create Azure Resource Group]]
*[[How to create Azure Route Table]]
*[[How to create Azure Storage Account]]
*[[How to create Azure Virtual Network (VNet)]]
*[[How to create HPC Cluster based on Azure CycleCloud]]
*[[How to create Windows Virtual Machine and perform login using RDP]]
*[[How to install Wordpress server based on Azure Container Instances]]
*[[How to install Wordpress server based on Azure Web App]]
*[[How to mount Azure Blob Storage inside a Linux machine]]
*[[Using Azure CLI for managing Azure resources]]
*[[Using PowerShell for managing Azure resources]]

Best practices for securing Azure subscriptions

2020-04-13T16:35:30Z

Eyales:

== Configure MFA (Multi-Factor Authentication) for any account with owner privileges ==
In-order to avoid potential compromise of credentials, it is recommended to configure multi-factor authentication for any account with owner privilege.
* Install Microsoft Authenticator app on your mobile device, as instructed:
: https://docs.microsoft.com/en-us/azure/active-directory/user-help/multi-factor-authentication-end-user-manage-settings#add-or-change-your-phone-number
* Login to the Azure Portal:
: https://portal.azure.com/
* From the top right pane, click on your username
* Click on View account
* Under "Manage account", click on "Additional security verification"
* Under "How would you like to respond", click on "Set up Authenticator app"
* Follow the on-screen instructions, including using your mobile device to scan the QR code, and then select Next
* You'll be asked to approve a notification through the Microsoft Authenticator app, to verify your information.
* Select Save

== Limit number of inbound ports ==
Allowing large number of inbound ports access Azure resources increase the chance of network breach.
Limit the number of inbound ports to required ports only and to specific resources or specific subnets.
* Login to the Azure Portal:
: https://portal.azure.com
* From the upper search pane, write "Network Security Groups"
* From the main pane, select an existing Network Security Group
* From the main pane, click on Inbound security rules
* Review all inbound rules
: Note: It is highly recommended that inbound access on SSH (port 22TCP) or RDP (port 3389TCP) will be limited to specific IP address or IP range from known source location.
* Update the Network Security Group as needed
* Save the Network Security Group
* Log off the Azure portal

== SQL Server Access Restricted ==
Allowing unnecessary inbound access to Azure SQL Server increase the chance of network breach.
Limit the inbound access to your Azure SQL servers to required sources only.
* Login to the Azure Portal:
: https://portal.azure.com
* From the upper search pane, write "Azure SQL"
* For each SQL server
* Click on Firewall / Virtual Networks
* Ensure that the firewall rules exist, and no rule has Start IP of 0.0.0.0 and End IP of 0.0.0.0 or other combinations which allows access to wider public IP ranges
* Configure the source CIDR/IP to the required subnet or required IP address.
* Log off the Azure portal

== Storage Blob Container Public Access ==
Allowing public access to storage blob containers increase the chance of data breach.
Make sure no storage blob container is publicly accessible.
* Login to the Azure Portal:
: https://portal.azure.com
* From the upper search pane, write "Storage accounts"
* From the main pane, select a storage account from the list
* For each storage account, go to Containers under BLOB SERVICE
* For each container, click Access policy
* Ensure that Public access level is set to Private (no anonymous access)
* Log off the Azure portal

Best practices for securing Azure subscriptions

2020-04-13T16:32:54Z

Eyales:

== Configure MFA (Multi-Factor Authentication) for any account with owner privileges ==
In-order to avoid potential compromise of credentials, it is recommended to configure multi-factor authentication for any account with owner privilege.
* Install Microsoft Authenticator app on your mobile device, as instructed:
: https://docs.microsoft.com/en-us/azure/active-directory/user-help/multi-factor-authentication-end-user-manage-settings#add-or-change-your-phone-number
* Login to the Azure Portal:
: https://portal.azure.com/
* From the top right pane, click on your username
* Click on View account
* Under "Manage account", click on "Additional security verification"
* Under "How would you like to respond", click on "Set up Authenticator app"
* Follow the on-screen instructions, including using your mobile device to scan the QR code, and then select Next
* You'll be asked to approve a notification through the Microsoft Authenticator app, to verify your information.
* Select Save

== Limit number of inbound ports ==
Allowing large number of inbound ports access Azure resources increase the chance of network breach.
Limit the number of inbound ports to required ports only and to specific resources or specific subnets.
* Login to the Azure Portal:
: https://portal.azure.com
* From the upper search pane, write "Network Security Groups"
* From the main pane, select an existing Network Security Group
* From the main pane, click on Inbound security rules
* Review all inbound rules
: Note: It is highly recommended that inbound access on SSH (port 22TCP) or RDP (port 3389TCP) will be limited to specific IP address or IP range from known source location.
* Update the Network Security Group as needed
* Save the Network Security Group
* Log off the Azure portal

== SQL Server Access Restricted ==
Allowing unnecessary inbound access to Azure SQL Server increase the chance of network breach.
Limit the inbound access to your Azure SQL servers to required sources only.
* Login to the Azure Portal:
: https://portal.azure.com
* From the upper search pane, write "Azure SQL"
* For each SQL server
* Click on Firewall / Virtual Networks
* Ensure that the firewall rules exist, and no rule has Start IP of 0.0.0.0 and End IP of 0.0.0.0 or other combinations which allows access to wider public IP ranges
* Configure the source CIDR/IP to the required subnet or required IP address.
* Log off the Azure portal

Best practices for securing Azure subscriptions

2020-04-13T16:31:15Z

Azure - How To

2020-04-13T16:30:21Z

Eyales:

*[[Best practices for securing Azure subscriptions]]
*[[How to configure MFA (Multi-Factor Authentication) for AD Azure account]]
*[[How to create a file share in Azure Files]]
*[[How to create Azure Virtual Machine and perform login using SSH]]
*[[How to create Azure Network Security Group]]
*[[How to create Azure Network Interface]]
*[[How to create Azure Resource Group]]
*[[How to create Azure Route Table]]
*[[How to create Azure Storage Account]]
*[[How to create Azure Virtual Network (VNet)]]
*[[How to create HPC Cluster based on Azure CycleCloud]]
*[[How to create Windows Virtual Machine and perform login using RDP]]
*[[How to install Wordpress server based on Azure Container Instances]]
*[[How to install Wordpress server based on Azure Web App]]
*[[How to mount Azure Blob Storage inside a Linux machine]]
*[[Using Azure CLI for managing Azure resources]]
*[[Using PowerShell for managing Azure resources]]

Best practices for AWS cost optimization

2020-04-13T15:54:13Z

Eyales:

== Unused EBS Volumes ==
* Login to the AWS Management Console
: https://console.aws.amazon.com/ec2/
* In the navigation panel, under Elastic Block Store, click Volumes.
* To identify any unattached EBS volumes, check their status under State column. If the status is available, the volume is not attached to an EC2 instance and can be safely deleted.
* Select the unused EBS volume -> click on Actions -> Delete volume
* Change the AWS region from the navigation bar and repeat the process for the other regions.
* Log off the AWS Management console.

== Low Utilization Amazon EC2 Instances ==
* Sign in to the AWS Management Console
: https://console.aws.amazon.com/ec2/
* In the left navigation panel, under INSTANCES section, choose Instances.
* Select the EC2 instance that you want to examine.
* Select the Monitoring tab from the dashboard bottom panel.
* Within the CloudWatch metrics section, perform the following actions:
:* Click on the CPU Utilization (Percent) usage graph thumbnail to open the instance CPU usage details box. Inside the CloudWatch Monitoring Details dialog box, set the following parameters:
::* From the Statistic dropdown list, select Average.
::* From the Time Range list, select Last 1 Week.
::* From the Period dropdown list, select 1 Hour.
:* Once the monitoring data is loaded, verify the instance CPU usage for the last 7 days. If the average usage (percent) has been less than 2%, e.g. , the selected EC2 instance qualifies as candidate for an idle instance. Click Close to return to the dashboard.
* Decide if you wish to terminate (delete) the EC2 instance.
* To terminate the EC2 instance -> from Actions -> select Instance State -> select Terminate -> choose Yes, Terminate
* Change the AWS region from the navigation bar and repeat the process for the other regions.
* Log off the AWS Management console.

== Unassociated Elastic IP Addresses ==
* Sign in to the AWS Management Console
: https://console.aws.amazon.com/vpc/
* In the left navigation panel, under Virtual Private Cloud section, choose Elastic IPs.
* Select Unassociated from the Filter dropdown menu to filter all the available EIPs and return the unattached ones. The filtering process should return the Elastic IPs that are not currently associated with any running EC2 instances or Elastic Network Interfaces (ENIs). The unattached EIPs returned at this step can be safely released (see Remediation/Resolution section).
* Change the AWS region from the navigation bar and repeat the process for the other regions.
* Log off the AWS Management console.

== Underutilized Amazon Redshift Clusters ==
* Login to the AWS Management Console
: https://console.aws.amazon.com/redshift/
* In the left navigation panel, under Redshift Dashboard, click Clusters.
* Choose the Redshift cluster that you want to examine then click on its identifier link listed in the Cluster column.
* On the cluster settings page, select the Performance tab to access the monitoring panel.
* On the monitoring panel displayed for the selected cluster, perform the following actions:
:* To verify the Redshift cluster Database Connections usage graph, follow the steps below:
::* From the Time Range dropdown list, select Last 1 Week.
::* From the Period list, select 1 Hour.
::* From the Statistic dropdown list, select Average.
::* And from the Metrics dropdown list, select DatabaseConnections.
:* Once the monitoring data is loaded into the Database Connections usage graph, check the number of database connections for the last 7 days. If the average usage (count) has been less than 1, e.g., the selected Redshift cluster qualifies as candidate for the idle cluster.
* If you no longer need your RedShift cluster, you can delete it.
* On the navigation menu, choose CLUSTERS.
* Choose the cluster to delete.
* For Actions, choose Delete. The Delete cluster page appears.
* Choose Delete cluster.
* Change the AWS region from the navigation bar and repeat the process for the other regions.
* Log off the AWS Management console.

Best practices for AWS cost optimization

2020-04-13T15:51:42Z

Eyales: Created page with "== Unused EBS Volumes == * Login to the AWS Management Console : https://console.aws.amazon.com/ec2/ * In the navigation panel, under Elastic Block Store, click Volumes. * To..."

How To

2020-04-13T15:49:06Z

Eyales:

Best practices for securing AWS account

2020-04-13T15:02:27Z

Eyales:

== Changing IAM user password ==
In-order to avoid potential compromise of credentials, it is recommended to replace every IAM user’s password, every 90 days.
* Sign in to the AWS Management Console and open the IAM console at:
: https://console.aws.amazon.com/iam/
* In the navigation pane, choose Users.
* Choose the name of the user whose password you want to change.
* Choose the Security credentials tab, and then under Sign-in credentials, choose Manage password next to Console password.
* Choose whether to have IAM generate a password or create a custom password:
:* To have IAM generate a password, choose Autogenerated password.
:* To create a custom password, choose Custom password, and type the password.
* To require the user to create a new password when signing in, choose Require password reset. Then choose Apply.
* If you choose the option to generate a password, choose Show in the New password dialog box. This lets you view the password so you can share it with the user.
* Log off the AWS Management Console.

== Changing IAM user access keys ==
In-order to avoid potential compromise of credentials, it is recommended to replace every IAM user’s access keys, every 90 days.
* While the first access key is still active, create a second access key.
:* Sign in to the AWS Management Console and open the IAM console at
:: https://console.aws.amazon.com/iam/
:* In the navigation pane, choose Users.
:* Choose the name of the intended user, and then choose the Security credentials tab.
:* Choose Create access key and then choose Download .csv file to save the access key ID and secret access key to a .csv file on your computer. Store the file in a secure location. You will not have access to the secret access key again after this closes. After you have downloaded the .csv file, choose Close.
* Update all applications and tools to use the new access key.
* Determine whether the first access key is still in use by reviewing the Last used column for the oldest access key. One approach is to wait several days and then check the old access key for any use before proceeding.
* Choose Make inactive to deactivate the first access key.
* Use only the new access key to confirm that your applications are working.
* After you wait some period of time to ensure that all applications and tools have been updated, you can delete the first access key:
:* Sign in to the AWS Management Console and open the IAM console at
:: https://console.aws.amazon.com/iam/
:* In the navigation pane, choose Users.
:* Choose the name of the intended user, and then choose the Security credentials tab.
:* Locate the access key to delete and choose its X button at the far right of the row. Then choose Delete to confirm.
* Log off the AWS Management console.

== Limit number of inbound ports ==
Allowing large number of inbound ports access AWS resources increase the chance of network breach.
Limit the number of inbound ports to required ports only and to specific resources or specific subnets.
* Open the Amazon EC2 console at
: https://console.aws.amazon.com/ec2/
* In the navigation pane, choose Security Groups.
* Select the security group to update, and choose Inbound Rules to update a rule for inbound traffic.
* Choose Edit. Modify the rule entry as required and choose Save.
: Note: It is highly recommended that inbound access on SSH (port 22TCP) or RDP (port 3389TCP) will be limited to specific IP address or IP range from known source location.
* Log off the AWS Management console.

== Amazon EBS Public Snapshots ==
In-order to avoid data theft, it is recommended to make sure no Amazon EBS snapshots are configured with public access.
* Open the Amazon EC2 console at
: https://console.aws.amazon.com/ec2/
* In the left navigation panel, under ELASTIC BLOCK STORE section, choose Snapshots.
* Select the volume snapshot that you want to examine.
* Select Permissions tab from the dashboard bottom panel and check the snapshot access permissions. If the selected EBS volume snapshot is publicly accessible, the EC2 dashboard will display the following status: "This snapshot is currently Public."
* Change permissions so that no EBS volume snapshot is configured with public access.
* Logoff the AWS Management console.

== Amazon RDS Public Snapshots ==
In-order to avoid data theft, it is recommended to make sure no Amazon RDS snapshots are configured with public access.
* Login to the AWS Management Console.
: https://console.aws.amazon.com/rds/
* In the left navigation panel, under RDS Dashboard, click Snapshots.
* Select Manual Snapshots from the Filter dropdown menu to display only manual database snapshots.
* Select the snapshot that you want to examine.
* Click Snapshot Actions button from the dashboard top menu and select Share Snapshot option.
* On the Manage Snapshot Permissions page, check the DB Snapshot Visibility setting. If the setting value is set to Public, the selected Amazon RDS database snapshot is publicly accessible, therefore all AWS accounts and users have access to the data available on the snapshot.
* Change permissions so that no Amazon RDS snapshot is configured with public access.
* Logoff the AWS Management console.

== Amazon RDS Security Group Access Risk ==
Allowing unnecessary inbound access to Amazon RDS resources increase the chance of network breach.
Limit the inbound access to your RDS instance to required sources only.
* Login to the AWS Management Console
: https://console.aws.amazon.com/rds/
* In the navigation panel, under RDS Dashboard, click Security Groups.
* Select the DB security group that you want to examine and click on the details button (magnifying glass icon).
* Check the CIDR/IP value listed in the Details column for each authorized connection. If the security group contains any rules that have set the CIDR/IP to 0.0.0.0/0 and the Status to authorized, the selected DB security group configuration is insecure and does not restrict access to the database instance(s).
* Configure the source CIDR/IP to the required subnet or required IP address.
* Log off the AWS Management console.

== Amazon S3 Bucket Permissions ==
Allowing public access to S3 buckets increase the chance of data breach.
Make sure no S3 bucket is publicly accessible.
* Sign in to the AWS Management Console
: https://console.aws.amazon.com/s3/
* Select the S3 bucket that you want to examine and click the Properties tab from the S3 dashboard top right menu.
* In the Properties panel, click the Permissions tab and check the Access Control List (ACL) for any grantee named "Everyone". A grantee can be an AWS account or an AWS S3 predefined group. The grantee called "Everyone" is an AWS predefined group that allows access to everyone (i.e. anonymous users). If the bucket ACL configuration does specify the "Everyone" predefined group with the List (READ) permission enabled, the selected S3 bucket is publicly accessible for content listing and is rendered as insecure.
* Make sure no S3 bucket is publicly accessible.
* Log off the AWS Management console.

Best practices for securing AWS account

2020-04-13T15:01:22Z

Eyales:

Best practices for securing AWS account

2020-04-13T14:58:35Z

Eyales:

Best practices for securing AWS account

2020-04-13T14:56:22Z

Eyales:

Best practices for securing AWS account

2020-04-13T14:54:17Z

Eyales:

Best practices for securing AWS account

2020-04-13T14:51:09Z

Eyales: Created page with "== Changing IAM user password == In-order to avoid potential compromise of credentials, it is recommended to replace every IAM user’s password, every 90 days. * Sign in to t..."

How To

2020-04-13T14:49:05Z

Eyales:

*[[AWS CLI Cheat Sheet]]
*[[Best practices for managing AWS account]]
*[[Best practices for securing AWS account]]
*[[How to add permissions to AWS resources using AWS CLI on Windows client]]
*[[How to configure MFA (Multi-Factor Authentication) for AWS IAM User]]
*[[How to configure S3 bucket]]
*[[How to connect to S3 bucket using Windows client]]
*[[How to create a new security group]]
*[[How to create a route table]]
*[[How to create a subnet]]
*[[How to create a user account in AWS IAM]]
*[[How to create an EBS volume]]
*[[How to create an AWS Managed Microsoft AD directory]]
*[[How to create Amazon EC2 instance and perform login using SSH]]
*[[How to create Amazon Virtual Private Cloud (VPC)]]
*[[How to create AWS ParallelCluster with Slurm scheduler]]
*[[How to create budget and billing alerts]]
*[[How to create Windows based Amazon EC2 instance from the AWS Marketplace]]
*[[How to mount Amazon S3 Storage inside a Linux machine]]
*[[How to register for the AWSome Day]]
*[[How to set up Amazon FSx for Windows File Server]]
*[[How to sync files to Amazon Glacier]]
*[[Recommendations for configuring an AWS linked account]]
*[[Using PowerShell for managing AWS resources]]

How to configure MFA (Multi-Factor Authentication) for Google G Suite or Gmail account

2020-01-22T09:09:22Z

Eyales: Created page with "* Install Google Authenticator, as instructed on: : https://support.google.com/accounts/answer/1066447 : https://apps.apple.com/us/app/google-authenticator/id388497605 * Login..."

* Install Google Authenticator, as instructed on:
: https://support.google.com/accounts/answer/1066447
: https://apps.apple.com/us/app/google-authenticator/id388497605
* Login to the Google Account console:
: https://myaccount.google.com/
* From the left pane, click on Security
* Under "Signing in to Google", click on 2-Step Verification
* Click on "Get started"
* Enter your Google G Suite or Gmail password
* Under "Authenticator app", click on Set Up
* Choose which phone you have and click Next
* From your mobile device, click Scan a barcode
* Scan the barcode
* Click Next
* Enter the code shown on the Google Authenticator app on your phone
* Click Verify
* Click Done

== Additional information: ==
: https://support.google.com/a/answer/2537800
: https://support.google.com/a/answer/175197
: https://myaccount.google.com/signinoptions/two-step-verification

GCP How To

2020-01-22T09:08:25Z

Eyales:

*[[How to configure Google Cloud Storage bucket]]
*[[How to configure MFA (Multi-Factor Authentication) for Google G Suite or Gmail account]]
*[[How to create a GCP firewall rule]]
*[[How to create a GCP subnet]]
*[[How to create a GCP route]]
*[[How to create a GCP VM instance and perform login using SSH]]
*[[How to create GCP budget and billing alerts]]
*[[How to create Google Virtual Private Cloud (VPC) Network]]
*[[How to create HPC Cluster with Slurm scheduler on Google Cloud Platform]]
*[[How to grant Google Cloud IAM role to a new member]]
*[[How to mount Google Cloud Storage inside a Linux machine]]
*[[How to sync files to Google Coldline Storage]]
*[[Using Google Cloud SDK CLI Tools for managing GCP resources]]
*[[Using PowerShell to manage GCP resources]]

How to configure MFA (Multi-Factor Authentication) for AWS IAM User

2020-01-22T09:00:40Z

Eyales: Created page with "* Install Virtual MFA application on your mobile device, as instructed: : https://aws.amazon.com/iam/features/mfa/ * Login to the AWS console: : https://console.aws.amazon.com..."

* Install Virtual MFA application on your mobile device, as instructed:
: https://aws.amazon.com/iam/features/mfa/
* Login to the AWS console:
: https://console.aws.amazon.com/iam/
* From the left pane, click on Users
* In the User Name list, select the target user
* Choose the Security credentials tab -> click Next to Assigned MFA device, choose Manage
* In the Manage MFA Device wizard, choose Virtual MFA device, and then choose Continue
: IAM generates and displays configuration information for the virtual MFA device, including a QR code graphic.
* From the mobile device, open the virtual MFA application
: If the virtual MFA app supports multiple virtual MFA devices or accounts, choose the option to create a new virtual MFA device or account
* In the Manage MFA Device wizard, in the MFA code 1 box, type the one-time password that currently appears in the virtual MFA device.
: Wait up to 30 seconds for the device to generate a new one-time password. Then type the second one-time password into the MFA code 2 box. Choose Assign MFA

== Additional information: ==
* https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa_enable_virtual.html#enable-virt-mfa-for-iam-user

How To

2020-01-22T08:59:53Z

Eyales:

*[[AWS CLI Cheat Sheet]]
*[[Best practices for managing AWS account]]
*[[How to add permissions to AWS resources using AWS CLI on Windows client]]
*[[How to configure MFA (Multi-Factor Authentication) for AWS IAM User]]
*[[How to configure S3 bucket]]
*[[How to connect to S3 bucket using Windows client]]
*[[How to create a new security group]]
*[[How to create a route table]]
*[[How to create a subnet]]
*[[How to create a user account in AWS IAM]]
*[[How to create an EBS volume]]
*[[How to create an AWS Managed Microsoft AD directory]]
*[[How to create Amazon EC2 instance and perform login using SSH]]
*[[How to create Amazon Virtual Private Cloud (VPC)]]
*[[How to create AWS ParallelCluster with Slurm scheduler]]
*[[How to create budget and billing alerts]]
*[[How to create Windows based Amazon EC2 instance from the AWS Marketplace]]
*[[How to mount Amazon S3 Storage inside a Linux machine]]
*[[How to register for the AWSome Day]]
*[[How to set up Amazon FSx for Windows File Server]]
*[[How to sync files to Amazon Glacier]]
*[[Recommendations for configuring an AWS linked account]]
*[[Using PowerShell for managing AWS resources]]

How to configure MFA (Multi-Factor Authentication) for AD Azure account

2020-01-22T08:58:34Z

Eyales: Created page with "* Install Microsoft Authenticator app on your mobile device, as instructed: : https://docs.microsoft.com/en-us/azure/active-directory/user-help/multi-factor-authentication-end..."

* Install Microsoft Authenticator app on your mobile device, as instructed:
: https://docs.microsoft.com/en-us/azure/active-directory/user-help/multi-factor-authentication-end-user-manage-settings#add-or-change-your-phone-number
* Login to the Azure Portal:
: https://portal.azure.com/
* From the top right pane, click on your username
* Click on View account
* Under "Manage account", click on "Additional security verification"
* Under "How would you like to respond", click on "Set up Authenticator app"
* Follow the on-screen instructions, including using your mobile device to scan the QR code, and then select Next
: You'll be asked to approve a notification through the Microsoft Authenticator app, to verify your information.
* Select Save

== Additional information: ==
* https://docs.microsoft.com/en-us/azure/active-directory/user-help/multi-factor-authentication-end-user-manage-settings

Azure - How To

2020-01-22T08:57:29Z

Eyales:

*[[How to configure MFA (Multi-Factor Authentication) for AD Azure account]]
*[[How to create a file share in Azure Files]]
*[[How to create Azure Virtual Machine and perform login using SSH]]
*[[How to create Azure Network Security Group]]
*[[How to create Azure Network Interface]]
*[[How to create Azure Resource Group]]
*[[How to create Azure Route Table]]
*[[How to create Azure Storage Account]]
*[[How to create Azure Virtual Network (VNet)]]
*[[How to create HPC Cluster based on Azure CycleCloud]]
*[[How to create Windows Virtual Machine and perform login using RDP]]
*[[How to install Wordpress server based on Azure Container Instances]]
*[[How to install Wordpress server based on Azure Web App]]
*[[How to mount Azure Blob Storage inside a Linux machine]]
*[[Using Azure CLI for managing Azure resources]]
*[[Using PowerShell for managing Azure resources]]

How to mount Amazon S3 Storage inside a Linux machine

2019-09-23T11:28:21Z

Eyales:

== Creating IAM user ==
* Login to the IAM console:
: https://console.aws.amazon.com/iam/
* From the left pane, click on Users -> click on “Add user” -> specify the user name -> access type: “Programmatic access” -> do not select “AWS Management Console access” -> click “Next: Permissions”
* From the “add user to group”, click on “Create group” -> from the policy list, select “AmazonS3FullAccess” -> click “Next: Review” -> click on “Create user”
* Download the CSV file with the “Access key ID” and “Secret access key” and save the CSV file in a secure location
* Click Close

== S3FS installation ==
* Login using SSH to the target Linux machine using privileged account.
* Follow the instructions below to install the S3FS Fuse adapter:
: https://github.com/s3fs-fuse/s3fs-fuse/wiki/Installation-Notes
* Run the command sudo vi /etc/passwd-s3fs to create config file with the following content:
: '''bucketName:accessKeyId:secretAccessKey'''
: Note 1: Replace '''bucketName''', with the target bucket name
: Note 2: Replace '''accessKeyId''', with the relevant value from the credentials created on the IAM console
: Note 3: Replace '''secretAccessKey''', with the relevant value from the credentials created on the IAM console
* Change the permissions of the passwd-s3fs file (Amazon Linux):
: '''sudo chown ec2-user:ec2-user /etc/passwd-s3fs'''
: '''sudo chmod 600 /etc/passwd-s3fs'''
* Change the permissions of the passwd-s3fs file (Ubuntu):
: '''sudo chown ubuntu:ubuntu /etc/passwd-s3fs'''
: '''sudo chmod 600 /etc/passwd-s3fs'''

== Mount phase ==
* Run the commands below to mount the Amazon S3 storage (Amazon Linux):
: '''sudo mkdir -p /dev/cloudstorage'''
: '''sudo chown -R ec2-user:ec2-user /dev/cloudstorage/'''
: '''sudo chmod 777 /dev/cloudstorage'''
: '''/usr/bin/s3fs MyBucket /dev/cloudstorage -o passwd_file=/etc/passwd-s3fs'''
: Note: Replace '''MyBucket''', with the target bucket name
* Run the commands below to mount the Amazon S3 storage (Ubuntu):
: '''sudo mkdir -p /dev/cloudstorage'''
: '''sudo chown -R ubuntu:ubuntu /dev/cloudstorage/'''
: '''sudo chmod 777 /dev/cloudstorage'''
: '''/usr/bin/s3fs MyBucket /dev/cloudstorage -o passwd_file=/etc/passwd-s3fs'''
: Note: Replace '''MyBucket''', with the target bucket name
* To view the content of the Amazon S3 bucket, switch to the new mount point:
: '''cd /dev/cloudstorage'''

== Unmount phase ==
* en completing the work on the bucket, from the Linux SSH console, and run the commands below to unmount the Amazon S3 Storage:
: '''cd ~'''
: '''sudo fusermount -u /dev/cloudstorage'''
* Logoff the Linux machine

== Additional references regarding permanent mount using FSTAB ==
: https://github.com/s3fs-fuse/s3fs-fuse/wiki/Fuse-Over-Amazon
: https://github.com/s3fs-fuse/s3fs-fuse
: https://www.systutorials.com/docs/linux/man/1-s3fs/

How to create AWS ParallelCluster with Slurm scheduler

2019-08-01T13:17:51Z

Eyales:

== Environment preparation phase within AWS Management Console ==
* Login the AWS IAM console:
: https://console.aws.amazon.com/iam/
* From the left pane, click on Policies -> click on Users -> Add User -> specify the name '''parallelcluster-user''' -> Access type: Programmatic access -> click Next: Permissions -> Set permissions -> select a group with '''“AdministratorAccess”''' role -> click Next: Tags -> click Next: Review -> click on Create user -> click on Download .csv and keep it in a secured location -> click on Close
* Follow the instructions below to create a key pair to access the cluster machines:
: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#having-ec2-create-your-key-pair
* Follow the instructions below to create S3 bucket (with unique name) for storing data to export and import data to/from the FSx Lustre storage:
: https://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html
: Note 1: Document the S3 bucket name for use inside the ParallelCluster config file
: Note 2: Create a folder called '''export''' (in small letters), inside the S3 bucket
* In-case you wish to create a dedicate VPC and subnet for the HPC cluster, follow the instructions below:
: https://docs.aws.amazon.com/directoryservice/latest/admin-guide/gsg_create_vpc.html
* Logoff the AWS console

== Python installation phase on Linux (Debian / Ubuntu) ==
* Login to a Linux machine using SSH, and follow the instructions below to install Python 3:
: https://docs.aws.amazon.com/cli/latest/userguide/install-linux-python.html
: Note: In-case you already have Python 3 install, use the command below to upgrade to the latest build:
: '''sudo apt-get upgrade python3'''
* To install pip3, run the command below:
: '''sudo apt install python3-pip'''

== Python installation phase on CentOS 7 ==
* Login to the CentOS machine using SSH, and follow the instructions below to install Python3 and Python3-PIP:
: https://www.rosehosting.com/blog/how-to-install-python-3-6-4-on-centos-7/

== Python installation phase on Windows ==
* Login to a Windows machine using privileged account, and follow the instructions below to install Python 3 and PIP:
: https://docs.aws.amazon.com/cli/latest/userguide/install-windows.html

== AWS ParallelCluster installation phase ==
* Run the commands below to install the AWS ParallelCluster:
:* Linux:
:: '''sudo pip install aws-parallelcluster'''
:* Windows:
:: '''pip install aws-parallelcluster'''
* Run the command below to verify the installed version:
: '''pcluster version'''
* Follow the instructions below to install the AWS CLI:
: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html
* Run the command below in-order to configure AWS CLI:
: '''aws configure'''
:* '''AWS Access Key ID''' – Specify the value from the CSV of the previously created IAM user '''parallelcluster-user'''
:* '''AWS Secret Access Key''' – Specify the value from the CSV of the previously created IAM user '''parallelcluster-user'''
:* '''Default region name''' – specify a region such as '''eu-west-1'''
:: Full list: https://docs.aws.amazon.com/general/latest/gr/rande.html
:* '''Default output format''': JSON
* Run the command below to setup the initial configuration:
: '''pcluster configure'''
:* '''Cluster Template''': Specify here a custom name for the HPC template (such as HPC Cluster)
:* '''AWS Region ID''': Specify the same region you specified for the aws configure command (such as '''eu-west-1''')
:* '''VPC Name''': Specify the same name as the Cluster Template (such as '''HPC Cluster''')
:* '''Key Name''': Specify the name of the EC2 Key pair previously created
:* '''VPC ID''': Specify the name of the target VPC ID to deploy the HPC cluster into
:: Note: The full list of VPC’s can be found within the AWS management console: https://console.aws.amazon.com/vpc
:* '''Master Subnet ID''': Specify here the name of the target subnet ID to deploy the HPC cluster into
:: Note: The full list of subnets can be found within the AWS management console: https://console.aws.amazon.com/vpc
* Edit the ParallelCluster '''config''' file:
:* Linux: The file is located inside '''~/.parallelcluster/config'''
:* Windows: The file is located inside '''%UserProfile%\.parallelcluster\config'''
* Add the following parameters to the '''[cluster]''' section (for a large cluster):
: '''base_os = centos7'''
: '''master_instance_type = c5n.xlarge'''
: '''compute_instance_type = c5n.18xlarge'''
: '''cluster_type = ondemand'''
: '''initial_queue_size = 2'''
: '''scheduler = slurm'''
: '''placement_group = DYNAMIC'''
: '''enable_efa = compute'''
: '''fsx_settings = fs'''
: Note: For small cluster, add the following parameters to the '''[cluster]''' section:
: '''base_os = centos7'''
: '''master_instance_type = m4.large'''
: '''compute_instance_type = m4.large'''
: '''cluster_type = ondemand'''
: '''initial_queue_size = 2'''
: '''max_queue_size = 3'''
: '''scheduler = slurm'''
: '''placement_group = DYNAMIC'''
: '''fsx_settings = fs'''
* Add the following entire section to the '''config''' file:
: '''[fsx fs]'''
: '''shared_dir = /fsx'''
: '''storage_capacity = 3600'''
: '''imported_file_chunk_size = 1024'''
: '''export_path = s3://bucket/export'''
: '''import_path = s3://bucket'''
: '''weekly_maintenance_start_time = 1:00:00'''
: Note 1: The storage_capacity is the size of the FSx Lustre storage in GB
: Note 2: Replace the value of '''bucket''' with the previously S3 bucket name
* Run the command below to deploy the new cluster:
: '''pcluster create mycluster'''
: Note: Replace '''mycluster''' with your target HPC cluster name (without spaces)
* Go to the CloudFormation console to view the deployment status:
: https://eu-west-1.console.aws.amazon.com/cloudformation/home
* Wait for the cluster deployment to complete.
* Document the '''MasterPublicIP''' and '''ClusterUser'''.

== Increase EC2 service limit ==
* In-case you need to increase the EC2 service limit (for example number of EC2 instances from a specific instance type), follow the instructions below:
: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-resource-limits.html#request-increase

== Connecting to the HPC cluster (from Linux Machine) ==
* Run the command below to connect using SSH to the master server:
: '''pcluster ssh mycluster -i /path/to/keyfile.pem'''
: Note 1: Replace '''mycluster''' with the previously create cluster name
: Note 2: Replace '''/path/to/keyfile.pem''' with the actual path and key file name
* Run the command below to verify the state of the cluster:
: '''sinfo'''

== Connecting to the HPC cluster (from Windows Machine) ==
* Download '''puttygen.exe''' from:
: http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html
* Run the '''puttygen.exe'''
* Click on “Load” -> change the file extension from “Putty Private key files” to “All Files” -> locate the private key pair and click on Open -> click on OK -> click on “Save private key” -> click on “Yes” -> save the private key file with PPK extension -> close '''puttygen.exe'''
* Download '''Putty''' from:
: https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html
* Run '''putty.exe'''
* From the left pane, under “Connection” -> expand SSH -> click on “Auth” -> from the main pane, under “Authentication parameters”, click on “Browse” -> locate the SSH private key generated by '''puttygen.exe'''
* From the left pane, click on “Session” -> from the main pane, under “Host Name (or IP address)” specify the following:
: '''user@IP_Address'''
: Note 1: Replace '''user''' with the previously documented '''ClusterUser''' value
: Note 2: Replace '''IP_Address''' with the previously documented '''MasterPublicIP'''
* Under “Saved Sessions”, specify a name for this newly created connection.
* Click on Save
* Click on Open
* Run the command below to verify the state of the cluster:
: '''sinfo'''

== Common actions to control the cluster ==
* Displays a list of stacks that are associated with AWS ParallelCluster:
: '''pcluster list'''
* Displays a list of all instances in a cluster:
: '''pcluster instances mycluster'''
: Note: Replace '''mycluster''' with the previously create cluster name
* View the current status of the cluster:
: '''pcluster status mycluster'''
: Note: Replace '''mycluster''' with the previously create cluster name
* Updates a running cluster by using the values in the configuration file:
: '''pcluster update mycluster -c ~/.parallelcluster/config'''
: Note 1: Replace '''mycluster''' with the previously create cluster name
: Note 2: Replace '''~/.parallelcluster/config''' with the target config file location
* Stops the compute fleet, leaving the master node running:
: '''pcluster stop mycluster'''
: Note: Replace '''mycluster''' with the previously create cluster name
* Starts the compute fleet for a cluster that has been stopped:
: '''pcluster start mycluster'''
: Note: Replace '''mycluster''' with the previously create cluster name

== Delete AWS ParallelCluster ==
* In-case you wish to keep the AWS ParallelCluster master node static IP, login to the AWS console:
: https://console.aws.amazon.com/ec2/
* From the left pane, click on Elastic IPs -> select the public IP of the master node -> Actions -> Disassociate address
* From command prompt (the same machine you used the pcluster commands), run the command below to delete the cluster:
: '''pcluster delete mycluster'''
: Note: Replace '''mycluster''' with the previously create cluster name

== Important notes regarding shared storage ==
* Long term data must be stored inside S3 bucket
* The Amazon FSx for Lustre storage (mount '''/fsx''') will be used for the duration of the compute job

== References ==
* Getting started with AWS ParallelCluster:
: https://aws-parallelcluster.readthedocs.io/en/latest/getting_started.html#
* Setting Up AWS ParallelCluster
: https://docs.aws.amazon.com/parallelcluster/latest/ug/getting_started.html
* Install AWS ParallelCluster in a Virtual Environment
: https://docs.aws.amazon.com/parallelcluster/latest/ug/install-virtualenv.html
* A Scientist's Guide to Cloud-HPC: Example with AWS ParallelCluster, Slurm, Spack, and WRF
: https://jiaweizhuang.github.io/blog/aws-hpc-guide/
* Launch your first sample HPC environment on AWS and review important concepts along the way
: https://aws.amazon.com/getting-started/use-cases/hpc/
* AWS ParallelCluster Wiki:
: https://github.com/aws/aws-parallelcluster
* Deploying an Elastic HPC Cluster
: https://d1.awsstatic.com/Projects/P4114756/deploy-elastic-hpc-cluster_project.pdf
* Scale HPC Workloads with Elastic Fabric Adapter and AWS ParallelCluster
: https://idk.dev/scale-hpc-workloads-with-elastic-fabric-adapter-and-aws-parallelcluster/
* Best Practices for Running Ansys Fluent Using AWS ParallelCluster
: https://aws.amazon.com/blogs/opensource/best-practices-running-ansys-fluent-aws-parallelcluster/
* AWS ParallelCluster with AWS Directory Services Authentication
: https://aws.amazon.com/blogs/opensource/aws-parallelcluster-aws-directory-services-authentication/
* Adding support for FSx for Lustre:
: https://aws-parallelcluster.readthedocs.io/en/develop/configuration.html#fsx-section
* Getting Started with Amazon FSx for Lustre
: https://docs.aws.amazon.com/fsx/latest/LustreGuide/getting-started.html
* Amazon FSx for Lustre Lustre User Guide
: https://docs.aws.amazon.com/fsx/latest/LustreGuide/LustreGuide.pdf

How to create AWS ParallelCluster with Slurm scheduler

2019-07-29T13:24:31Z

Eyales:

== Environment preparation phase within AWS Management Console ==
* Login the AWS IAM console:
: https://console.aws.amazon.com/iam/
* From the left pane, click on Policies -> click on Users -> Add User -> specify the name '''parallelcluster-user''' -> Access type: Programmatic access -> click Next: Permissions -> Set permissions -> select a group with '''“AdministratorAccess”''' role -> click Next: Tags -> click Next: Review -> click on Create user -> click on Download .csv and keep it in a secured location -> click on Close
* Follow the instructions below to create a key pair to access the cluster machines:
: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#having-ec2-create-your-key-pair
* Follow the instructions below to create S3 bucket (with unique name) for storing data to export and import data to/from the FSx Lustre storage:
: https://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html
: Note 1: Document the S3 bucket name for use inside the ParallelCluster config file
: Note 2: Create a folder called '''export''' (in small letters), inside the S3 bucket
* In-case you wish to create a dedicate VPC and subnet for the HPC cluster, follow the instructions below:
: https://docs.aws.amazon.com/directoryservice/latest/admin-guide/gsg_create_vpc.html
* Logoff the AWS console

== Python installation phase on Linux (Debian / Ubuntu) ==
* Login to a Linux machine using SSH, and follow the instructions below to install Python 3:
: https://docs.aws.amazon.com/cli/latest/userguide/install-linux-python.html
: Note: In-case you already have Python 3 install, use the command below to upgrade to the latest build:
: '''sudo apt-get upgrade python3'''
* To install pip3, run the command below:
: '''sudo apt install python3-pip'''

== Python installation phase on CentOS 7 ==
* Login to the CentOS machine using SSH, and follow the instructions below to install Python3 and Python3-PIP:
: https://www.rosehosting.com/blog/how-to-install-python-3-6-4-on-centos-7/

== Python installation phase on Windows ==
* Login to a Windows machine using privileged account, and follow the instructions below to install Python 3 and PIP:
: https://docs.aws.amazon.com/cli/latest/userguide/install-windows.html

== AWS ParallelCluster installation phase ==
* Run the commands below to install the AWS ParallelCluster:
:* Linux:
:: '''sudo pip install aws-parallelcluster'''
:* Windows:
:: '''pip install aws-parallelcluster'''
* Run the command below to verify the installed version:
: '''pcluster version'''
* Follow the instructions below to install the AWS CLI:
: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html
* Run the command below in-order to configure AWS CLI:
: '''aws configure'''
:* '''AWS Access Key ID''' – Specify the value from the CSV of the previously created IAM user '''parallelcluster-user'''
:* '''AWS Secret Access Key''' – Specify the value from the CSV of the previously created IAM user '''parallelcluster-user'''
:* '''Default region name''' – specify a region such as '''eu-west-1'''
:: Full list: https://docs.aws.amazon.com/general/latest/gr/rande.html
:* '''Default output format''': JSON
* Run the command below to setup the initial configuration:
: '''pcluster configure'''
:* '''Cluster Template''': Specify here a custom name for the HPC template (such as HPC Cluster)
:* '''AWS Region ID''': Specify the same region you specified for the aws configure command (such as '''eu-west-1''')
:* '''VPC Name''': Specify the same name as the Cluster Template (such as '''HPC Cluster''')
:* '''Key Name''': Specify the name of the EC2 Key pair previously created
:* '''VPC ID''': Specify the name of the target VPC ID to deploy the HPC cluster into
:: Note: The full list of VPC’s can be found within the AWS management console: https://console.aws.amazon.com/vpc
:* '''Master Subnet ID''': Specify here the name of the target subnet ID to deploy the HPC cluster into
:: Note: The full list of subnets can be found within the AWS management console: https://console.aws.amazon.com/vpc
* Edit the ParallelCluster '''config''' file:
:* Linux: The file is located inside '''~/.parallelcluster/config'''
:* Windows: The file is located inside '''%UserProfile%\.parallelcluster\config'''
* Add the following parameters to the '''[cluster]''' section (for a large cluster):
: '''base_os = centos7'''
: '''master_instance_type = c5n.xlarge'''
: '''compute_instance_type = c5n.18xlarge'''
: '''cluster_type = ondemand'''
: '''initial_queue_size = 2'''
: '''scheduler = slurm'''
: '''placement_group = DYNAMIC'''
: '''enable_efa = compute'''
: '''fsx_settings = fs'''
: Note: For small cluster, add the following parameters to the '''[cluster]''' section:
: '''base_os = centos7'''
: '''master_instance_type = m4.large'''
: '''compute_instance_type = m4.large'''
: '''cluster_type = ondemand'''
: '''initial_queue_size = 2'''
: '''max_queue_size = 3'''
: '''scheduler = slurm'''
: '''placement_group = DYNAMIC'''
: '''fsx_settings = fs'''
* Add the following entire section to the '''config''' file:
: '''[fsx fs]'''
: '''shared_dir = /fsx'''
: '''storage_capacity = 3600'''
: '''imported_file_chunk_size = 1024'''
: '''export_path = s3://bucket/export'''
: '''import_path = s3://bucket'''
: '''weekly_maintenance_start_time = 1:00:00'''
: Note 1: The storage_capacity is the size of the FSx Lustre storage in GB
: Note 2: Replace the value of '''bucket''' with the previously S3 bucket name
* Run the command below to deploy the new cluster:
: '''pcluster create mycluster'''
: Note: Replace '''mycluster''' with your target HPC cluster name (without spaces)
* Go to the CloudFormation console to view the deployment status:
: https://eu-west-1.console.aws.amazon.com/cloudformation/home
* Wait for the cluster deployment to complete.
* Document the '''MasterPublicIP''' and '''ClusterUser'''.

== Increase EC2 service limit ==
* In-case you need to increase the EC2 service limit (for example number of EC2 instances from a specific instance type), follow the instructions below:
: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-resource-limits.html#request-increase

== Connecting to the HPC cluster (from Linux Machine) ==
* Run the command below to connect using SSH to the master server:
: '''pcluster ssh mycluster -i /path/to/keyfile.pem'''
: Note 1: Replace '''mycluster''' with the previously create cluster name
: Note 2: Replace '''/path/to/keyfile.pem''' with the actual path and key file name
* Run the command below to verify the state of the cluster:
: '''sinfo'''

== Connecting to the HPC cluster (from Windows Machine) ==
* Download '''puttygen.exe''' from:
: http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html
* Run the '''puttygen.exe'''
* Click on “Load” -> change the file extension from “Putty Private key files” to “All Files” -> locate the private key pair and click on Open -> click on OK -> click on “Save private key” -> click on “Yes” -> save the private key file with PPK extension -> close '''puttygen.exe'''
* Download '''Putty''' from:
: https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html
* Run '''putty.exe'''
* From the left pane, under “Connection” -> expand SSH -> click on “Auth” -> from the main pane, under “Authentication parameters”, click on “Browse” -> locate the SSH private key generated by '''puttygen.exe'''
* From the left pane, click on “Session” -> from the main pane, under “Host Name (or IP address)” specify the following:
: '''user@IP_Address'''
: Note 1: Replace '''user''' with the previously documented '''ClusterUser''' value
: Note 2: Replace '''IP_Address''' with the previously documented '''MasterPublicIP'''
* Under “Saved Sessions”, specify a name for this newly created connection.
* Click on Save
* Click on Open
* Run the command below to verify the state of the cluster:
: '''sinfo'''

== Common actions to control the cluster ==
* Displays a list of stacks that are associated with AWS ParallelCluster:
: '''pcluster list'''
* Displays a list of all instances in a cluster:
: '''pcluster instances mycluster'''
: Note: Replace '''mycluster''' with the previously create cluster name
* View the current status of the cluster:
: '''pcluster status mycluster'''
: Note: Replace '''mycluster''' with the previously create cluster name
* Updates a running cluster by using the values in the configuration file:
: '''pcluster update mycluster -c ~/.parallelcluster/config'''
: Note 1: Replace '''mycluster''' with the previously create cluster name
: Note 2: Replace '''~/.parallelcluster/config''' with the target config file location
* Stops the compute fleet, leaving the master node running:
: '''pcluster stop mycluster'''
: Note: Replace '''mycluster''' with the previously create cluster name
* Starts the compute fleet for a cluster that has been stopped:
: '''pcluster start mycluster'''
: Note: Replace '''mycluster''' with the previously create cluster name

== Delete AWS ParallelCluster ==
* In-case you wish to keep the AWS ParallelCluster master node static IP, login to the AWS console:
: https://console.aws.amazon.com/ec2/
* From the left pane, click on Elastic IPs -> select the public IP of the master node -> Actions -> Disassociate address
* From command prompt (the same machine you used the pcluster commands), run the command below to delete the cluster:
: '''pcluster delete mycluster'''
: Note: Replace '''mycluster''' with the previously create cluster name

== Important notes regarding shared storage ==
* Long term data must be stored inside S3 bucket
* The Amazon FSx for Lustre storage (mount '''/fsx''') will be used for the duration of the compute job

== References ==
* Getting started with AWS ParallelCluster:
: https://aws-parallelcluster.readthedocs.io/en/latest/getting_started.html#
* Setting Up AWS ParallelCluster
: https://docs.aws.amazon.com/parallelcluster/latest/ug/getting_started.html
* Install AWS ParallelCluster in a Virtual Environment
: https://docs.aws.amazon.com/parallelcluster/latest/ug/install-virtualenv.html
* A Scientist's Guide to Cloud-HPC: Example with AWS ParallelCluster, Slurm, Spack, and WRF
: https://jiaweizhuang.github.io/blog/aws-hpc-guide/
* Launch your first sample HPC environment on AWS and review important concepts along the way
: https://aws.amazon.com/getting-started/use-cases/hpc/
* AWS ParallelCluster Wiki:
: https://github.com/aws/aws-parallelcluster
* Deploying an Elastic HPC Cluster
: https://d1.awsstatic.com/Projects/P4114756/deploy-elastic-hpc-cluster_project.pdf
* Scale HPC Workloads with Elastic Fabric Adapter and AWS ParallelCluster
: https://idk.dev/scale-hpc-workloads-with-elastic-fabric-adapter-and-aws-parallelcluster/
* Adding support for FSx for Lustre:
: https://aws-parallelcluster.readthedocs.io/en/develop/configuration.html#fsx-section
* Getting Started with Amazon FSx for Lustre
: https://docs.aws.amazon.com/fsx/latest/LustreGuide/getting-started.html
* Amazon FSx for Lustre Lustre User Guide
: https://docs.aws.amazon.com/fsx/latest/LustreGuide/LustreGuide.pdf

How to create AWS ParallelCluster with Slurm scheduler

2019-07-29T12:16:37Z

Eyales:

== Environment preparation phase within AWS Management Console ==
* Login the AWS IAM console:
: https://console.aws.amazon.com/iam/
* From the left pane, click on Policies -> click on Users -> Add User -> specify the name '''parallelcluster-user''' -> Access type: Programmatic access -> click Next: Permissions -> Set permissions -> select a group with '''“AdministratorAccess”''' role -> click Next: Tags -> click Next: Review -> click on Create user -> click on Download .csv and keep it in a secured location -> click on Close
* Follow the instructions below to create a key pair to access the cluster machines:
: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#having-ec2-create-your-key-pair
* Follow the instructions below to create S3 bucket (with unique name) for storing data to export and import data to/from the FSx Lustre storage:
: https://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html
: Note 1: Document the S3 bucket name for use inside the ParallelCluster config file
: Note 2: Create a folder called '''export''' (in small letters), inside the S3 bucket
* In-case you wish to create a dedicate VPC and subnet for the HPC cluster, follow the instructions below:
: https://docs.aws.amazon.com/directoryservice/latest/admin-guide/gsg_create_vpc.html
* Logoff the AWS console

== Python installation phase on Linux (Debian / Ubuntu) ==
* Login to a Linux machine using SSH, and follow the instructions below to install Python 3:
: https://docs.aws.amazon.com/cli/latest/userguide/install-linux-python.html
: Note: In-case you already have Python 3 install, use the command below to upgrade to the latest build:
: '''sudo apt-get upgrade python3'''
* To install pip3, run the command below:
: '''sudo apt install python3-pip'''

== Python installation phase on Windows ==
* Login to a Windows machine using privileged account, and follow the instructions below to install Python 3 and PIP:
: https://docs.aws.amazon.com/cli/latest/userguide/install-windows.html

== AWS ParallelCluster installation phase ==
* Run the commands below to install the AWS ParallelCluster:
:* Linux:
:: '''sudo pip install aws-parallelcluster'''
:* Windows:
:: '''pip install aws-parallelcluster'''
* Run the command below to verify the installed version:
: '''pcluster version'''
* Follow the instructions below to install the AWS CLI:
: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html
* Run the command below in-order to configure AWS CLI:
: '''aws configure'''
:* '''AWS Access Key ID''' – Specify the value from the CSV of the previously created IAM user '''parallelcluster-user'''
:* '''AWS Secret Access Key''' – Specify the value from the CSV of the previously created IAM user '''parallelcluster-user'''
:* '''Default region name''' – specify a region such as '''eu-west-1'''
:: Full list: https://docs.aws.amazon.com/general/latest/gr/rande.html
:* '''Default output format''': JSON
* Run the command below to setup the initial configuration:
: '''pcluster configure'''
:* '''Cluster Template''': Specify here a custom name for the HPC template (such as HPC Cluster)
:* '''AWS Region ID''': Specify the same region you specified for the aws configure command (such as '''eu-west-1''')
:* '''VPC Name''': Specify the same name as the Cluster Template (such as '''HPC Cluster''')
:* '''Key Name''': Specify the name of the EC2 Key pair previously created
:* '''VPC ID''': Specify the name of the target VPC ID to deploy the HPC cluster into
:: Note: The full list of VPC’s can be found within the AWS management console: https://console.aws.amazon.com/vpc
:* '''Master Subnet ID''': Specify here the name of the target subnet ID to deploy the HPC cluster into
:: Note: The full list of subnets can be found within the AWS management console: https://console.aws.amazon.com/vpc
* Edit the ParallelCluster '''config''' file:
:* Linux: The file is located inside '''~/.parallelcluster/config'''
:* Windows: The file is located inside '''%UserProfile%\.parallelcluster\config'''
* Add the following parameters to the '''[cluster]''' section (for a large cluster):
: '''base_os = centos7'''
: '''master_instance_type = c5n.xlarge'''
: '''compute_instance_type = c5n.18xlarge'''
: '''cluster_type = ondemand'''
: '''initial_queue_size = 2'''
: '''scheduler = slurm'''
: '''placement_group = DYNAMIC'''
: '''enable_efa = compute'''
: '''fsx_settings = fs'''
: Note: For small cluster, add the following parameters to the '''[cluster]''' section:
: '''base_os = centos7'''
: '''master_instance_type = m4.large'''
: '''compute_instance_type = m4.large'''
: '''cluster_type = ondemand'''
: '''initial_queue_size = 2'''
: '''max_queue_size = 3'''
: '''scheduler = slurm'''
: '''placement_group = DYNAMIC'''
: '''fsx_settings = fs'''
* Add the following entire section to the '''config''' file:
: '''[fsx fs]'''
: '''shared_dir = /fsx'''
: '''storage_capacity = 3600'''
: '''imported_file_chunk_size = 1024'''
: '''export_path = s3://bucket/export'''
: '''import_path = s3://bucket'''
: '''weekly_maintenance_start_time = 1:00:00'''
: Note 1: The storage_capacity is the size of the FSx Lustre storage in GB
: Note 2: Replace the value of '''bucket''' with the previously S3 bucket name
* Run the command below to deploy the new cluster:
: '''pcluster create mycluster'''
: Note: Replace '''mycluster''' with your target HPC cluster name (without spaces)
* Go to the CloudFormation console to view the deployment status:
: https://eu-west-1.console.aws.amazon.com/cloudformation/home
* Wait for the cluster deployment to complete.
* Document the '''MasterPublicIP''' and '''ClusterUser'''.

== Increase EC2 service limit ==
* In-case you need to increase the EC2 service limit (for example number of EC2 instances from a specific instance type), follow the instructions below:
: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-resource-limits.html#request-increase

== Connecting to the HPC cluster (from Linux Machine) ==
* Run the command below to connect using SSH to the master server:
: '''pcluster ssh mycluster -i /path/to/keyfile.pem'''
: Note 1: Replace '''mycluster''' with the previously create cluster name
: Note 2: Replace '''/path/to/keyfile.pem''' with the actual path and key file name
* Run the command below to verify the state of the cluster:
: '''sinfo'''

== Connecting to the HPC cluster (from Windows Machine) ==
* Download '''puttygen.exe''' from:
: http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html
* Run the '''puttygen.exe'''
* Click on “Load” -> change the file extension from “Putty Private key files” to “All Files” -> locate the private key pair and click on Open -> click on OK -> click on “Save private key” -> click on “Yes” -> save the private key file with PPK extension -> close '''puttygen.exe'''
* Download '''Putty''' from:
: https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html
* Run '''putty.exe'''
* From the left pane, under “Connection” -> expand SSH -> click on “Auth” -> from the main pane, under “Authentication parameters”, click on “Browse” -> locate the SSH private key generated by '''puttygen.exe'''
* From the left pane, click on “Session” -> from the main pane, under “Host Name (or IP address)” specify the following:
: '''user@IP_Address'''
: Note 1: Replace '''user''' with the previously documented '''ClusterUser''' value
: Note 2: Replace '''IP_Address''' with the previously documented '''MasterPublicIP'''
* Under “Saved Sessions”, specify a name for this newly created connection.
* Click on Save
* Click on Open
* Run the command below to verify the state of the cluster:
: '''sinfo'''

== Common actions to control the cluster ==
* Displays a list of stacks that are associated with AWS ParallelCluster:
: '''pcluster list'''
* Displays a list of all instances in a cluster:
: '''pcluster instances mycluster'''
: Note: Replace '''mycluster''' with the previously create cluster name
* View the current status of the cluster:
: '''pcluster status mycluster'''
: Note: Replace '''mycluster''' with the previously create cluster name
* Updates a running cluster by using the values in the configuration file:
: '''pcluster update mycluster -c ~/.parallelcluster/config'''
: Note 1: Replace '''mycluster''' with the previously create cluster name
: Note 2: Replace '''~/.parallelcluster/config''' with the target config file location
* Stops the compute fleet, leaving the master node running:
: '''pcluster stop mycluster'''
: Note: Replace '''mycluster''' with the previously create cluster name
* Starts the compute fleet for a cluster that has been stopped:
: '''pcluster start mycluster'''
: Note: Replace '''mycluster''' with the previously create cluster name

== Delete AWS ParallelCluster ==
* In-case you wish to keep the AWS ParallelCluster master node static IP, login to the AWS console:
: https://console.aws.amazon.com/ec2/
* From the left pane, click on Elastic IPs -> select the public IP of the master node -> Actions -> Disassociate address
* From command prompt (the same machine you used the pcluster commands), run the command below to delete the cluster:
: '''pcluster delete mycluster'''
: Note: Replace '''mycluster''' with the previously create cluster name

== Important notes regarding shared storage ==
* Long term data must be stored inside S3 bucket
* The Amazon FSx for Lustre storage (mount '''/fsx''') will be used for the duration of the compute job

== References ==
* Getting started with AWS ParallelCluster:
: https://aws-parallelcluster.readthedocs.io/en/latest/getting_started.html#
* Setting Up AWS ParallelCluster
: https://docs.aws.amazon.com/parallelcluster/latest/ug/getting_started.html
* Install AWS ParallelCluster in a Virtual Environment
: https://docs.aws.amazon.com/parallelcluster/latest/ug/install-virtualenv.html
* A Scientist's Guide to Cloud-HPC: Example with AWS ParallelCluster, Slurm, Spack, and WRF
: https://jiaweizhuang.github.io/blog/aws-hpc-guide/
* Launch your first sample HPC environment on AWS and review important concepts along the way
: https://aws.amazon.com/getting-started/use-cases/hpc/
* AWS ParallelCluster Wiki:
: https://github.com/aws/aws-parallelcluster
* Deploying an Elastic HPC Cluster
: https://d1.awsstatic.com/Projects/P4114756/deploy-elastic-hpc-cluster_project.pdf
* Scale HPC Workloads with Elastic Fabric Adapter and AWS ParallelCluster
: https://idk.dev/scale-hpc-workloads-with-elastic-fabric-adapter-and-aws-parallelcluster/
* Adding support for FSx for Lustre:
: https://aws-parallelcluster.readthedocs.io/en/develop/configuration.html#fsx-section
* Getting Started with Amazon FSx for Lustre
: https://docs.aws.amazon.com/fsx/latest/LustreGuide/getting-started.html
* Amazon FSx for Lustre Lustre User Guide
: https://docs.aws.amazon.com/fsx/latest/LustreGuide/LustreGuide.pdf