AWS Cloudformation Deployment

Index

Introduction

Panintelligence is the easy-to-use and quick-to-deploy solution to unlocking powerful data insights. We're the only software solution on the market that brings together business intelligence and machine learning to a new self-service level, empowering you to gain full control over your data.

Access unprecedented styling options to make our software look just like your own, and hook up cloud data warehousing and ETL (Extract, Transform and Load) tools, as well as your core product, to create a truly seamless analytics experience.

Simply connect your databases to the Panintelligence dashboard. The Panintelligence dashboard only has read access so it won’t modify any of your data.

Installation (AWS CloudFormation)

Below is a AWS CloudFormation script template which is infrastructure as code which uses AWS services to provide Operational Excellence, Security, Reliability, Performance Efficiency and Cost Optimisation.

Infrastructure as Code (IaC) means to manage your IT infrastructure using configuration files. AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less time managing those resources and more time focusing on your applications that run in AWS.

To create a template that describes all the AWS resources that are required (For example, Amazon EC2 instances or Amazon RDS DB instances), and AWS CloudFormation takes care of provisioning and configuring those resources.

To individually create and configure AWS resources manually on AWS Management Console is not required. AWS CloudFormation handles the dependencies. Find more information on AWS Cloudformation here.

For best results, please use our set of AWS (Amazon Web Services) CloudFormation scripts which will deploy Panintelligence in the most optimal way.

Please feel free to contribute to our Github project to maintain and continually improve our deployment methodology!

Github has instructions on how to deploy and this document to further explain the architecture.

• AWS Cloudformation script:

System Architecture

At Panintelligence we follow AWS five pillars of success, for more information on what are AWS five pillars of success click here.

The architecture diagram below shows an overview of how the components are connected:

1. The internet gateway allows traffic into the AWS VPC that’s attached to the public route table. Inside the route table, the AWS Application Load Balancer has a route to the internet gateway. The AWS Application Load Balancer will listen to port 80 and port 443. The security group of the AWS Application load Balancer will also allow port 80 and 443

2. The AWS Application Load Balancer will direct traffic to a healthy EC2 target to access the Panintelligence dashboard on the web browser

3. The AWS EC2 instances are part of an auto scaling group based on resource demand. Due to multiple instances, the instances connect to an external Maria DB which is an AWS RDS MariaDB for the persistent storage and fault resilience. The auto scaling rule will scale out if it hits 70% of CPU usage. It will scale back down when the CPU usage is below 20%

4. As long as you’ve got access to the s3 bucket based on your IAM permissions, you can upload ‘images’, ‘themes’ and ‘excel-data’ files to the bucket. Once they are uploaded, it will set off an object creation trigger to the AWS Lambda to migrate the files onto AWS EFS. The AWS EFS is attached to the auto scaling group so the instances have persistent storage

5. The folders within the S3 bucket will contain your own personal themes, images and excel data

6. The RDS MariaDB will contain your own personal dashboard configurations

7. Allows outbound network traffic using the NAT Gateway

Open

Skills required

Minimum skills to set it up:

• AWS Knowledge - Recommended for people who have obtained AWS Cloud Practitioner exam or higher to understand each AWS service

• Basic Linux AWS CloudShell skills - You will need to navigate through AWS CloudShell which is a Linux environment to be able to pull the project from Git

• Git - Some basic understanding on how to use git and be able to pull a project

• Networking/Security - You will need to understand how the infrastructure is built and how each service communicates with each other. To set it up, you will only need to understand what parameter values you need to enter for the project

• AWS CloudFormation - Infrastructure as code allows you to build AWS services through a configuration text file so you don’t have to manually create services. You will need to understand how it works but the instructions are in the GitHub project on how to deploy it

Advanced skills to configure other aspects of the infrastructure:

• Networking/Security - If you wish to configure the AWS CloudFormation script and you wish to add more services, you will need to make sure you know the Security Groups, Network Access Control Lists and Route tables

• Python - We have a Lambda function which is written in python. The python script grabs an S3 object event trigger to push to AWS EFS. If you wish to modify it, you will need Python skills

• MariaDB/SQL commands - MariaDB knowledge on how to access AWS RDS MariaDB and view the Panintelligence dashboard database

• Docker and Docker-compose - We install the application using configured docker-compose scripts.

Resources and prerequisites

To complete this documentation, we assume you have a Route 53 hosted zone, an ACM certificate (AWS Certificate Manager) that is validated and covers the domain you will be using for the load balancer, an S3 bucket for backups and a key pair to allow you SSH (Secure Shell Protocol) access to the server.

Service quotas

For the AWS Services that you will be using, you will need to be aware of your service level quotas. You don’t want to hit your limit on your account, however you can always submit a Service quota increase with AWS. For more information on what are AWS Service quotas, click here.

The table below contains information on the specific resources that are used in the architecture. Please put in the region name in the URL that you are building Panintelligence in.

How to obtain a Panintelligence licence key?

You will need to contact your Panintelligence customer success manager or support@panintelligence.com for pricing information and to obtain a licence for use with our software for the BYOL (Bring your own licence) version of Panintelligence.

Developer and trial both have limited use case licences embedded in the AMI image (Amazon Machine Image). Our Metered offering charges based on units (users) and dimensions (analytics, scheduler, reports).

Automated Licence Manager

For more information on how to use the automated licence feature:

Automated Licence Manager

How to obtain the Panintelligence Marketplace AMI ID?

• Go to your AWS (Amazon Web Services) console and search for ‘AWS marketplace subscriptions’ on the services

  Open

  

• Inside the AWS (Amazon Web Services) marketplace subscriptions, click on ‘Discover products’ on your left and search for ‘Panintelligence’

• Select one of the Panintelligence products that you wish to use. For this example we will use ‘Panintelligence BYOL (Bring your own licence)’

• Click on to ‘Subscribe’ or ‘Continue to Subscribe’ and await to confirm subscription

• Go back to AWS (Amazon Web Services) marketplace subscriptions console in AWS (Amazon Web Services) and you should see your Panintelligence subscription. Click on to your Panintelligence subscription

• Click to launch instance on the right

• Depending on what region you wish to deploy it in, please select the region and the AMI ID will change. Please copy that AMI ID and keep that safe

Technical Datasheet requirements

The technical datasheet offers some guidance on how much resources you would require depending on your infrastructure and users.

Please take a look at this link.

Operating System

In addition, on Panintelligence AMI we have Linux Ubuntu 20.04 as the operating system.

Docker overview

For more information on how the docker configuration within the AMI products:

AWS AMI products

Size requirements and recommendations

Due to the recommended architecture, you have separated the dependencies on the EC2 instance to EFS and AWS RDS MariaDB.

EC2 instance type:

Instance types comprise varying combinations of CPU, memory, storage, and networking capacity and give you the flexibility to choose the appropriate mix of resources for your applications. Below is the recommended instance types to use.

EBS volume type:

It is recommended to have 15GiB or higher for your EBS volume attached to the EC2. Depending on your data connections and users, it can be higher. Please take a look at the Technical Datasheet/ System Requirements documentation for more information.

Amazon RDS MariaDB Instance size:

Security

It is important to highlight the security configurations.

When building the AWS Cloudformation deployment you will need assign a IAM User to be able to create and deploy AWS Cloudformation stacks.

Public access

The Application load balancer listens to port 80 and 443. Port 80 redirects to port 443. This is to allow the public to access the Panintelligence dashboard.

Private access

AWS Lambda has only access to the S3 bucket to upload the AWS Lambda ZIP and the S3 event triggers the AWS Lambda to migrate the S3 objects (images, themes, excel-data) to AWS EFS for persistent data.

Least privilege

AWS RDS MariaDB uses AWS secret manager to generate the database password. In addition, the infrastructure rotates the secret details every 30 days. To find the sensitive information, you can locate it in AWS secret manager.

AWS EC2 user data contains sensitive information due to the Panintelligence use of the external AWS RDS MariaDB details. Therefore, the infrastructure has a secret manager endpoint attached to allow the EC2 instances to retrieve the secrets. This is because we cannot store secrets in plain text and only the EC2 instances are allowed to retrieve them.

We can’t show the secrets in plain text therefore in the AWS Cloudformation deployment we use the AWS secret manager api call to retrieve the secrets. Due to auto-scaling policy, we put the environment variables in the user data so when the EC2 instance boots up, it will have the correct information. For more information, take a look at the AWS Cloudformation project with the EC2 stacks: https://github.com/Panintelligence/aws-deployment/tree/main/nested-stacks

How to obtain the secrets?

AWS Management console:

1. If not already logged into the console, go to the console at https://console.aws.amazon.com/secretsmanager/ and log into the Secrets Manager service.

2. On the Secrets list page, choose the name of the new secret you created.

   Secrets Manager displays the Secrets details page for your secret.

3. In the Secret value section, choose Retrieve secret value.

4. You can view your secret as either key-value pairs, or as a JSON text structure.

AWS CLI:

1. Open a command prompt to run the AWS CLI. If you haven't installed the AWS CLI yet, see Installing the AWS Command Line Interface.

2. Using credentials with permissions to access your secret, type the following command and parameters.

Review the VersionIdsToStages response value. The output contains a list of all active versions of the secret and the staging labels attached to each version. In this tutorial, you should see a single version ID (a UUID type value) mapping to a single staging label AWSCURRENT.

If you want details for a version with a different staging label than AWSCURRENT, you must include the --version-stage parameter in the previous command. Secrets Manager uses AWSCURRENT as the default value.

The rest of the output includes the JSON version of your secret value in the SecretString response field.

Encryption at rest

The AWS Cloudformation deployment script contains sensitive data. We store the Panintelligence repository data inside the external RDS MariaDB. We store the images and themes that side loads through AWS S3 to AWS EFS.

S3:

The s3 bucket is encrypted by AES 256. The data is encrypted at rest. Here is a snippet of the AWS Cloudformation script:

AWS RDS MariaDB:

The AWS RDS MariaDB is encrypted using rotated keys for the password using secret manager. With RDS-encrypted resources, data is encrypted at rest, including the underlying storage for a database (DB) instance, its automated backups, read replicas, and snapshots. This capability uses the open standard AES-256 encryption algorithm to encrypt your data, which is transparent to your database engine.

AWS EFS:

AWS EFS is encrypted by default. Amazon EFS uses industry-standard AES-256 encryption algorithm to encrypt EFS data and metadata at rest.

Identity and access management (IAM)

AWS Identity and Access Management (IAM) is a web service that helps you securely control access to AWS resources. You use IAM to control who is authenticated (signed in) and authorized (has permissions) to use resources. Please take a look at the documentation regarding IAM:

• What is IAM?

• Best practices and least privilege in IAM

We have two IAM Roles within the AWS Cloudformation script:

• EC2 role:

• Lambda role:

• Cloudwatch logging permissions:

• S3 permissions are locked down by the resource arn:

• EFS permissions locked down by the resource arn:

EC2 Network interface permissions:

Multi-Availability zone deployment

The architecture of the AWS Cloudformation script has a MultiAZ RDS and MultiAZ EC2. A multi-AZ distribution model gives you high availability in case a major failure occurs in an entire Availability Zone. If a storage volume on your primary instance fails in a Multi-AZ deployment, Amazon RDS automatically initiates a failover to the up-to-date standby (or to a replica in the case of Amazon Aurora).

For example, if it’s in the region Ireland you can enter ‘eu-west-1a’ and ‘eu-west-1b’ in the AWS Cloudformation parameters.

Please click the related links for more information on availability zones and RDS Multi Availability zones.

Auto-scaling policy

Estimated time to boot an EC2 instance is 3 minutes to have it fully functioning. In addition, estimated time to down scale 5 minutes.

We have added some elasticity to the infrastructure to allow auto-scaling to scale in and scale out depending on the resource demand. Currently, when you set it up, the script desired state is one EC2 instance however, if the Panintelligence dashboard EC2 has a lot of traffic and the CPU usage increases to 70%, the auto scaling policy will increase the instances and decrease the instance size when the CPU is below 10%. Here is a diagram below to show this.

If you wish to know more about auto-scaling, please take a look at this document.

Persistent storage on AWS RDS

Due to having an external AWS RDS MariaDB to store your Panintelligence dashboard repository, it will be persistent when you scale in or scale out your EC2 instances.

Persistent storage on AWS EFS

Due to having AWS EFS, when you scale in or scale out your EC2 instances. Your themes, images and excel-data will be the same across all instances so you don’t have to individually configure each EC2 instance.

How does S3 bucket side loading work?

Uploading your themes, images and excel data

In order to customise the look and feel of Panintelligence and to move your own data into the instance, we need themes (comprising of CSS (Cascading Style Sheets) files) and images. Some default themes have been provided (within Gitlab), however you may wish to add more. In order to simplify this process, we’ve included a ‘Side Load’ process which allows you to drop files into an S3 bucket and for them to be mounted automatically within your Panintelligence cluster.

If you are following our Github Cloudformation deployment, you have to copy your theme folder to the s3 bucket that was created.

How to upgrade

You can update the AMI image by updating the AWS (Amazon Web Services) CloudFormation stack with the new AMI-ID.

Releasing new versions without downtime

At Panintelligence, we think it’s super important that users always have access to their data. Therefore, we support Blue/Green deployment methodologies when deploying a new version of our software. In order to do this on AWS (Amazon Web Services), once you’ve updated the AWS (Amazon Web Services) CloudFormation stack with the new AMI-ID, you can simply increase the desired count of your Panintelligence instances in the AWS (Amazon Web Services) Autoscaling group by one. Once your new instance has started, you can reduce it back to the previous level. This will stop the oldest instance first (the old version).

Enter the AMI ID for the region you have deployed in and replace the value for $AMIID. Update the stack once you are happy.

Networking and security groups

The EC2 Panintelligence private security group with only port 8224 inbound and outbound to everywhere so you can connect to outbound databases.

The Application load balancer inbound port 80 and port 443 which is only web traffic and outbound to everywhere.

The RDS MariaDB has inbound route port 3306 within the VPC:

Backup and restore

The EC2 relies on the AWS RDS MariaDB for the Panintelligence repository. The EC2 can be destroyed without you losing data as it’s on the external database.

Database restore:

For more information on how to backup and restore your AWS RDS, please look at these links:

• Backing up and restoring an Amazon RDS DB instance - Amazon Relational Database Service

• Restoring from a DB snapshot - Amazon Relational Database Service

How to restore from a time period?

• Log into AWS Management console → Go into AWS RDS console → Click into ‘Databases’ → Click into ‘dashboard’

• On the right hand side, click ‘Actions’ and click on ‘Restore to point in time’. This will allow you to restore your database from a point of time depending if you have snapshots being created. By default, it will create snapshots for you.

• Once you have created a new Database instance with a different name, the database will have the data for the time you specified. However, as it’s a new database, you will need to configure the AWS CloudFormation script. In PanintelligenceThree script, you will need to edit and update the cloudformation script to the new details of the restored database:

• Once the stack is updated, you will need to scale out your autoscaling group to create a new EC2 instance and then terminate the old EC2 instance.

Restoring Database from AZ failure:

Due to having an Multi Availability zone deployment, your data will be kept secure if an availability zone goes down. However, you can restore the database via a snapshot.

1. Sign in to the AWS Management Console and open the Amazon RDS console at https://console.aws.amazon.com/rds/.

2. In the navigation pane, choose Snapshots.

3. Choose the DB snapshot that you want to restore from.

4. For Actions, choose Restore snapshot.

5. On the Restore snapshot page, for DB instance identifier, enter the name for your restored DB instance.

6. Choose Restore DB instance.

7. Once you have created a new Database instance with a different name, the database will have the data for the time you specified. However, as it’s a new database, you will need to configure the AWS CloudFormation script. In PanintelligenceThree script, you will need to edit and update the cloudformation script to the new details of the restored database:

8. Once the stack is updated, you will need to scale out your autoscaling group to create a new EC2 instance and then terminate the old EC2 instance.

Restoring from Application Load Balancer AZ failure:

You can enable or disable the Availability Zones for your load balancer at any time. After you enable an Availability Zone, the load balancer starts routing requests to the registered targets in that Availability Zone. Your load balancer is most effective if you ensure that each enabled Availability Zone has at least one registered target.

After you disable an Availability Zone, the targets in that Availability Zone remain registered with the load balancer, but the load balancer will not route requests to them.

To update Availability Zones using the console

1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

2. On the navigation pane, under LOAD BALANCING, choose Load Balancers.

3. Select the load balancer.

4. On the Description tab, under Basic Configuration, choose Edit Availability Zones.

5. To enable a zone, select the check box for that zone and select one subnet. If there is only one subnet for that zone, it is selected. If there is more than one subnet for that zone, select one of the subnets.

6. To change the subnet for an enabled Availability Zone, choose Change subnet and select one of the other subnets.

7. To remove an Availability Zone, clear the check box for that Availability Zone.

8. Choose Save.

Restoring the AWS EFS:

To restore an Amazon EFS file system

1. Open the AWS Backup console at https://console.aws.amazon.com/backup.

2. Your EFS backup vault receives the access policy Deny backup:StartRestoreJob upon creation. If you are restoring your backup vault for the first time, you must change your access policy as follows.

   1. Choose Backup vaults.

   2. Choose the backup vault containing the recovery point you would like to restore.

   3. Scroll down to the vault Access policy

   4. If present, delete backup:StartRestoreJob from the Statement. Do this by choosing Edit, deleting backup:StartRestoreJob, then choosing Save policy.

3. In the navigation pane, choose Protected resources and the EFS file system ID you want to restore.

4. On the Resource details page, a list of recovery points for the selected file system ID is shown. To restore a file system, in the Backups pane, choose the radio button next to the recovery point ID of the file system. In the upper-right corner of the pane, choose Restore.

5. Specify the restore parameters for your file system. The restore parameters you enter are specific to the resource type that you selected.

   You can perform a Full restore, which restores the entire file system. Or, you can restore specific files and directories using Item-level restore.

   1. Choose the Full restore option to restore the file system in its entirety including all root level folders and files.

   2. Choose the Item-level restore option to restore a specific file or directory. You can select and restore up to five items within your Amazon EFS.

      To restore a specific file or directory, you must specify the relative path related to the mount point. For example, if the file system is mounted to /user/home/myname/efs and the file path is user/home/myname/efs/file1, enter /file1. Paths are case sensitive and cannot contain special characters, wildcard characters, and regex strings.

      1. In the Item path text box, enter the path for your file or folder.

      2. Choose Add item to add additional files or directories. You can select and restore up to five items within your EFS file system.

6. For Restore location

   1. Choose Restore to directory in source file system if you want to restore to the source file system.

   2. Choose Restore to a new file system if you want to restore to a different file system.

7. For File system type

   1. (Recommended) Choose Regional if you want to restore your file system across multiple AWS Availability Zones.

   2. Choose One Zone if you want to restore your file system to a single Availability Zone. Then, in the Availability Zone dropdown, choose the destination for your restore.

   For more information, see Managing Amazon EFS storage classes in the Amazon EFS User Guide.

8. For Performance

   1. If you chose to perform a Regional restore, choose either (Recommended) General purpose or Max I/O.

   2. If you chose to perform a One Zone restore, you must choose (Recommended) General purpose. One Zone restores do not support Max I/O.

9. For Enable encryption

   1. Choose Enable encryption, if you want to encrypt your file system. KMS key IDs and aliases appear in the list after they have been created using the AWS Key Management Service (AWS KMS) console.

   2. In the KMS key text box, choose the key you want to use from the list.

10. For Restore role, choose the IAM role that AWS Backup will assume for this restore.

    Note

    If the AWS Backup default role is not present in your account, a Default role is created for you with the correct permissions. You can delete this default role or make it unusable.

11. Choose Restore backup.

    The Restore jobs pane appears. A message at the top of the page provides information about the restore job.

    Note

    If you only keep one weekly backup, you can only restore to the state of the file system at the time you took that backup. You can't restore to prior incremental backups.

12. Once you have created a restored EFS with a different name, the EFS will have the data for the time you specified. However, as it’s a new EFS, you will need to configure the AWS CloudFormation script. In PanintelligenceThree script, you will need to edit and update the cloudformation script to the new details of the restored EFS:

13. Once the stack is updated, you will need to scale out your autoscaling group to create a new EC2 instance and then terminate the old EC2 instance.

Time to deploy and restore

Time to build will take around 45-80 minutes due to waiting for the stack to be completed. This is due to the AWS RDS MariaDB and the EC2 instances being created.

Time to restore:

Restoring the AWS EFS: Estimated time 10-20 minutes.

Restoring the RDS: it takes around 10-20 minutes if you want to rebuild the RDS snapshot from scratch.

Multi-AZ deployment: If an availability zone goes down, it takes around 30-120 seconds to change availability zone .

Health checks, Logging and Troubleshooting

How to check if your Panintelligence dashboard is healthy? How can you troubleshoot if there are issues?

Target groups:

• Log into AWS Management console → Go into EC2 console → On the left side, click into the ‘Target Groups’ tab

• Click into the Panintelligence Target:

• The image below shows where to check for any unhealthy targets

• If you do see any unhealthy targets, you can check the ‘Monitoring’ tab to look at the metrics for more details at a specific time

AWS CloudWatch to check AWS Lambda:

• Lambda: To view the logs you can check AWS Cloudwatch and go into your log groups and find the lambda group of ‘lambda-to-efs’. Please follow this documentation for further information about AWS Cloudwatch

To look at the logs and check the health of the Lambda function to sideload your s3 objects to EFS.

• Log into AWS Management console → Go into AWS CloudWatch console → Click into ‘Log groups’:

• Click into the log group that looks like ‘PanintelligenceTwo-S3ToEFSLambda-*’ :

• If you see any ‘ERROR’ logs, there is a problem when you try to transfer your images and themes from S3 to EFS via Lambda.

• If you are having problems uploading files onto the EC2, check the logs and the lambda configurations that you are uploading the files to the correct folders to S3. These are the allowed folders to pass over to EFS:

AWS RDS MariaDB to check if the Panintelligence Repo is healthy:

• AWS RDS MariaDB: To view the logs, please follow these instructions linked here:

• Log into AWS Management console → Go into AWS RDS console → Click into ‘Databases’ → Click into ‘dashboard’:

• You can check the ‘Monitoring’ tab. This is to see if you are hitting max connections, the CPU Utilisation and memory space :

• You can even go further and check the RDS MariaDB Logs, select the logs near the bottom and select ‘View’ as highlighted below:

Accessing your instance for administration

In order for command line access to your Panintelligence instance, assuming you’ve installed using the CloudFormation script, you will be able to use SSM. On the EC2 Menu, highlight your instance and select the ‘connect’ button, where different options will be presented to you. Select the ‘Session Manager’ tab and click connect.

Database Administration

Panintelligence comes with a database which contains audit information, chart and layout details as well as user information. We strongly recommend that you run this database on AWS RDS mariadb. In order to log into this database, you will need to jump onto it via the application server.

1. Log into your application server using the session manager.

2. Navigate to /var/panintelligence and cat dashboard.env which will contain your database credentials. make a note of the host, username and password.

3. Run:

Panintelligence EC2 Logs and trouble shooting

Within the EC2 instance, you can check the health within.

1. Log into your application server using the session manager.

2. Navigate to /var/panintelligence and docker ps which will contain the status of the dashboard containers. It should be all running other than the MariaDB:

Estimated cost of Infrastructure

Using AWS pricing calculator and the infrastructure that is suggested above. This is calculated for on-demand service and on the high usage of the infrastructure AWS services. This may be lower or higher depending on your egress data and ingress data traffic.

Estimated monthly cost: 82.50 USD

Estimated annual cost: 990.00 USD

Support

From time to time, it may become necessary to ask for assistance. We’re always happy to help!

Please take a look at our support agreement and our SLA agreements on our website here: https://www.panintelligence.com/support-level-responsibility/ . You could always reach out to our support team at support@panintelligence.com.