Versions Compared

Old Version 32

changes.mady.by.user Karen Hogan

Saved on

compared with

New Version Current

changes.mady.by.user development@panintelligence.com

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The architecture diagram below shows an overview of how the components are connected:

  1. The internet gateway allows traffic into the AWS VPC that’s attached to the public route table. Inside the route table, the AWS Application Load Balancer has a route to the internet gateway. The AWS Application Load Balancer will listen to port 80 and port 443. The security group of the AWS Application load Balancer will also allow port 80 and 443

  2. The AWS Application Load Balancer will direct traffic to a healthy EC2 target to access the Panintelligence dashboard on the web browser

  3. The AWS EC2 instances are part of an auto scaling group based on resource demand. Due to multiple instances, the instances connect to an external Maria DB which is an AWS RDS MariaDB for the persistent storage and fault resilience. The auto scaling rule will scale out if it hits 70% of CPU usage. It will scale back down when the CPU usage is below 20%

  4. As long as you’ve got access to the s3 bucket based on your IAM permissions, you can upload ‘images’, ‘themes’ and ‘excel-data’ files to the bucket. Once they are uploaded, it will set off an object creation trigger to the AWS Lambda to migrate the files onto AWS EFS. The AWS EFS is attached to the auto scaling group so the instances have persistent storage

  5. The folders within the S3 bucket will contain your own personal themes, images and excel data

  6. The RDS MariaDB will contain your own personal dashboard configurations

...

  1. Allows outbound network traffic using the NAT Gateway

Image Added

Skills required

Minimum skills to set it up:

...

  • Networking/Security - If you wish to configure the AWS CloudFormation script and you wish to add more services, you will need to make sure you know the Security Groups, Network Access Control Lists and Route tables

  • Python - We have a Lambda function which is written in python. The python script grabs an S3 object event trigger to push to AWS EFS. If you wish to modify it, you will need Python skills

  • MariaDB/SQL commands - MariaDB knowledge on how to access AWS RDS MariaDB and view the Panintelligence dashboard database

  • Docker and Docker-compose - We install the application using configured docker-compose scripts.

Resources and prerequisites

...

Resource

Description

How it is used?

Route 53

Amazon Route 53 is a highly available and scalable cloud Domain Name System (DNS) web service. For more information, please see Setting up Amazon Route 53 documentation

You can attach your domain name to the AWS Application Load Balancer to point to the Panintelligence dashboard.

AWS ACM

AWS Certificate Manager is a service that lets you easily provision, manage, and deploy public and private Secure Sockets Layer/Transport Layer Security (SSL/TLS) certificates for use with AWS services and your internal connected resources. For more information, please see Setting up - AWS Certificate Manager documentation

In order to use port 443/HTTPS in the AWS Application Load Balancer, you will need an SSL certificate.

EC2 Key Pair

A key pair, consisting of a public key and a private key, is a set of security credentials that you use to prove your identity when connecting to an Amazon EC2 instance. Amazon EC2 stores the public key on your instance, and you store the private key. For more information, please see Amazon EC2 key pairs and Linux instances - Amazon Elastic Compute Cloud documentation

If you wish to SSH into the EC2 instance, you will need the Key pair on your local machine.

AWS S3 Bucket

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. For more information, please see AWS S3 Bucket documentation

The architecture requires the user to upload a lambda zip provided in the Git repository and another s3 bucket is created to store images, themes and excel-data.

AWS CloudFormation

AWS CloudFormation gives you an easy way to model a collection of related AWS and third-party resources, provision them quickly and consistently, and manage them throughout their life cycles, by treating infrastructure as code. For more information, please see AWS Cloudformation documentation

AWS CloudFormation allows you to build the infrastructure instead of manually configuring each component.

AWS SSM

SSM Agent makes it possible for Systems Manager to update, manage, and configure these resources. For more information, please see AWS SSM documentation

We use AWS SSM Agent to SSH into the EC2 instance on AWS Management console instead of local machine.

AWS CloudShell

AWS CloudShell is a browser-based, pre-authenticated shell that you can launch directly from the AWS Management Console. You can run AWS CLI commands against AWS services using your preferred shell (Bash, PowerShell, or Z shell). And you can do this without needing to download or install command line tools. For more information, please see What is AWS CloudShell documentation

Instead of doing it on a local machine, you can run the shell commands on AWS CloudShell.

AWS Internet gateway

An internet gateway is a horizontally scaled, redundant, and highly available VPC component that allows communication between your VPC and the internet. For more information, please see Internet Gateways documentation

The Panintelligence dashboard requires web browser access.

AWS IAM

AWS Identity and Access Management (IAM) enables you to manage access to AWS services and resources securely. Using IAM, you can create and manage AWS users and groups, and use permissions to allow and deny their access to AWS resources. For more information, please see AWS IAM documentation

IAM permissions allows you to have fine grain control on who and what has access to resources.

AWS Security groups

A security group acts as a virtual firewall for your instance to control inbound and outbound traffic.  For more information, please AWS Security Groups documentation

Increase protection to your infrastructure.

AWS Application Load Balancer

Elastic Load Balancing automatically distributes your incoming traffic across multiple targets, such as EC2 instances, containers, and IP addresses, in one or more Availability Zones. It monitors the health of its registered targets, and routes traffic only to the healthy targets.  For more information, please see AWS Application Load Balancer documentation

The ALB directs traffic to the healthy EC2 targets.

AWS VPC

Amazon Virtual Private Cloud (Amazon VPC) enables you to launch AWS resources into a virtual network that you've defined. This virtual network closely resembles a traditional network that you'd operate in your own data centre, with the benefits of using the scalable infrastructure of AWS. For more information, please see AWS VPC documentation

We use the AWS VPC to launch resources in the virtual network.

Subnets

You need to specify a logical address to specific resources. For more information, please see Subnets documentation

Configure resources to specific subnet cidr blocks.

NACL

A network access control list (ACL) is an optional layer of security for your VPC that acts as a firewall for controlling traffic in and out of one or more subnets. For more information, please see NACL documentation

Configure additional security.

AWS Lambda

AWS Lambda is a serverless compute service that lets you run code without provisioning or managing servers, creating workload-aware cluster scaling logic, maintaining event integrations, or managing runtimes. For more information, please see AWS Lambda documentation

The infrastructure uses AWS Lambda to side load S3 objects to AWS EFS

AWS RDS MariaDB

Amazon Relational Database Service (Amazon RDS) is a web service that makes it easier to set up, operate, and scale a relational database in the AWS Cloud. For more information, please see AWS RDS MariaDB documentation

The Panintelligence dashboard uses AWS RDS MariaDB as an external DB.

AWS EFS

Amazon Elastic File System (Amazon EFS) provides a simple, serverless, set-and-forget elastic file system for use with AWS Cloud services and on-premises resources. For more information, please see AWS EFS documentation

AWS EFS is used to keep persistent data for themes, images and excel data.

AWS Auto scaling

AWS Auto Scaling monitors your applications and automatically adjusts capacity to maintain steady, predictable performance at the lowest possible cost. For more information, please see AWS Auto scaling documentation

Auto scaling is used to increase or decrease the EC2 instances depending on traffic.

AWS EC2

Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides secure, resizable compute capacity in the cloud. For more information, please see AWS EC2 documentation

Using AWS EC2 to stand up Panintelligence AMI.

AWS AMI

An Amazon Machine Image (AMI) provides the information required to launch an instance.  For more information, please see AWS AMI documentation

Panintelligence has four AMI’s on the marketplace.

AWS NAT gateway

A NAT gateway is a Network Address Translation (NAT) service. You can use a NAT gateway so that instances in a private subnet can connect to services outside your VPC but external services cannot initiate a connection with those instances. Please see AWS NAT gateway documentation

Allows you to use Panintelligence Automated Lincence Manager

Service quotas

For the AWS Services that you will be using, you will need to be aware of your service level quotas. You don’t want to hit your limit on your account, however you can always submit a Service quota increase with AWS. For more information on what are AWS Service quotas, click here.

...

AWS Resource

Service quotas that the infrastructure uses

Notes

AWS EC2

1-5 EC2 instances

AWS Route 53 (Optional)

1

Launch configuration per region

1

Step adjustments per step scaling policy

2

One policy to decrease EC2 instances and one to increase EC2 instances.

Target groups per Auto Scaling group

1

AWS Auto scaling per region

1

AWS VPC

1

AWS Internet gateway

1

General Purpose SSD (gp2) volume storage

2-10

AWS Application load balancer

1

AWS Lambda Elastic network interfaces per VPC

1

AWS Lambda Function time out

15

AWS Lambda temporary storage

Not available

AWS Default quota value is 512MB, A Panintelligence theme or image would not hit the limit.

AWS CloudFormation stacks

10

EFS per VPC

1

EFS Mount targets

2

EFS attached security group

1

Interface VPC endpoints per VPC

5

Route tables per VPC

2

Private and public route table

Routes per route table

2

Subnets per VPC

6

Security groups

6

Network ACLs per VPC

2

S3 Buckets

2

Secrets per account

6

DB instance

1

DB subnet group

1

DB Parameter group

1

DB security group

1

AWS NAT Gateway

1

How to obtain a Panintelligence licence key?

...

Developer and trial both have limited use case licences embedded in the AMI image (Amazon Machine Image). Our Metered offering charges based on units (users) and dimensions (analytics, scheduler, reports).

...

Automated Licence Manager

For more information on how to use the automated licence feature:

Automated Licence Manager

How to obtain the Panintelligence Marketplace AMI ID?

...

In addition, on Panintelligence AMI we have Linux Ubuntu 20.04 as the operating system.

Docker overview

For more information on how the docker configuration within the AMI products:

AWS AMI products

Size requirements and recommendations

...

How to obtain the secrets?

AWS Management console:

  1. If not already logged into the console, go to the console at https://console.aws.amazon.com/secretsmanager/ and log into the Secrets Manager service.

  2. On the Secrets list page, choose the name of the new secret you created.

    Secrets Manager displays the Secrets details page for your secret.

  3. In the Secret value section, choose Retrieve secret value.

  4. You can view your secret as either key-value pairs, or as a JSON text structure.

AWS CLI:

  1. Open a command prompt to run the AWS CLI. If you haven't installed the AWS CLI yet, see Installing the AWS Command Line Interface.

  2. Using credentials with permissions to access your secret, type the following command and parameters.

Code Block
aws secretsmanager describe-secret --secret-id MyRDSSecret

...

Due to having an Multi Availability zone deployment, your data will be kept secure if an availability zone goes down. However, you can restore the database via a snapshot.

  1. Sign in to the AWS Management Console and open the Amazon RDS console at https://console.aws.amazon.com/rds/.

  2. In the navigation pane, choose Snapshots.

  3. Choose the DB snapshot that you want to restore from.

  4. For Actions, choose Restore snapshot.

  5. On the Restore snapshot page, for DB instance identifier, enter the name for your restored DB instance.

  6. Choose Restore DB instance.

  7. Once you have created a new Database instance with a different name, the database will have the data for the time you specified. However, as it’s a new database, you will need to configure the AWS CloudFormation script. In PanintelligenceThree script, you will need to edit and update the cloudformation script to the new details of the restored database:

    Code Block
    DBHost: {Enter new db hostname}
DBPassword: {Enter new db password}
DBPort: {Enter new db port}
DBUsername: {Enter new db username}

8. Once the stack is updated, you will need to scale out your autoscaling group to create a new EC2 instance and then terminate the old EC2 instance.

...

To update Availability Zones using the console

  1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

  2. On the navigation pane, under LOAD BALANCING, choose Load Balancers.

  3. Select the load balancer.

  4. On the Description tab, under Basic Configuration, choose Edit Availability Zones.

  5. To enable a zone, select the check box for that zone and select one subnet. If there is only one subnet for that zone, it is selected. If there is more than one subnet for that zone, select one of the subnets.

  6. To change the subnet for an enabled Availability Zone, choose Change subnet and select one of the other subnets.

  7. To remove an Availability Zone, clear the check box for that Availability Zone.

  8. Choose Save.

Restoring the AWS EFS:

To restore an Amazon EFS file system

  1. Open the AWS Backup console at https://console.aws.amazon.com/backup.

  2. Your EFS backup vault receives the access policy Deny backup:StartRestoreJob upon creation. If you are restoring your backup vault for the first time, you must change your access policy as follows.

    1. Choose Backup vaults.

    2. Choose the backup vault containing the recovery point you would like to restore.

    3. Scroll down to the vault Access policy

    4. If present, delete backup:StartRestoreJob from the Statement. Do this by choosing Edit, deleting backup:StartRestoreJob, then choosing Save policy.

  3. In the navigation pane, choose Protected resources and the EFS file system ID you want to restore.

  4. On the Resource details page, a list of recovery points for the selected file system ID is shown. To restore a file system, in the Backups pane, choose the radio button next to the recovery point ID of the file system. In the upper-right corner of the pane, choose Restore.

  5. Specify the restore parameters for your file system. The restore parameters you enter are specific to the resource type that you selected.

    You can perform a Full restore, which restores the entire file system. Or, you can restore specific files and directories using Item-level restore.

    1. Choose the Full restore option to restore the file system in its entirety including all root level folders and files.

    2. Choose the Item-level restore option to restore a specific file or directory. You can select and restore up to five items within your Amazon EFS.

      To restore a specific file or directory, you must specify the relative path related to the mount point. For example, if the file system is mounted to /user/home/myname/efs and the file path is user/home/myname/efs/file1, enter /file1. Paths are case sensitive and cannot contain special characters, wildcard characters, and regex strings.

      1. In the Item path text box, enter the path for your file or folder.

      2. Choose Add item to add additional files or directories. You can select and restore up to five items within your EFS file system.

  6. For Restore location

    1. Choose Restore to directory in source file system if you want to restore to the source file system.

    2. Choose Restore to a new file system if you want to restore to a different file system.

  7. For File system type

    1. (Recommended) Choose Regional if you want to restore your file system across multiple AWS Availability Zones.

    2. Choose One Zone if you want to restore your file system to a single Availability Zone. Then, in the Availability Zone dropdown, choose the destination for your restore.

    For more information, see Managing Amazon EFS storage classes in the Amazon EFS User Guide.

  8. For Performance

    1. If you chose to perform a Regional restore, choose either (Recommended) General purpose or Max I/O.

    2. If you chose to perform a One Zone restore, you must choose (Recommended) General purpose. One Zone restores do not support Max I/O.

  9. For Enable encryption

    1. Choose Enable encryption, if you want to encrypt your file system. KMS key IDs and aliases appear in the list after they have been created using the AWS Key Management Service (AWS KMS) console.

    2. In the KMS key text box, choose the key you want to use from the list.

  10. For Restore role, choose the IAM role that AWS Backup will assume for this restore.

    Note

    If the AWS Backup default role is not present in your account, a Default role is created for you with the correct permissions. You can delete this default role or make it unusable.

  11. Choose Restore backup.

    The Restore jobs pane appears. A message at the top of the page provides information about the restore job.

    Note

    If you only keep one weekly backup, you can only restore to the state of the file system at the time you took that backup. You can't restore to prior incremental backups.

  12. Once you have created a restored EFS with a different name, the EFS will have the data for the time you specified. However, as it’s a new EFS, you will need to configure the AWS CloudFormation script. In PanintelligenceThree script, you will need to edit and update the cloudformation script to the new details of the restored EFS:

    Code Block
    EfsFileSystemId: {New Filesystem id here}

  13. Once the stack is updated, you will need to scale out your autoscaling group to create a new EC2 instance and then terminate the old EC2 instance.

Time to deploy and restore

...

Panintelligence comes with a database which contains audit information, chart and layout details as well as user information. We strongly recommend that you run this database on AWS RDS mariadb. In order to log into this database, you will need to jump onto it via the application server.

  1. Log into your application server using the session manager.

...

Substitute user to pi-user sudo su - pi-user

  1. Navigate to

...

  1. /

...

  1. var/

...

  1. panintelligence and cat

...

  1. dashboard.

...

  1. env which will contain your database credentials. make a note of the host, username and password.

...

  2. Run:

    Code Block
    docker run -it mariadb:latest bash
mysql -h<<DB RDS Host>> -u<<Username>> -p<<Password>> -P3306

Panintelligence EC2 Logs and trouble shooting

Within the EC2 instance, you can check the health within.

  1. Log into your application server using the session manager.

...

Substitute user to pi-user sudo su - pi-user

  1. Navigate to

...

  1. /

...

  1. var/

...

  1. panintelligence and docker ps which will contain the status of the dashboard containers. It should be all running other than the MariaDB:

...

Info

If you see any other service not running, please check the logs /opt/pi/Dashboard/tomcatvar/panintelligence/logs / to investigate. You could always start the services manually too. In addition, if a container is unhealthy you can use “docker logs <<CONTAINER ID >>”

Estimated cost of Infrastructure

...

Region

Service

Monthly

First 12 months total

Currency

Configuration summary

EU (Ireland)

Amazon EC2

31.87

382.44

USD

Operating system (Linux), Quantity (1), Pricing strategy (On-Demand Instances), Storage amount (15 GB), Instance type (t3.medium)

EU (Ireland)

S3 Standard

0.02

0.24

USD

S3 Standard storage (1 GB per month)

EU (Ireland)

Data Transfer

0

0

USD

EU (Ireland)

Amazon Elastic File System (EFS)

0.17

2.04

USD

Desired Storage Capacity (2 GB per month)

EU (Ireland)

AWS Lambda

0

0

USD

Number of requests (15)

EU (Ireland)

Amazon RDS for MariaDB

31.34

376.08

USD

Storage volume (General Purpose SSD (gp2)), Storage amount (20 GB per month), Quantity (1), Instance type (db.t2.micro), Deployment selection (Multi-AZ), Pricing strategy (OnDemand)

EU (Ireland)

Application Load Balancer

19.1

229.2

USD

Number of Application Load Balancers (1)

EU (Ireland)

NAT Gateway

35.18

422.16

USB

Number of NAT Gateway (1). Estimated 3GB egress traffic (Can be lower estimated network traffic)

Support

From time to time, it may become necessary to ask for assistance. We’re always happy to help!

...