Containers represent a repeatable, immutable, and portable way to run software on-premise or in the cloud. Coupled with Container orchestration, it means an opportunity to deploy software in a highly resilient, and scalable fashion. At Panintelligence, we offer our software in containers to facilitate your stable deployment of our software.

Architecture overview

Migrating from all-in-one, Marialess, and other deployments of Panintelligence

What are microservices

Traditionally built software required components to be deployed next to each other with mutual dependencies. If there were to be a failure with any part of that software, or if a package needed to be updated that caused issues with other parts of the software, this would create difficulties. Containers represent the opportunity to create scope by which smaller components for software that can still talk to each other, can be deployed in a way that no longer is mutually reliant on other functions. Pi is designed this way.

5 distinct microservices form the architectural blend for Pi. These are:

Dashboard

This is the nucleus of the pi infrastructure. Nothing much will work without this.

Scheduler

This handles the report generation and task scheduler for the dashboard.

Analytics

This is the analytics engine for the dashboard, formally known as Pirana. This microservice is fundamental to forming the mushroom maps in the dashboard.

Renderer

This is the service that takes HTML charts and renders them into whatever shape the end user requires. The more resources you can dedicate to the renderer, the better. Fewer resources means slower renders.

excel-reader

This stages an Excel file into a SQLite database for consumption with the dashboard.

why would I want to use separate containers?

Some elements of any software solution are more heavily accessed than others. By breaking it into smaller chunks, you have better control over scaling each part. Additionally, in the event of a failure, you don’t lose access to other functioning areas of the software, as you would with a monolith. Likewise, a misbehaving component will not collapse your entire stack. It also permits you to monitor individual components more closely for failures and develop automated recovery plans.

how do I migrate from a single container-based installation to separate containers?

Database

you will need to be able to execute mysqldump and mysql commands against your database. You could install MySQL locally, or you could make use of docker by running the following command on your local machine.

This will put you into an interactive session on the container which will have the SQL tools you need to effect a backup and restore. Please note that container filesystems are immutable. If you remove your container, any extract you’ve taken will be lost, unless you copy it to your host filesystem. You can copy files from your container by following this set of instructions.

Panintelligence uses a repository database containing metadata on defining elements in the dashboard such as categories, charts, users, user roles and permissions, and audit information. To back this up, you will need to perform the following action:

You will then need to insert this data into your new database. be mindful, that if you have already created your database by launching Panintelligence and are trying to overwrite, you will need to stop your dashboard server. This is because there are table locks generated by the dashboard application which will prevent the drop action of the SQL script you generated when creating your backup.

Persistent items

you will now want to take a backup of the following items and mount the volumes to your new deployment. you can find information about docker volumes here.

• Themes

  • ${source_dir}/Dashboard/tomcat/webapps/panMISDashboardResources/themes

• Images

  • ${source_dir}/Dashboard/tomcat/webapps/panMISDashboardResources/images

• Custom JDBC

  • ${source_dir}/Dashboard/tomcat/custom_jdbc_drivers

• SVG

  • ${source_dir}/Dashboard/tomcat/webapps/panMISDashboardResources/svg

Quick Start

Links here to a docker-compose including MariaDB container which is our reference deployment with only our software + maria (i.e. the most straightforward setup utilizing multiple containers).

Common Deployment Pattern

AWS - ECS

terraform

• install terraform Install Terraform | Terraform | HashiCorp Developer

• ensure you have AWS CLI installed and configured https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

• download the terraform script found at https://github.com/Panintelligence/terraform-ecs

• create the “hosted_zone_edit_role” and permission

  aws iam create-policy \ --policy-name pi-hosted-zone-edit \ --policy-document '{ "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Action": [ "route53:ListTagsForResource", "route53:ListResourceRecordSets", "route53:GetHostedZone", "route53:ChangeResourceRecordSets" ], "Resource": "arn:aws:route53:::hostedzone/${HOSTED_ZONE_ID}" }, { "Sid": "", "Effect": "Allow", "Action": "route53:ListHostedZones", "Resource": "*" }, { "Sid": "", "Effect": "Allow", "Action": "route53:GetChange", "Resource": "arn:aws:route53:::change/*" } ] }' aws iam create-role \ --role-name MyExampleRole \ --assume-role-policy-document '{ "Version":"2012-10-17", "Statement": [ {"Effect":"Allow", "Principal":{"AWS":"arn:aws:iam::${ACCOUNT_ID}:root"}, "Action":"sts:AssumeRole"} ] }'

• configure your key and secret key before executing these scripts

  export AWS_ACCESS_KEY_ID="anaccesskey" export AWS_SECRET_ACCESS_KEY="asecretkey" export AWS_REGION="us-west-1" export DEPLOYMENT_NAME="sampledeployment" export HOSTED_ZONE_ID="your aws hosted zone id" export CERTIFICATE_ARN="your certificate arn" export HOSTED_ZONE_EDIT_ROLE_ARN="role that permits editing of your hosted zone" export DASHBOARD_DOCKER_TAG="2024_04" export RENDERER_DOCKER_TAG="2024_04" export PIRANA_DOCKER_TAG="2024_04" export DB_PASSWORD="5UP3RsECUR3p455W0Rd123!" export DB_USERNAME="pi_db_admin" export DOCKER_USERNAME="yourgithubusername" export DOCKER_PASSWORD="yourgithubaccesstoken" export LICENCE_KEY="panintelligence-licence" export DOMAIN="example.com" export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) export STATE_BUCKET="${ACCOUNT_ID}-panintelligence-tfstate"

• create the s3 state bucket

  aws s3api create-bucket --bucket ${STATE_BUCKET} --create-bucket-configuration LocationConstraint=$AWS_REGION

• create the efs_prep lambda function

  <project_dir>/build_lambda.sh

• create your ACM certificate as per instructions https://docs.aws.amazon.com/acm/latest/userguide/gs-acm-request-public.html

• initialise the terraform configuration

  terraform init -backend-config="bucket=${STATE_BUCKET}" \ -backend-config="region=${AWS_REGION}" \ -backend-config="key=pi_dashboard/${DEPLOYMENT_NAME}-terraform.tfstate"

• plan the changes

  terraform plan -out=plan \ -var="deployment_name=${DEPLOYMENT_NAME}" \ -var="hosted_zone_id=${HOSTED_ZONE_ID}" \ -var="certificate_arn=${CERTIFICATE_ARN}" \ -var="hosted_zone_edit_role_arn=${HOSTED_ZONE_EDIT_ROLE_ARN}" \ -var="dashboard_docker_tag=${DASHBOARD_DOCKER_TAG}" \ -var="renderer_docker_tag=${RENDERER_DOCKER_TAG}" \ -var="dashboard_db_password=${DB_PASSWORD}" \ -var="dashboard_db_username=${DB_USERNAME}" \ -var="docker_hub_credentials={\"username\":\"${DOCKER_USERNAME}\",\"password\":\"${DOCKER_PASSWORD}\"}" \ -var="licence_key=${LICENCE_KEY}" \ -var="region=${AWS_REGION}"

• apply the configuration to your target aws account

  terraform apply plan

• invoke the configuration lambda

  aws lambda invoke --function-name ${DEPLOYMENT_NAME}_dashboard_prep --payload '{}' out --log-type Tail

Deleteing EFS

• remove the EFS backup vault

  $EFS_VAULT_NAME=panintelligence_efs_backup_${DEPLOYMENT_NAME} EFS_BACKUP_ARN = $(aws backup list-recovery-points-by-backup-vault --backup-vault-name ${VAULT_NAME}" --query 'RecoveryPoints[].RecoveryPointArn' --output text) aws backup delete-recovery-point --backup-vault-name "${VAULT_NAME}" --recovery-point-arn "${EFS_BACKUP_ARN}"

• tear down using terraform scripts

  terraform plan -destroy -out=plan \ -var="deployment_name=${DEPLOYMENT_NAME}" \ -var="hosted_zone_id=${HOSTED_ZONE_ID}" \ -var="certificate_arn=${CERTIFICATE_ARN}" \ -var="hosted_zone_edit_role_arn=${HOSTED_ZONE_EDIT_ROLE_ARN}" \ -var="dashboard_docker_tag=${DASHBOARD_DOCKER_TAG}" \ -var="renderer_docker_tag=${RENDERER_DOCKER_TAG}" \ -var="dashboard_db_password=${DB_PASSWORD}" \ -var="dashboard_db_username=${DB_USERNAME}" \ -var="docker_hub_credentials={\"username\":\"${DOCKER_USERNAME}\",\"password\":\"${DOCKER_PASSWORD}\"}" \ -var="licence_key=${LICENCE_KEY}" \ -var="region=${AWS_REGION}"

• remove s3 terraform state files and bucket

  aws s3 rm s3://${STATE_BUCKET} --recursive aws s3api delete-bucket --bucket ${STATE_BUCKET}

Resilience

It’s a good idea to deploy your dashboard across more than one availability zone and also to employ an auto-scaling group on the analytics, renderer, and dashboard services. You must only run a single Scheduler task. The dashboard is based on Java, which ringfences memory, as a result, you must use CPU as the value to trigger scaling.

Container insights

In order to gain more insight into the performance of your instances running in an ECS cluster, it will be important to enable “container insights”. Be aware, there is an additional cost. You will receive basic telemetry on your service performance without container insights.

Logs

you must enable a log group for each of your task definitions to ensure you receive logs. Without this, the logs from the container will not be captured. As a prerequisite, you will need to create the custom log group before defining this in your task definition.

Kubernetes

Persistence

Panintelligence requires some files to be shared amongst pods. to achieve this end, you will need to create a persistent volume. This must be of a storage Capacity of at least 5Gi, however it’s advised to give some space to grow (should you wish to include more custom jdbc, images or themes). an example is shown below

For simplicity, you can choose to copy this to a file pi-pv.yaml and execute kubectl apply -f pi-pv.yaml to create the persistent volume for use by the panintelligence deployment in your cluster.

Secrets

To access panintelligence containers, we will need to whitelist your user. Please contact your CSM who will grant you this access. you will then need to create a secret which is to be used by the helm charts.

Helm

Please find our helm charts at https://github.com/Panintelligence/helm

Ingress

here is an example ingress for the dashboard.

Azure

Azure allows for multiple deployment methodologies. The most commonly used are App service and kubernetes.

App Service

As previously stated, Panintelligence can be deployed as distinct web applications, which allows you to choose which components you wish to deploy based on your requirements, tailoring your deployment plan to your current needs.

Please note: azure have deprecated mariadb. we’re working to update this document to reflect this. You’re welcome to make these changes for yourself in the meantime!

execute the following statements:

Prerequisites

Please run to set these environment variables before executing any other scripts.

Dashboard

please run the prerequisites before running this script in the AZ cli.

Renderer

please run the prerequisites before running these scripts.

After your service is verified as being available, you will need to set up the renderer by logging into the dashboard, and then setting the PAN_RENDERER_URL to the URL of the renderer you’ve just deployed. it should look something like “pan<your environment name><environment>renderer”. If you apply a custom domain, this will change.

Open

Scheduler

Analytics

Pulling the images to a local repository

We recommend not relying on a third party to host your containers for your business continuity. Therefore we highly recommend pulling, retagging, and then pushing your containers to your private image repository.

We’d also advise against using the “latest” tag in production but use specific versions so you can control when and how you migrate to the latest version of our software.

Healthchecks

how do we know the container is functioning? we use a health check! for orchestration, you can use the following endpoints that will all return a http status code of 200 when the software is functioning well:

Server

Renderer

Scheduler

Pi Analytics

Configuring your containers

Containers are immutable. If a container stops working (determined by the healthcheck) then you will want to start another container immediately. some configuration will need to be persisted using environment variables, and some using volumes

Environment Variables

Our example scripts below show the minimum environment variables you will need to get up and running, there are however many other options which are listed on our Environment Variables documentation page

Example of a standard environment configuration

Volumes

Certain items used by Panintelligence software need to be stored on a file system (these will be read-write items). Below is a list of the volumes you may need to setup and what they are used for. Our example scripts always include these.

Dashboard

Themes

Internal Location: /var/panintelligence/Dashboard/tomcat/webapps/panMISDashboardResources/themes

Use: Custom UI themes for Panintelligence software

Images

Internal Location: /var/panintelligence/Dashboard/tomcat/webapps/panMISDashboardResources/images

Use: custom images

Keys

Internal Location: /var/panintelligence/Dashboard/keys/

Use: Generated keys used for communication between dashboard and scheduler services

Custom JDBC

Internal Location: /var/panintelligence/Dashboard/tomcat/custom_jdbc_drivers

Use: Additional JDBC drivers that are not bundled, used for free-format jdbc connections

SVG

Internal Location: /var/panintelligence/Dashboard/tomcat/webapps/panMISDashboardResources/svg

Use: SVG images for custom maps functionality

Specifications

Server

Notes

We’ve stress-tested this combination to 45 concurrent users. The simulation for the test included the simulant users making constant requests to the dashboard. This is heavy, almost excessive use of the dashboard, and not precisely the usage patterns you’d find from a user. As a result, under real-world conditions, this configuration will likely support many more users.

Renderer

Notes

If you’re rendering in “real time” then you will want to give the renderer as much resource as you can spare. This is because the more resources you give the renderer, the faster the results.

Scheduler

Analytics

Questions and Answers

When configured to use Redis for persistent session sharing amongst application nodes, how do i verify it’s working

You can achieve this one of 3 ways. You could drain all your application nodes and restart them. Once they’re reporting as healthy, if you’re able to log in without needing to authenticate, then your session data has persisted.

The second method is far less invasive. When starting the dashboard, a log entry is made stating that a connection to the redis service is assumed.

The third method is to connect to the redis system and view the records changing. You may wish to add an agent that is reactive to changes to your redis collection for the dasboard session data and present this information back as a metric to your monitoring solution.

Shared volumes

The dashboard holds some persistent files that dictate the look and the feel of Pi. Here are some suggestions on how you can manage this data within your Pi deployment.

Writing changes to files

You may wish to guard your production system from changes and push modifications through source control. When deploying using kubernetes, it could be worth deploying your theme assets using a configmap. This would mean that themes are non editable in production, so it would be worth generating these themes either on a local machine or a development instance. Please contact your CSM to obtain a licence for this application.

Write operations to themes is normally done by an administrator, which is a restricted user, therefore the chances of conflict during updates is greatly reduced, however they do sometimes occur.

What sort of storage do i need in my shared volume?

This depends on how many themes, images to serve the dashboard directly, locales, svgs and the number and size of your custom JDBC drivers. Having said that, with the exception of custom JDBC, the majority of these files tend to be smaller. For example, your images should be optimised for web, so shouldn’t be more than a few KB in size. To this end, generally less than 100MB and often far less storage is required. Additionally, the data will not require fast access, so using a cloud based object based file system to back a volume is an option, if you’re worried about scale.

What about backing up these volumes?

This data tends to be slow moving, once set will not change often. If your subjecting this data to a change control process, then it would be even easier to manage configuration drift. You should backup your data to 3 locations, including one that is immutable.

Can I use my Redis service to host images and themes?

It’s not recommended to use redis to store images and themes, and is presently not a feature that Pi supports.

Scaling

Dashboard

state for the dashboard app node is held in your persistent mounted volumes as detailed above, with session data being held in Redis where configured. If you do not have session sync enabled, you would need to take advantage of sticky sessions from your load balancer.

Renderer

The renderer is stateless. You can scale the renderer either vertically (speed up renders) or horizontally (meet demand profile more accurately). You may wish to bring additional renderer nodes online if you know reports are run at specific times during the day.

Scheduler

Currently Pi only needs one single renderer application service. Scaling this to more than one node may have adverse effects and send multiple reports, where one was expected. We have several feature upgrades planned for the renderer over the coming months, so speak to your CSM.