September 2024 - Dashboard Release Notes

 

  • Customer Request tickets are highlighted with the icon

  • The 2024_09 release contains x1 Schema change

  • IMPORTANT: New Minimum Server Requirements

 

IMPORTANT: As a precaution, we removed the ability to download the September 24 release whilst we investigated a couple of issues. To take advantage of the changes listed in this release note, you will need to upgrade to the November 24 release.


 

N.B. We have revised the minimum server requirements for a production system node. This also includes a recommended change to environment variables - see HERE for more information.

NEW FEATURES

 

Custom Fields (BETA)

-

29554

Custom Fields

Custom Fields (BETA)

-

29554

Custom Fields

We’ve added an innovative way to unlock key-value data in your databases which has traditionally been difficult to report on.

Known as Custom Fields, you can now provide additional configuration to have the Dashboard create objects automatically from the key-value data where previously you would have needed to manually create these for each item. Click HERE for more information on the Custom Fields feature.

When opting to use a BETA feature, we always recommend doing so in a Test environment first, ensuring you are familiar with the feature before you use it in your Production instance. We would encourage you to reach out to your Customer Success Manager to discuss such features, and arrange additional support from our Professional Service team, who can assist with introducing the features into your instances.

 

Single Chart Word Export - Performance Improvement (BETA)

pi#2438

29822 & 32731

Exporting - Word

Single Chart Word Export - Performance Improvement (BETA)

pi#2438

29822 & 32731

Exporting - Word

We’re excited to offer a completely rebuilt Word (docx) export functionality which is almost instant and reflects any changes you make to charts on your screen (including legend selection).

You can try it out by enabling the beta feature BROWSER_WORD_EXPORT (in pancredit_parameters table) and we’d love to hear your feedback and incorporate feedback into our upcoming releases. At present it only supports certain chart types (highcharts) which doesn’t include tables, card charts, maps. Any unsupported will simply fall back to the existing export functionality.

Example of the section from the new Word output:

Screenshot from 2024-08-15 09-23-13.png

 

 

Improved Preview Of Chart Headings - When Categories Load/Refresh

pi#2532

31377

Category Preview

Improved Preview Of Chart Headings - When Categories Load/Refresh

pi#2532

31377

Category Preview

Before charts are fully rendered, we populate the cell with the Chart Title to give an indication of what is loading in. Previously, this was just the Chart Name but now this preview will more closely resemble the panel heading that will appear when the chart fully loads. It will show some of the icons that will be displayed as well as the title/subtitle appropriate for that chart. Replacement cannot be done at this stage meaning any form of replacement within the title/subtitle will result in the preview not including that element. Translations, however, are available so you can expect any translated values to be represented in the accurately.

 

New User/Role Permission (For Admin Users) - To Access The API Configuration Tab

pi#2524

24595 & 31607

User/Role Permissions

New User/Role Permission (For Admin Users) - To Access The API Configuration Tab

pi#2524

24595 & 31607

User/Role Permissions

A new User/Role permission has been added at the Admin level, to determine whether or not the User is allowed to access/edit the configuration within the APIs tab. The new “edit apis” permission defaults to ‘True’ for Admin users at the User Permission level and ‘False’ for Roles.

 

CHANGES

 

Exporting - Enhanced Single Chart PDF & PowerPoint (Performance Improvements) - REMOVED FROM BETA

pi#2600

-

Exporting - PDF & PowerPoint

Exporting - Enhanced Single Chart PDF & PowerPoint (Performance Improvements) - REMOVED FROM BETA

pi#2600

-

Exporting - PDF & PowerPoint

Our greatly enhanced, rebuilt pdf and ppt exports for single charts have been enabled by default in this release. If you experience any issue please let us know, for a limited time you can switch back to the old mode by using the beta flag OLD_PDF_EXPORT and OLD_PPT_EXPORT.

 

Exporting - Enhanced Single Chart PDF & PowerPoint - A Drill Down Chart

pi#2599

-

Exporting - PDF & PowerPoint

Exporting - Enhanced Single Chart PDF & PowerPoint - A Drill Down Chart

pi#2599

-

Exporting - PDF & PowerPoint

Resolved a Known Issue with exporting drilldown-level charts, when using the enhanced Single Chart PDF & PowerPoint export functionality - a Known Issue with the August 24 release.

When a user exports a chart at a drilldown level, the export will now correctly capture the chart from the appropriate hierarchy level, provided the chart type is supported. If the chart type is unsupported, the export will automatically revert to using the renderer service.

 

Editable Tables - Improved Performance When Sending All Data

pi#2545

32019

Editable Tables

Editable Tables - Improved Performance When Sending All Data

pi#2545

32019

Editable Tables

By removing some additional variable replacement that wasn’t useful we’ve improved the performance of editable tables. Especially when sending large volumes of data.

 

Editable Tables - Page Resize & Other Changes Causing Duplicate Data To Be Sent

pi#2550

32019

Editable Tables

Editable Tables - Page Resize & Other Changes Causing Duplicate Data To Be Sent

pi#2550

32019

Editable Tables

Resolved an issue where page actions such as a resize could result in duplicate rows of table data could be sent to the editable tables endpoint.

 

Reports - All Charts Load In Parallel - Improved Performance

pi#2539

31184

Reports - Performance

Reports - All Charts Load In Parallel - Improved Performance

pi#2539

31184

Reports - Performance

Improved report performance by allowing each chart within a report to load in parallel. This does not affect how the charts are rendered, only how they are returned from the back end. Report performance logs have been updated to display the loading times for each individual chart, as well as an overall loading time, which will now be lower due to the requests for individual charts being asynchronous.

 

Schedules - Improvement Reliability When Scheduling Tables (Excel Format)

scheduler#88

31192 & 31778

Scheduler & Exports

Schedules - Improvement Reliability When Scheduling Tables (Excel Format)

scheduler#88

31192 & 31778

Scheduler & Exports

Enhanced the reliability of scheduling tables in Excel format by implementing additional checks to ensure the file's integrity before returning it to the scheduler, along with improvements in naming conventions, increased logging, and better error handling.

 

Analytics - Unable To Save Analytics Model

pirana#52

31872

Analytics

Analytics - Unable To Save Analytics Model

pirana#52

31872

Analytics

Despite having the necessary permissions, Admin Users (other than Super Admin) were unable to save Analytics models to the Data Connection, due to a CORS setting issue.

 

Analytics - Encoded Values Appearing On Analytics Charts

pi#2536

31871

Analytics

Analytics - Encoded Values Appearing On Analytics Charts

pi#2536

31871

Analytics

Comparators like ‘>=' and special characters like '&’ would appear HTML encoded on Analytics Charts.

 

Password Reset - Invalid Details Led To Exception

pi#2634

-

Password Reset

Password Reset - Invalid Details Led To Exception

pi#2634

-

Password Reset

Using the password reset function with invalid details previously lead to an exception being displayed.

 

Organisation - Incorrect Organisation Name Showing In The Configuration Panel

pi#2648

33177

Organisations

Organisation - Incorrect Organisation Name Showing In The Configuration Panel

pi#2648

33177

Organisations

Within the Organisations Configuration Panel dropdown list, users will now only see Organisation values where they are either an admin or have 'schedule' permission for those Organisations. If they are an admin, all content will be displayed relating to those Organisations, otherwise an error will be returned that informs the user that they do not have access.

Security - Resolved some issues with CORS settings

pi#2615

-

Security

Security - Resolved some issues with CORS settings

pi#2615

-

Security

Resolved an issue where the CORS settings were not being applied correctly.

Performance - Several Performance Improvements

pi#2666 & pi#2625

-

Performance

Performance - Several Performance Improvements

pi#2666 & pi#2625

-

Performance

A number of areas of performance improvement were identified and implemented

 

Translations - Several Improvements To Enable More Translations

-

32769, 32816, 32821, 31377

Multi Lingual

Translations - Several Improvements To Enable More Translations

-

32769, 32816, 32821, 31377

Multi Lingual

A number of additional areas where translations occur in the dashboard have now been added.

 

SCHEMA CHANGES

 

Description

GitLab

ZenDesk

Column

Description

GitLab

ZenDesk

Column

Column ‘edit apis’

  • added to mis_users and mis_roles tables

pi#2524

24595 & 31607

User Roles / Permissions

 

MINIMUM SERVER REQUIREMENTS

 

Background

During our internal pre-release testing of this September 2024 dashboard release, we observed some notable performance degradation within the dashboard. This coupled with a handful of similiar issues reported in releases since our May 24 release, triggered a review of our minimum system requirements - to ensure these were set sufficiently, and to understand the cause.

Between the May 24 and August 24 releases, we have introduced several changes that have individually contributed to the overall degradation we observed, however no one specific change was the cause;

  • Changes to frameworks / libraries 

    • Java

    • Tomcat

    • Grails

    • Angular 

  • Removal of some server level caching - to make provision for horizontally scaling deployments 

  • Redis for maintaining session across application nodes 

  • Multi-lingual additions 

  • New Category Object (filter) features 

  • Organisations

  • Linked Data Connections 

  • Custom Fields logic 

  • Two performance related code changes have also been introduced in this September 24 release

    • Improved handling of Encrypted Variables 

    • Application Optimisation - to reduce requests when refreshing the page 

 

Analysis & Testing

The chart below logs the output produced by the dashboard (MIS_AUDITS) and shows the number of chart requests served by the dashboard per minute.  We use other metrics – but this served rate has proven to be highly correlated to performance and the user experience when interacting with the system.  The vast majority of requests are serving charts.  We tested the ability to perform configuration actions at the same time. 

 

image-20241014-072615.png

 

The Green band shows the high load areas we observe on customers dashboards that run on our Pi SaaS instances. 

The Pink line shows the point of instability we noticed during our internal pre-release testing for the September 24 release - that caused nodes to be impacted.  We were able to observe this consistently, and this was before we made subsequent code changes to address the issues, and were based on a node with 1vCPU and 2GB RAM (these being our previous recommended settings for a single application node). 

 

To observe the impact of changing resources, we performed multiple test runs, making amendments to

  • vCPU’s

  • RAM

  • maxThreads (environment variable)

 

The table below details the finding we recorded for each run and configuration change.

Run 

vCPU 

RAM (GB) 

Max Threads 

Observed 

20 

System hit a limit at 170 charts served per minute.  The server remained stable at this point, and when load was reduced continued to operate efficiently. 

30 

The system performed well and was able to perform to ~350 chart requests. The server remained stable 

40 

The system performed well and was able to perform to ~400 chart requests. The server remained stable 

 

 

50 

The system was still able to perform to ~350 chart requests per minute, but the observed performance was significantly reduced. 

40 

The system was significantly slower even whilst no load was provided.  We struggled to load the system beyond ~120 charts per minute.  The server did remain stable. 

40 

The system was able to perform ~470 charts requests per minute, was stable and user experience was good even while loaded.  It was observed that the additional 4GB of RAM was not used by the system. 

40 

Following test 6, we re-performed test 5 to confirm that the additional 4GB of RAM was unnecessary, and the experience on this test 5 and 6 was comparable. 

2 load balanced * 2vCPU 

4GB 

40 

Where able to show that the resources where effectively run over load balanced nodes, and achieved ~1,000 charts per minute, and the solution was very stable even under this load. 

As Above 

 

 

Confirmed that the system could be loaded and unloaded without effect. 

 

Results & Recommendations 

Based on our findings, we recommend that the following resource and configuration changes are applied to ensure optimal performance and stability of application nodes.

 

Resource Requirement

  • 2vCPU 

  • 4GB RAM 

Configuration Changes

Tomcat MaxThreads should be set to the same value as the repository MaxConnections (our recommended value is 40)

  • PI_TOMCAT_MAX_THREADS = 40

  • PI_TOMCAT_MAX_CONNECTIONS = 40

Also, with the increase in memory to 4GB RAM, you should make sure 3GB is allocated to the Tomcat, leaving 1GB for the operating system.

  • PI_TOMCAT_MAX_MEMORY = 3072