2022 04 05 - Licence Manager/piET - System Outage

Date of Incident:Apr 5, 2022

Outage Duration 10 Minutes

Date of Resolution Apr 5, 2022

Description

Piet in production and test were down. When going to the website, a Cloudfront error would display and nothing on the site could be accessed. It was noticed immediately and down for less than ten minutes.

How did we detect the issue?

The issue was detected when Leah went onto customers.panintelligence.cloud following a successful merge of her changes into the master branch of the Piet project, which is also the production branch. The pipelines passed with no errors but an error displayed on the front end.

The error message explicitly stated that it was an error related to Cloudfront permissions - investigation was not needed to know what was causing the problem.

Impact

This could have had a big impact if not detected immediately. Piet is used internally and by our customers but due to quick detection, it was not reported by anyone.

How have we communicated the issue

Due to the quick detection and knowledge of how to quickly resolve the problem, customers/staff were not notified. However, perhaps this step should have been carried out even if it was back up in just a few minutes.

Resolution Steps

The issue was related to permissions on the new Cloudfront@edge lambda functions included in the merge. There were two plans to resolve this.

Plan A: After immediately showing Myles the problem, he went onto the AWS management console to delete the lambdas that were causing the problem. This was the quicker way to resolve the issue so they went ahead with this resolution.

Plan B: Leah quickly reverted her changes and created a new merge request into the production branch which would revert piet back to its original working state.

The permissions were then fixed in the test environment before pushing up to the production branch.

Retrospective

While the issue was detected almost immediately on Piet in production, customers.panintelligence-test.cloud had the error message but this was not checked before pushing up to master. Simply seeing the pipeline has passed is not enough to assume that the changes are working.

In addition, the first action should have been to inform staff/customers that the website is down, followed by another announcement when it had been fixed.