/
Vulnerability Update - Log4J2 / Log4Shell

Vulnerability Update - Log4J2 / Log4Shell

Owned by Adam Voakes
Last updated: Jan 04, 2022
7 min read

Background

Apache have recently disclosed critical level vulnerability in Log4J2 known as Log4Shell. If you hadn’t heard of Apache Log4J2, it is a logging library. Log4J2 is used by a very large percentage of Java programs developed in the last decade for both server and client applications. Java is also one of the top programming languages used by businesses, and something Panintelligence utilise in our dashboard. As soon as Panintelligence became aware of these vulnerabilities, we have put plans in place to upgrade our Log4J2 libraries to the newest versions… which have changed multiple times in a short period of time to address;

 

  • CVE-2021-44228 - Apache library fix provided version 2.15.0 - Included in Dashboard Release 2021-11-25.1 - Released Dec 14, 2021

  • CVE-2021-45046 - Apache library fix provided version 2.16.0 - Included in Dashboard Release 2021-11-25.2 - Released Dec 16, 2021

  • CVE-2021-45105 (NEW) - Apache library fix provided version 2.17.0 - Included in Dashboard Release 2021-11-25.4 - Released Dec 20, 2021*

*the ommisions of .3 patch release is deliberate

 

IMPORTANT - If you are using a dashboard version that pre dates our May 2021 release, you do not need to upgrade to the latest patch version to satisfy the vulnerabilities, as Log4J2 is not utilised until our May 2021 version onwards. However you should look for and delete drill-jdbc-all-1.16.0.jar in C:\Program Files\dashboard\Dashboard\tomcat\jdbc_drivers.

We would still recommend you upgrade to take advantage of other less critical security fixes, and all the great functionality that we have added to the product since May 2021… however it is not imperative to do so, to overcome the vulnerabilities listed above if you are on a version before May 2021.

Before performing an upgrade to our latest November patch release, its imperative that you first read through the Release Notes and our Upgrade guides fully, especially if you have not performed one recently. It is worthwhile reviewing all the release notes for each of the releases inbetween your current version and the one you are planning to upgrading too. This helps ensure you are fully aware of the system changes and any additional configuration change, or changes in procedure you may need to make during the upgrade process.

Events

To help keep customers informed of our latest responses to the Log4J2 issue, we have created an events table to detail the key steps we have/are taking;

Date

Event

Date

Event

Dec 13, 2021

We became aware of a critical level vulnerability in Log4J2 known as Log4Shell (CVE-2021-44228) from Apache, and started to investigate and develop a patch to overcome this on top of our original November 21 release

Dec 14, 2021

RELEASE - We issued a patch release (2021-11-25.1) to our original November 2021 release to overcome the CVE-2021-44228 vulnerability, which can be downloaded from our Customer Portal site for Windows and Linux installers, or through DockerHub for those using that approach. Our AMIs through the AWS Marketplace have been updated too.

Dec 15, 2021

Unfortunately, Apache have found a second vulnerability (CVE-2021-45046) involving Log4J2. Their initial fix for Log4Shell, version 2.15.0 was added to our 2021-11-25.1 release, however they have advised that this did not address all issues in non-default configurations and could be abused for DoS- denial of service attacks through malicious input. The Apache Software Foundation (ASF) have pushed out a new fix for Log4J2, 2.16.0.

We have received this fix and are actively working on providing a 2nd patch to our November 21 release to mitigate against this. After successful testing, the new patch was released by close of play Dec 16, 2021. This has been uploaded to our Customer Portal, DockerHub and AWS Marketplace areas for download.

IMPORTANT - In addition to the new (CVE-2021-45046) vulnerability, the mitigation advice provided by Apache has also changed. Please revisit the Guidance section below if you have already applied mitigations rather than upgrading. We have left the old guidance on the page for transparency however it has been crossed through.

Dec 16, 2021

RELEASE - Today we released a 2nd patch (2021-11-25.2) to our original November 21 release, to address the latest Apache vulnerability (CVE-2021-45046). This can be downloaded from either our Customer Portal site for Windows and Linux installers, or through DockerHub for those using that approach. Our AMIs obtained from the AWS Marketplace have been updated too.

As mentioned in yesterdays update, Apaches mitigation advice has changed, therefore please ensure you review the changes described in the Guidance section below, and take any nessisary actions needed. If you have any active instances of the dashboard running, but they are not actively in use (maybe for trials or where licences have expired), we always recommend uninstall these when they are no longer needed.

We are not currently aware of any compromises or malicious activity involving our products or services.

Dec 16, 2021

We have established that releases prior to May 2021 do not contain the vulnerable libraries so you would not need to follow the mitigation advice. However we would still recommend you upgrade to take advantage of other less critical security fixes.

IMPORTANT - Before performing any upgrades to our November patched release, its imperative that you first read through the Release Notes fully, and that of other releases inbetween your current version and the one you are upgrading too. This helps ensure you are fully aware of the system changes and any additional configuration change you may need to make during the upgrade process.

Dec 18, 2021

Unfortunately Apache have issued a further library update (2.17.0) for the Log4J2 library, as a further vulnerability CVE-2021-45105 has been found with version 2.16.0. The latest issue relates to denial of service not remote acccess as was the case with the previous vulnerabilities. We included version 2.16.0 in our 2nd patched release only last week (2021-11-25.2) on Dec 16, 2021.

We are currently working on producing another patched release (2021_11_25.4) to incorporate the new 2.17.0 library, and subject to successful testing we expect to issues this by CoB on Monday Dec 20, 2021

Dec 20, 2021

RELEASE - Today we released another patch (2021-11-25.4) to our original November 21 release - this addresses Apache vulnerability (CVE-2021-45105). This can be downloaded from either our Customer Portal site for Windows and Linux installers, or through DockerHub for those using that approach. Our AMI's which are available on the AWS Marketplace, have been updated too.

IMPORTANT - As a precaution, we have removed the Apache Drill and Firebolt JDBC drivers from Tomcat until we have hear back from them to clarify if they carry the Log4J2 vulnerabilities or not. If you are currently using these, after upgrading you will need to add them yourself under tomcat/custom_jdbc_drivers and ensure their safety.

If you happened to locate and upgrade to a 2021-11-25.3 release today (which was intended for internal testing only), we recommend upgrading 2021-11-25.4 to ensure you are aligned with the very latest release.

For those who have previously upgraded to either our original November release, or the patched .1, .2 or .3 versions, we highly recommend that you upgrade to the .4 version at your earliest opportunity.

Jan 4, 2022

A further vulnerability (CVE-2021-44832) has been reported which relates to a very specific type of configuration which we do not use in the dashboard (JDBC Appender with a JNDI LDAP data source URI). Our January 22 release will contain the very latest Log4J 2 library but we will not be issuing this version in an emergency patch before then because it is not exploitable in our configuration. It is still possible to manually update the libraries in the manner described previously if you are especially concerned, but we do not specifically recommend this.

 

Mitigation Guidance

Dec 15, 2021 Apache have informed the community that their initial mitigation advice does not fully remedy the issue so we have updated our guidance accordingly:

We advise if you are unable to upgrade to our latest release you should take the following action:

  • Stop the application (use the services tool and stop the tomcat service)

  • Within the filesystem running the application locate any instances of log4j-*-2*.jar, these will typically be located in 3 locations:

    • tomcat/webapps/pi/WEB-INF/lib

    • tomcat/webapps/panMISDashboardResources/WEB-INF/lib

    • tomcat/webapps/panLicenceManager/WEB-INF/lib

  • Delete these jar files and replace like-for-like with the Log4J 2.17 files from https://customers.panintelligence.cloud/downloads

  • Restart the application

  • You should not see any change in behaviour but rest assured if log4j-core-2*.jar does not predate 2.17.0 you are fully protected.

 

Dec 20, 2021 - If you are using a dashboard version that pre dates our May 2021 release, you do not need to upgrade to the patched version for security reasons, as Log4J2 is not utilised until the May 2021 version onwards. However, you should look for and delete drill-jdbc-all-1.16.0.jar in C:\Program Files\dashboard\Dashboard\tomcat\jdbc_drivers.

 

Previous Apache community advice (based on available information at the time - Dec 14, 2021 )

To help assure your systems remain secure, we recommend that you upgrade to the latest versions of the dashboard, following the standard upgrade guidelines. In the event that you are unable to take the new release, there are alternative ways to mitigate the issue, such as:

 

We are currently reacting to the changes being shared by Apache, and updating our documentation as things evolve, however our initial guidance around the mitigations are as follows;

 

Windows

Linux

Windows

Linux

  • September 2021, October 2021 and November 2021 (initial release):

    • Edit the file Dashboard/service/install_jvm_service.bat and add

    • ;-Dlog4j2.formatMsgNoLookups=true

    • to the end of the –JvmOptions line (inside the speech marks)

    • The configuration tool will need to be re-run to remove and reinstall the service.

  •  Edit the file Dashboard/tomcat/bin/setenv.sh and add

  • (space)-Dlog4j2.formatMsgNoLookups=true

  • to the end of the export JAVA_OPTS line (inside the speech marks).

  • When Tomcat is restarted this will take effect.

  • Prior to September 2021 release:

    •  Edit the file Dashboard/tomcat/bin/service.bat and add

    •  ;-Dlog4j2.formatMsgNoLookups=true

    • to the end of the –JvmOptions line (inside the speech marks)

    • The configuration tool will need to be re-run to remove and reinstall the service.

 

 

Known Issue - 2021-11-25.1

The patch release 2021-11-25.1 includes new JAVA libraries for Log4J2, however when you perform a Windows upgrade the old libraries may still be present. To avoid them being flagged in a security scan, you can manually remove these;

C:\Program Files\Installation_location\Dashboard\tomcat\webapps\panLicenceManager\WEB-INF\lib\log4j-1.2-api-2.14.1.jar
C:\Program Files\Installation_location\Dashboard\tomcat\webapps\panLicenceManager\WEB-INF\lib\log4j-api-2.14.1.jar
C:\Program Files\Installation_location\Dashboard\tomcat\webapps\panLicenceManager\WEB-INF\lib\log4j-core-2.14.1.jar
C:\Program Files\Installation_location\Dashboard\tomcat\webapps\panMISDashboardResources\WEB-INF\lib\log4j-1.2-api-2.14.1.jar
C:\Program Files\Installation_location\Dashboard\tomcat\webapps\panMISDashboardResources\WEB-INF\lib\log4j-api-2.14.1.jar
C:\Program Files\Installation_location\Dashboard\tomcat\webapps\panMISDashboardResources\WEB-INF\lib\log4j-core-2.14.1.jar

 

If you are upgrading using the 2021-11-25.2 release (issued Dec 16, 2021), the above issue has now been addressed as part of the release and therefore no action is needed - the above only applied to those who have installed the 2021-11-25.1 release

 

We strongly recommend all customers upgrade to the latest patched version, or apply the advised mitigations described above as soon as possible.

Should you have any questions please do not hesitate to contact us

Thank you and kind regards

 

Charly