Apache Log4j Incident Management

This page from the N.C. Joint Cybersecurity Task Force contains the latest information and resources regarding the Apache Log4j vulnerability for North Carolina state agencies, local governments, academic institutions and private sector entities.

The JCTF wants to emphasize the criticality of this vulnerability and set expectations that this is not going to be a quick or easy effort. It can be expected that overall remediation activities will take several weeks or months.

Apache Log4j Vulnerable Vendors

  • View a list of vendors confirmed or suspected to be affected by the Log4j vulnerabilities.

Log4j Incident Management Survey

The Joint Cybersecurity Task Force has created a survey to help assess the state’s risk of exposure and potential risk of exploitation. In addition, the JCTF can offer free cybersecurity support services to local and state government and critical infrastructure partners in remediation efforts by request. Please take 5-10 minutes to complete the survey.

Complete Survey

Reporting Exposures or Compromises

To report a cybersecurity incident please use the Statewide Cybersecurity Incident Report Form. If your organization is exposed, requires immediate or additional assistance from the N.C. Joint Cybersecurity Task Force please contact the N.C. Emergency Management 24-Hour Watch Center, at NCEOC@ncdps.gov or at 1-800-858-0368. 

Note: All reported sensitive security information is protected under N.C.G.S. 132-6.1(c) and is not for public disclosure. 

Tab/Accordion Items

Apache Log4j Vulnerability Overview

The N.C. Joint Cybersecurity Task Force and its partners are responding to the widespread vulnerability and potential exploitation of a critical remote code execution vulnerability (CVE-2021-44228) in Apache’s Log4j software library, version 2.0-beta9 to 2.14.1, also known as “Log4Shell” and “Logjam.”

Log4j is broadly used in a variety of consumer and enterprise services. An unauthenticated remote actor could exploit this vulnerability to take control of an affected system.  

Example of Log4j Vulnerability Attack Chain
View a larger version of this image.

 

(Updated December 29, 2021) Organizations are urged to upgrade to Log4j 2.17.1 (Java 8), 2.12.4 (Java 7) and 2.3.2 (Java 6). 

See CISA's joint Alert AA21-356A: Mitigating Log4Shell and Other Log4j-Related Vulnerabilities for more information. 

NOTE: Affected organizations that have already upgraded to Log4j 2.15.0, 2.16.0, and 2.17.0 will need to upgrade to Log4j 2.17.1 to be protected against CVE-2021-44228, CVE-2021-45046, and CVE-2021-45105.

(Updated December 20, 2021) As of December 17, 2021, Apache released Log4j 2.17.0 for Java 8 users to address a denial-of-service (DOS) vulnerability—CVE-2021-45105.

On Dec. 10, 2021, Apache did release a security update,  Log4j 2.15.0, to address the CVE-2021-44228 vulnerability. 

It was found that the Log4j 2.15.0 security update was incomplete in certain non-default configurations.

As of Dec. 13, 2021 Apache released Log4j version 2.16.0 in a security update to address the vulnerability discovered in the 2.15.0 security update, CVE-2021-45046. A remote attacker can exploit this second Log4j vulnerability to cause a denial-of-service (DOS) condition in certain non-default configurations.

What We Know

  • Log4j affected versions  >=2.0 and <=2.17.0
  • While Log4j 1.x versions are not affected by this vulnerability they are end of life and have other known vulnerabilities such as CVE-2019-17571 and CVE-2021-4104. All organizations are urged to upgrade to the latest Log4j version or discontinue use. 
  • Active exploitation of this vulnerability is occurring (none identified in North Carolina).
  • The exploit is evolving to overcome the protective measures having been or in the process of being operationalized.
  • The vulnerability impacts the server, which interprets the log message generated by, in most cases, simple HTTP. connections and not necessarily the public-facing server which received the initial connection request.
  • No major disruptive cyber incidents attributed to this vulnerability have been made public.

What We're Doing

  • Continue to work with our local, state, federal, and other critical infrastructure partners to alert and provide updated remediation guidance as the situation changes to include agency security liaisons.
  • Scanned our approved internal environments using Tenable.sc to identify areas of impact to this vulnerability.
  • Applied recommended remediations to the NCDIT infrastructure (includes cloud environment), as required.
  • Provided out-of-cycle vulnerability scans and remediation guidance to agencies supported by NCDIT’s Enterprise Vulnerability Management Team.
  • Providing incident response and recovery resources to qualified impacted organizations, as requested.
  • Continuing to provide relevant and updated information, as provided.

What We All Must Continue to Do

  • Consult the Blue Team CheatSheet *Log4Shell*, as well as the Apache Log4j Vulnerable Vendors webpage for lists of affected applications and recommendations:  
    • If your application is not listed in the aforementioned lists, run the “Log4Shell Vulnerability tester provided by Huntress” to test whether your applications are vulnerable to Log4j  (https://log4shell.huntress.com).
  • Upgrade to latest Apache Log4j version (2.16.0).
  • Follow remediation guidance from 3rd-party vendors whose applications/platforms/tools are operational in your respective environments.
  • Actively engage your cloud service providers to verify they are remediating this vulnerability in their environments.
  • All organizations are strongly encouraged to report your findings to DIT.ThreatManagement@nc.gov.
  • All state agencies must report your findings along with remediation plans by 3 p.m. on Dec. 17, 2021 to DIT.ThreatManagement@nc.gov.

The JCTF wants to emphasize the criticality of this vulnerability and set expectations that this is not going to be a quick or easy effort. It can be expected that overall remediation activities will take several weeks or months.

Mitigation Guidance from JCTF & Partners

  1. Immediately identify services and systems within your organization that are vulnerable AND exploitable to the Log4j vulnerability.
  2. Take immediate action to mitigate and prevent exploitation of identified services and systems. The image below provides examples of mitigation techniques. 
    The Log4j JDNI Attack & How to Prevent It Infographic
    View a larger version of this image.
  3. Act to securely patch all services and systems prioritized by exploitability AND asset/service criticality within your organization. Note: It is recommended all versions of Log4j vulnerable assets be upgraded to the most current patch available. 
  4. Complete secure patching of all vulnerable systems and services.
  5. Actively threat hunt to identify any indicators of compromise.
  6. Report any signs of compromise to the N.C. Joint Cybersecurity Task Force and to the N.C. Department of Information Technology, as required.
  7. Continually reference updated information and guidance on mitigating, managing, and remediating Log4j vulnerabilities to include seeking guidance from affected vendor and third-party assets and services within your infrastructure. 

Patched All Log4j vulnerabilities have been patched to the most updated security update (2.16.0 as of Dec. 13, 2021).
Exposed Systems and services using Log4j have not been patched and is vulnerable but, no IOCs have been identified on the network at this time.
Compromised IOCs associated with the vulnerability have been observed in the network.
Threat Hunting In the process of searching the network and IT infrastructure to determine if a compromise has occurred. 
Remediated Log4j vulnerabilities have been patched, and threat hunting has completed. If compromise was identified, the system has since been cleared.

Indicators of Compromise

Disclaimer: Due to the evolving nature of this threat, these lists are not comprehensive and are subject to change.

Below is a list of reported indicators of compromise. It is recommended that detection of any of these IOCs should be reviewed by your organization's security team. If additional assistance is needed please contact the North Carolina Joint Cybersecurity Task Force at jctf@nc.gov

Log4j updated IOCs and payloads as of 20DEC2021 (PDF)

Talos Intelligence Log4j Security Blog IOCs (webpage - scroll to bottom and select relevant links under IOC section)

Detecting anomalous network traffic resulting from a successful Log4j attack (IronNet Blog, webpage)