What You Need to Know About Log4Shell, the Vulnerability Affecting Log4j

Unless you’ve been working in an air gapped system, the vulnerability affecting the log4j library, dubbed “Log4Shell” (CVE-2021-44228) has been impacting you just as it has been disrupting organizations all across the world. Initially discovered on Minecraft, this new critical vulnerability potentially affects 3 – 13 billion devices. It is also relatively easy to exploit when compared to other infamous bugs like Heartbleed.

Jake Kouns, Chief Innovation Officer at Flashpoint joined our Community Call: Log4J Vulnerability to explain what is known about Log4Shell, the potential impact, and how organizations can protect themselves against it.

What is Log4Shell?

Since its discovery in November, as an exploit used against Minecraft servers, the “Log4Shell” vulnerability has exploded across the Internet. The vulnerability is sending the security community into overdrive. The original CVE assignment for this issue has now been joined by two more. This is creating further confusion and headache for security teams.

There have been plenty of articles (some would say too many articles) on this topic. So, take your pick of any of the prior links if you want to get up to speed on the basics. We want to share some new information and perspective that you should be aware of when wrapping your head around Log4Shell.

It is important to call out that log4j is a popular logging framework in Java. This means it’s used in an extraordinary number of things. How many? Back in 2015, Oracle bragged that Java was running on 13 billion devices (even though we all know it’s 3 billion). Given the popularity of the library in question, Log4Shell might impact a large proportion of them, including IoT, ICS, medical, and more. We know it won’t impact all of them, but even a small percentage would mean a staggering number of devices. How prevalent is log4j? It’s used in the Mars 2020 Helicopter mission among other things.

CVE / NVD currently has log4j as the only impacted product in their list of CPEs. While the root vulnerability is in fact, in log4j, we have already determined that there are over 200 products affected and we’re still processing a long list of related advisories.

Products that are affected are from major vendors such as:

• Broadcom
• Cisco
• Elasticsearch
• F-secure
• Fedora
• HelpSystems
• HP
• IBM

• Informatica
• Microsoft
• National Security Agency (NSA)
• New Relic
• RedHat
• SonicWall
• Ubiquiti
• VMWare

There will be many more affected products still to be identified. And, of course, many more SaaS services will be impacted as well. It will take some vendors months to fully assess the impact of this vulnerability to their product portfolio.

Log4Shell: Vulnerability analysis

The vulnerability was given the nickname “Log4Shell” by LunaSec, because it is a remote code execution that is relatively simple to exploit in many services and products. If an attacker can generate an event that is logged, it can result in shell access to the system. We aren’t trying to split hairs here but “remote” actually depends on how the library is used. Just because a product uses log4j does not necessarily mean it is vulnerable remotely.

The message lookup feature is the real problem, so how it can be used actually depends on where the logged malicious input comes from. For some products it is remote, for others it will be local, and others still won’t be affected at all since they don’t log attacker-controlled input. Exploitation requires that an attacker can control the string being logged.

While the default assumption is that it can be triggered remotely until you know otherwise for a specific product or usage, the truth is that it all depends. If it is faster to upgrade the log4j library than perform a root-cause analysis, upgrade first.

“It is faster to upgrade the log4j library than perform a root-cause analysis, upgrade first.”

There have been a number of grim pronouncements from security vendors on just how nasty Log4Shell is:

• “The internet’s on fire right now,” – Adam Meyers, Senior Vice President of Intelligence, Crowdstrike [Source]
• “I’d be hard-pressed to think of a company that’s not at risk,” – Joe Sullivan, Chief Security Officer, Cloudflare [Source]
• “The Apache Log4j Remote Code Execution Vulnerability is the single biggest, most critical vulnerability of the last decade.” – Amit Yoran, CEO, Tenable [Source]
• “Worst bug impacting the Internet in the last 5 years, at least” – Matthew Prince, Co-founder & CEO, Cloudflare [Source]

Could Log4Shell be the “worst vulnerability ever”? At Flashpoint we avoid fear-mongering, so we will say “to be determined” for now, but the potential is certainly there, and organizations need to pay attention. While vulnerabilities in OpenSSL, such as Heartbleed, were prevalent, exploitation was more difficult on some servers. However, nearly a year later, CISA stated that Log4Shell was the most exploited vulnerability in 2021.

Log4Shell deserves to be taken seriously for the following reasons:

• Massive attack surface.
  • A huge number of products are impacted.
• Ease of exploitation.
  • Attackers were able to get remote code execution on Minecraft servers by simply pasting a short message into the chat box.
• Attention
  • This vulnerability is igniting discussion, in the press, social media and in the security community at large. Needless to say, our Social Risk Score is on fire!
  • When vulnerabilities get this kind of coverage, there’s a race to find further exploits. Swift action is required.

Variant

“Variant” might just be our word of the year at this point, with seemingly endless new COVID strains in the press, and, more cheerfully, for the Marvel fans, Loki variants… well now, yes, there is actually a Log4Shell variant to worry about.

Many people are referring to one CVE for Log4Shell, but there are actually three assignments as we go to press:

• CVE-2021-44228
• CVE-2021-4104
• CVE-2021-45046

The relationship between log4j2 (CVE-2021-44228) and log4j1 (CVE-2021-4104) could be confusing. There has been back and forth discussion over the past few days as to whether log4j 1.x is affected or not. As of December 13, RedHat assigned CVE-2021-4104 for a vulnerability under specific conditions. It requires considerable local access and will result in local privilege escalation.

That is important because a 1.x developer has stated:

• That it isn’t vulnerable
• But to be aware it is used in products “by an order of magnitude” despite 1.x being unsupported for years.

The developer was speaking about remote exploitation like CVE-2021-44228, where Red Hat’s CVE-2021-4104 is an entirely different attack vector. That means this has the potential to haunt us for a long time.

Add to that the CVE-2021-45046 assignment, which represents an ‘incomplete fix’ but is really the same vulnerability. MITRE and CVE Numbering Authorities (CNA) will assign a second CVE ID in cases of fixes not fully patching an issue. This helps some organizations in tracking an issue while introducing confusion to others.

Despite there being more than one CVE, many places have been treating them as a single issue, but this is definitely not the case. Based on our code review a few days ago and various discussion threads, we believe that 1.x is not affected as far as remote exploitation. We don’t believe that it includes the problematic message lookup functionality that was introduced in 2.0beta9 (or around that version). The vulnerability also only exists under very very limited circumstances i.e. requires JMS Appender class being in use (not often the case), JDNI support, and seemingly also config write access. Rob Fuller also just published a nice quick “Fact or Fiction” post on his LinkedIn page that helps understand the issue.

Recommendations

Yes, this appears to be an important issue and it needs to be taken seriously.

In order to help plan your approach here are some recommendations:

• Figure out where you may be affected
  • If you haven’t worked on implementing a Software Bill of Materials (SBOM) at your organization for your products, now would be a great time to get started!
  • Monitor for vendors/products that are impacted and assess your risk.
• Figure out the best solution/fix for your organization.
  • While the reported vulnerability was addressed in version 2.15.0, version 2.15.1 was released to disable JNDI access by default.
  • A more comprehensive fix has been released in version 2.16.0 where the problematic message lookups functionality has been removed.
  • Anyone not requiring JNDI access or message lookups are strongly encouraged to update to version 2.16.0 or later.
• As a workaround, pattern layout lookups within message text can be disabled by setting the “formatMsgNoLookups” option to “true”.
  • Note that this option is only available in version 2.10.0 and later.
• If running End of Life (EOL) 1.x version, it should not be used and it is recommended to upgrade to a current supported version.
  • Apache Log4j 1.2 reached end of life in August 2015
  • While it seems more difficult to exploit this version, be mindful there are other known vulnerabilities in this version as well.
• Expect a whole lot of vendors to be announcing patches very soon.
  • With so many products affected, start planning for a substantial patching effort in the short term for your organization.
  • You should also expect to see varying CVSSv2 and CVSSv3 scores.
    • The CVSS standard and scoring guidelines have some issues with this vulnerability.
    • Many organizations are rating Log4Shell a 10.0 for CVSSv3 and a 9.3 for CVSSv2.
    • If following the proper scoring standards, the scores should really be a 9.8 for CVSSv3 and 9.3 for CVSSv2.
    • As mentioned, depending on the implementation severity scores will vary, so expect to see inconsistency from your vendors.
    • In some implementations, exploitation may involve a “scope change”. That is a CVSSv3 designation take can take a remote code execution vulnerability from 9.8 to 10.0. In these cases, a log event may be generated on one system but passed to a second system (i.e. a logging host) where the code is executed. That is where the scope of the vulnerability impact changes.
• Expect lots of questions (and initial confusion!)
  • Whether it is from inside your company or outside (customers, vendors, partners), you should be prepared.
  • This is getting so much attention, you will be asked questions and you need to have answers ready.
• Expect answers!
  • You should also go get answers from your vendors as well.
  • Ask them a) if they have determined if they are impacted, and b) what they are proposing as solutions.

This vulnerability is unfortunately going to be around with us for years… remember Heartbleed? Even with all that press, years later servers were still vulnerable and not patched. This new log4j vulnerability will likely haunt us for years to come. As always, we will keep an eye on this vulnerability and continue to monitor the situation.

Got all that? You are just in time for your regularly scheduled Patch Tuesday, with many more vulnerabilities to assess and remediate!

How to remediate Log4Shell

Organizations have had a tough time with Log4j over the past week as multiple fixes have been published, with some of them having issues of their own. Here’s a quick recap:

• 2.15.0
  • This was the initial fix that was published.
• 2.15.1
  • Released in response to disable JNDI access by default.
• 2.16.0
  • A more comprehensive fix that removes the problematic message lookups functionality.
  • Customers not requiring JDNI access or message lookups are strongly encouraged to update to this or a newer version.
• 2.17.0
  • Released on December 17, this is currently the latest version with all fixes.
  • According to Apache, released because version 2.16 “does not always protect from infinite recursion in lookup evaluation”.
    • As such, this makes it vulnerable to CVE-2021-45105.
    • According to ZDNet, this issue was fixed in Log4j 2.17.0 and 2.12.3

If your organization hasn’t started or has just begun the process of updating Log4j, you need to start. If possible, you should update to 2.17.0 to ensure that systems aren’t susceptible to CVE-2021-45105.

In the span of less than a week, the number of affected products has increased from 200 to 900 and this number is likely to increase as our research teams process new advisories. Unfortunately, we are also seeing quite a bit of vulnerability disclosures from vendors that contain conflicting information and an estimated 34k GitHub repos that might need to be updated/fixed.

These will likely require different patches to correct and this will likely add a considerable amount of time to the remediation process. But other than having vendors clean up their advisories themselves, the best way for you to navigate through the confusion is to understand each Log4Shell variant and log4j related vulnerability.

Prioritizing Log4Shell

Having a grasp of what all the Log4j CVE Assignments are helps you keep a level head while you figure out which assets are affected. Since this is a library vulnerability, you need to ask yourself three things:

1. Am I using the Log4j library in my own code?
2. Are we using a vendor or library that includes Log4j?
3. Are we using a managed service that uses Log4j?

Finding those answers will help you take a risk-based approach to dealing with Log4j related issues and vulnerability management. Rather than wasting time and resources trying to find and fix every asset, you can instead focus on securing the ones that have the most impact and then move down in priority.

First mindmap – Am I vulnerable to Log4shell (v5.0, updated with higher impact for CVE-2021-45046) pic.twitter.com/ltBbtfnJ1n

— Loïc Castel (@Dick_Reverse) December 17, 2021

If you’re still having difficulty determining your organization’s attack surface in regards to Log4Shell, there are ways to find out where your organization might be vulnerable.

1. Software Bill of Materials (SBOM)

With Log4Shell being a library vulnerability, a Software Bill of Materials (SBOM) is extremely valuable since they list the various third-party libraries, open-source or commercial, that are used in software components. If your organization has been implementing this process, then you are better off compared to the many that haven’t.

For those that haven’t put SBOM into practice, this is actually a great time to start. Without this process in place, you will likely be spending a lot of time dissecting each of your deployed software anyways. So why not kill two birds with one stone and formally start the process? Rest assured this has not been, and will not be, the only critical library vulnerability you have to contend with.

2. Configuration Management Database (CMDB)

A CMDB can facilitate a risk-based approach since it helps you understand your assets by informing which software is being run on them and where they fit in your IT infrastructure. When paired with SBOM, this should provide you with most of the information you need.

But if you haven’t maintained your CMDB or don’t have one, you should use Log4Shell as an opportunity to start that process. During your research you will be capturing information like Affected Products, versions, and location, so take the time to consolidate this data into your new CMDB.

3. Scanning

Yes, we are aware that we recently published an article that details the deficiencies of vulnerability scanning. Scanning is a helpful tool and certainly has value, especially when you need to uncover unknown assets in your network. It just shouldn’t be the cornerstone of your vulnerability management program.

If you are one of the organizations that do not have SBOMs or a functional CMDB, you may want to consider this option. You will need to make sure that the scanner you choose has a signature written for every Log4Shell variant or at least make sure the scan doesn’t conflate every Log4j vulnerability as one issue. However, keep in mind that at the time of writing, CVE-2021-4125 is currently in RESERVED status. So this means that scanners sourcing from CVE/NVD won’t include this issue in their scans.

As long as you’re not treating scan reports as “end-all-be-all”, scanning should supplement your vulnerability management program. There are open source tools out there from vendors like JFrog and Arctic Wolf that can be useful in uncovering at-risk assets.

4. Risk scores

Once you surface your at-risk assets, risk scores are what actually enable you to prioritize and take a risk-based approach. An asset risk score is a numerical value that represents the asset’s overall importance to the organization based on use, type of data stored, likelihood of being exploited, and its exposure to vulnerabilities.

You can calculate it with this formula:

Asset Value x Threat Likelihood x Vulnerability Exposure

Future Log4Shell vulnerabilities

Implementing a risk-based approach is essential since it is likely that Log4Shell will not go away soon. Like we mentioned before, there is a lot of conflicting information out there in the form of vendor disclosures and GitHub repos, so there will be even more confusion as those resources are updated and fixed. Trust us, some vendor advisories are extremely confusing, some have contradictory information, and ten days later some are still “pending” more information as to the status of their products.

However, the thing that organizations should prepare for is the increased attention from vulnerability researchers. It is likely that the issues found so far only scratches the surface.

Security professionals may be shocked to know that the Log4Shell vulnerability was introduced to the Log4j codebase in July 2013 as part of the implementation of LOG4J2-313. CVE-2021-44228 was disclosed on December 10, 2021. Doing math, this means that these issues have gone unnoticed for over eight years. With as many as three billion devices affected, does this mean that malicious attackers could have been using Log4Shell this whole time? It is very possible.

To make matters worse, Log4j is an open source library and therefore, anyone could have found it. So why didn’t anyone report it? We’ve done a lot of research in this space (we’ve even published our own research too), and this situation is quite typical. This is yet another example disproving the notion that “With a thousand eyes, all defects are shallow“. Quite simply, thousands of eyes aren’t auditing these open source projects.

Vulnerability researchers are people, and like for everyone else, incentives play a large role in human behavior. Anything considered interesting will naturally get more attention; that is why well-known vendors and products tend to have “more vulnerabilities”. Once you also factor in bug bounties and other incentives, it makes sense that researchers may not spend a lot of time vetting an open source library.

But now that Log4j is getting a lot of attention, expect this to change. Researchers now have a plethora of reasons to dive deep into the Log4j code so don’t be surprised if even more vulnerabilities are found.

Preparing for what comes after Log4Shell

Even though Log4Shell is in contention for being the “worst vulnerability of all time”, it doesn’t mean that every other issue being disclosed is a pushover. New vulnerabilities are disclosed every day, and there are likely many that will require your attention. So what else has been going on?

Well, 543 vulnerabilities have been disclosed since Log4Shell’s publishing on December 10. Burdened with glorious purpose, Microsoft released a Patch Tuesday where 31 vulnerabilities had a CVSSv2 score of 10.0, and also included CVE-2021-43890, a Windows 10 vulnerability that was already being exploited in the wild. The following is a breakdown of the latest Patch Tuesday:

• 71 in Microsoft products
• 60 in Adobe products
• 60 in Apple products
• 48 in Siemens products
• 20 in IBM products
• 16 in SAP products
• 12 in Schneider Electric products
• Etc etc etc

• 194 (32.9%) had a remote attack vector
• 86 (15.8%) have public exploit information available
• 365 (61.8%) have an integrity impact
• 127 (22.8%) have a confidentiality impact

The point of this isn’t to downplay Log4Shell, but it is to show the volatility of the vulnerability disclosure landscape. It is vital that security teams don’t get lost in Log4Shell and are at least aware of new major issues being disclosed.

Implementing a risk-based Vulnerability Management program enables organizations to proactively address issues, instead of being forced to react to them. But in order to accomplish this, security teams will need comprehensive, detailed, and timely vulnerability intelligence that is contextualized to their environment.

How “big” is Log4Shell?

By now, the entire world is hopefully aware of what Log4Shell is, and why it’s a major problem. Since its discovery at the end of November last year, the news has been dominated by headlines touting its impact and how organizations need to pay attention, stop, and remediate Log4j issues. While more and more articles are published, each of them seems to be asking the same question, but they all seem unable to give a clear answer: Just how “big” is Log4Shell?

Log4Shell is a mega-vulnerability

If you are not familiar with what we do at Flashpoint, or what VulnDB® is, we can start with the fact that it is the most comprehensive, detailed, and timely source of vulnerability intelligence available on the market. That means the data that we provide our customers is independently aggregated, containing vulnerabilities found in IT, OT, IoT, CoTS, and third party libraries and dependencies.

VulnDB details over 330,000 vulnerabilities, including over 100,000 that CVE/NVD fails to report. This context helps us fully explain and contextualize Log4Shell’s size and impact, since it has become what we call a “mega-vulnerability.”

This term describes entries that take up significant bandwidth in our database, and notable “mega-vuln” entries include Spectre, POODLE, and Heartbleed. At first glance, it may seem that there’s a correlation between size and infamy, but the amount of attention a vulnerability receives is not the defining factor. Rather, to qualify as a mega-vulnerability, entries need to have hundreds or even thousands of vulnerability references while affecting a tremendous amount of products and vendors.

The accompanying charts illustrate how Log4Shell compares among its peer group:

Log4Shell’s unprecedented growth

Most mega-vulnerabilities take years to accumulate references and affected vendors/product information. But in just a month, Log4Shell has surpassed every other mega-vulnerability, except for one. At this of publishing, there are over 1,850 vulnerability references specifically citing Log4Shell and its variants, and they affect over 6,200 vendors/product combinations. Of those, over 275 are unique vendors and 1,677 unique products, meaning that some organizations will likely be impacted several times over.

In terms of affected vendors and products, Log4Shell falls slightly behind POODLE. However, if Log4Shell-related vendor advisories continue at their current pace, it will likely surpass POODLE within the next month. Nevertheless, the main highlight is the incredible amount of Log4Shell references circulating on the web.

Out of all 330,000 known vulnerabilities, Log4Shell has the most references by a wide margin. This means that there is an incredible amount of existing information out there. But if that’s the case, why are organizations seemingly struggling to mitigate and remediate affected assets?

Vulnerability references provide a clearer picture of risk

Despite the plethora of available information, organizations typically don’t have a tool that aggregates every known reference in one place. Vulnerability references provide much needed context for an issue, making it essential for intelligence vendors to constantly update their entries (especially Log4Shell) with as many of them as possible.

Even though they should, most vulnerability intelligence vendors do not do this. References can contain new information that could drastically affect the impact, severity, or solution availability for a given vulnerability. It’s common to see a vulnerability disclosed that does not initially have every detail needed to properly triage it. Key details like location, exploitability, or solution information are often found after the vulnerability’s initial disclosure, and they can come from third-party sources that are unrelated to the originating vendor.

As such, vulnerability references play a critical role in contextualizing the risk that an issue poses to organizations. By taking advantage of a feed that captures them and includes them with the actual vulnerability entry, organizations will be better informed if they need additional details specific to a vendor or product. Without references, security teams may question the provenance of the information in the vulnerability database. And when it comes to Log4Shell, security professionals don’t have the resources to scour the internet for every mention.

But depending on where you get your vulnerability data, hours spent researching Log4j issues will only scratch the surface. This is especially true if using publicly sourced data from CVE/NVD.

Although CVE disclaims that its list of vulnerability references should not be considered complete, in the end, what is provided simply doesn’t benefit the end user. On the other hand, VulnDB catalogs over 1,850 Log4Shell references compared to MITRE’s list of around 100, meaning that CVE/NVD users will be forced to find as much as 95% of the information that is actually available themselves.

The importance of standardized data

Some skeptics may argue that providing every reference could actually just overload security teams instead of providing a benefit. And in the age of overwhelming amounts of data, this concern has merit. If we were to dump every known reference on organizations without validating their value, the sheer amount of data may render the information unserviceable. But what if every vulnerability reference was sorted based on their contents and relevancy, with options to consume based on affected CPEs?

This is the importance of an independently researched, comprehensive, and detailed vulnerability database. By being proactive in searching and capturing published details, we are able to provide organizations with a more complete picture, allowing security professionals to make informed risk-based decisions. Sign up for a free trial today.