Averting Digital Apocalypse: The Backdoor Threat That Shook the World

The tech world was recently shaken by the revelation of a security breach within the open source software program XZ Utils. German software developer Andres Freund, working for Microsoft in San Francisco, stumbled upon suspicious activity during routine performance tests, uncovering a deliberate sabotage that could have had catastrophic consequences for internet servers worldwide.

Discovery of the Sabotage

Freund's keen observation skills averted a potential disaster, as he detected unauthorized changes in the latest version of XZ Utils before its widespread deployment. This incident underscored the critical role of individuals like Freund in safeguarding digital systems from malicious intent.

The backdoor within XZ Utils was discovered by Andres Freund, when he noticed unusual behavior associated with the sshd process.

Specifically, he observed that sshd was consuming an abnormally high amount of CPU resources during the login process. This unusual activity led him to further investigate the issue using the 'Valgrind' tool, which is used for memory debugging and profiling. During his investigation, he encountered numerous errors that prompted him to delve deeper into the problem.

The discovery process involved examining the XZ build process, where Freund identified the execution of the Build-to-Host.m4 script. This script contained a line that inserted an obfuscated script towards the end of the configure script, which is responsible for creating MakeFiles for xz-utils and liblzma. The obfuscated script, upon execution, performed checks for certain conditions, such as the operating system being x86-64 Linux and whether the build process was part of a Debian or RPM package build. The script aimed to modify the MakeFile of liblzma in a way that interfered with its symbol resolution process at runtime, effectively redirecting a specific symbol to point to malicious backdoor code.

This manipulation was particularly impactful during the public key authentication process of sshd, where the redirected symbol RSA_public_decrypt@....pl would trigger the execution of the attacker's code. This code was designed to extract and execute a payload from the public key used during authentication, effectively enabling remote code execution.

The requirements for exploitation, such as the need for the TERM environment variable to be unset, the path to the currently running binary to be /usr/sbin/sshd, and the absence of certain debugging environment variables, suggest that the attacker intended to complicate the analysis process for researchers and avoid detection

Focus on Open Source Software Security

The incident reignited discussions on the security of open source software, which forms the backbone of the internet economy. Volunteer-maintained projects like XZ Utils often operate with limited resources, making them susceptible to infiltration and exploitation by malicious actors.

It is an critical reminder of the vulnerabilities within the OSS ecosystem, particularly the risks associated with supply chain attacks. It highlights the need for vigilance, thorough vetting of contributions, and robust security practices in managing OSS projects. The OSS community, including projects, maintainers, and users, must continually work towards enhancing security measures, including more stringent code review processes, improved monitoring of contributions, and collaboration to quickly address vulnerabilities.

The Open Source Security Foundation (OpenSSF) and similar organizations play a crucial role in this regard, offering resources, training, and initiatives aimed at improving OSS security. Participation in community discussions, adherence to best practices, and proactive engagement in security exercises can help in preparing for and mitigating the impacts of future supply chain compromises.​

Background of XZ Utils

XZ Utils, a set of file compression tools integrated into Linux distributions, was primarily overseen by Lasse Collin.

XZ Utils is a set of free command-line tools for data compression, which includes xz for compression and decompression, similar to gzip and bzip2, and liblzma, a software library with an API akin to zlib. These tools are designed to handle single files or data streams, rather than bundling multiple files into a single archive. For creating archives, other programs like tar are used before compression with XZ Utils. This software is included in various operating systems, including Linux distributions and Windows, and is particularly useful for compressing software packages, kernel images, and other file formats due to its high compression ratio​

The project faced challenges as Collin struggled with personal issues, paving the way for the entry of a new developer, Jia Tan, into the project.

Infiltration by Jia Tan

Tan's gradual involvement in XZ Utils raised suspicions among cybersecurity experts, who later discovered the presence of a concealed backdoor in the software. Tan's actions highlighted the vulnerabilities inherent in projects heavily reliant on volunteer contributions.

The vulnerability identified as CVE-2024-3094 was discovered in XZ Utils versions 5.6.0 and 5.6.1, which introduced a backdoor into the software. This backdoor, which could be activated remotely, particularly affects systems connected via SSH, and could compromise system integrity and performance. The vulnerability was discovered by a Microsoft employee, and the impact of this security flaw is still under investigation. Due to the severity of the issue, with a CVSS score of 10 indicating the highest level of risk, organizations have been advised by the U.S. Cybersecurity and Infrastructure Security Agency (CISA) to downgrade to a previous, non-compromised version of XZ Utils​. (Source: "Microsoft FAQ and guidance for XZ Utils backdoor"/ Microsoft tech Community)

The technical intricacies of the vulnerability involve the insertion of an obfuscated script in the XZ build process, which modifies the MakeFile of liblzma to interfere with its symbol resolution process. This manipulation allows for the execution of malicious code during the public key authentication process of sshd, leading to remote code execution. The conditions for the backdoor's activation include the OS being x86-64 Linux and the XZ build process being part of a Debian or RPM package build. Additional runtime conditions further complicate the exploitation and analysis of the backdoor​. (Source: "Backdoor in XZ Utils allows RCE: everything you need to know" / Wiz)

The Backdoor Introduction

What is a Backdoor? Developer Sam James shared some details on Github.

The amazing feature of this backdoor attack was its masterful obfuscation.

Despite Collin's silence on the matter, investigations revealed Tan's deceptive practices in introducing the backdoor. Speculations regarding Tan's true identity and affiliations with sophisticated hacking groups added a layer of intrigue to the unfolding events.

“The care taken to hide the exploits in binary test files as well as the sheer time taken to gain a reputation in the open-source project to later exploit it are abnormally sophisticated,” said Molly, a system administrator at Electronic Frontier Foundation who goes by a mononym. “However, there isn’t any indication yet whether this was state sponsored, a hacking group, a rogue developer, or any combination of the above.” (Source: "Here's How Millions of Linux Computers Almost Got Hacked / Intercept)

Thomas Roccia, a Microsoft researcher, shared an illustrative post on Mastodon that effectively maps out the extensive and almost successful attempt to disseminate a backdoor. This effort, if successful, would have surpassed the scope of the SolarWinds incident that took place in 2020.

It was a meticulously planned attack.

Speculations on Jia Tan

The enigmatic persona of Jia Tan left many questioning the motives behind the security breach. The possibility of Tan being a pseudonym for a skilled hacker or a state-sponsored operative raised concerns about the sophistication of the attack on XZ Utils.

Who is Jia Tan? The Wired gives an interesting insight into how this person/group approached the hacking mod in XZ Util. Jia Tan first appeared in November 2021 with the GitHub username JiaT75 and took at least a year and a half before the backdoor mods were done. A battle of utmost patience.

Jia Tan exploited open source software’s crowdsourced approach to coding whereby anyone can suggest changes to a program on code repositories like GitHub, where the changes are reviewed by other coders before they’re integrated into the software. Peeling back Jia Tan’s documented history in the open source programming world reveals that they first appeared in November 2021 with the GitHub username JiaT75, then made contributions to other open source projects using the name Jia Tan, or sometimes Jia Cheong Tan, for more than a year before beginning to submit changes to XZ Utils. By January 2023, Jia Tan’s code was being integrated into XZ Utils. Over the next year, they would largely take control of the project from its original maintainer, Lasse Collin, a change driven in part by nagging emails sent to Collin by a handful users complaining about slow updates. (Whether those users were unwitting accomplices, or actually working with Jia Tan to persuade Collin to relinquish control, remains unclear. None of the users replied to requests for comment from WIRED.) Finally, Jia Tan added their stealthy backdoor to a version of XZ Utils in February of this year. (Source: "The Mystery of ‘Jia Tan,’ the XZ Backdoor Mastermind" / Wired)

People are saying that this could be a handle operated by extremely proficient state-sponsored hackers.

Andres Freund's Investigation

Freund's meticulous examination of XZ Utils exposed the hidden backdoor, preventing a potential cybersecurity crisis. His proactive approach and attention to detail exemplified the importance of continuous monitoring and scrutiny in software development processes.

Andres Freund's investigation into the XZ Utils backdoor began with the observation of anomalous behavior within his Debian sid environment. This led him to dive deeper into the issue, eventually uncovering a complex and obfuscated backdoor within the widely used command-line tool XZ Utils (liblzma), specifically in versions 5.6.0 and 5.6.1. (Source: Backdoored XZ Utils (CVE-2024-3094) / Rapid7)

The backdoor's build process involved multiple stages of obfuscation and execution, starting with changes to the liblzma library's build scripts. These changes were crafted to ensure that the malicious code would only activate under certain conditions, such as running on an x86-64 Linux system and being part of a Debian or RPM package build. This level of sophistication in the backdoor's design highlights the advanced methods used by attackers to conceal their actions and target specific systems.

Freund meticulously analyzed the backdoor's code, identifying the conditions under which it would modify the build process to inject the malicious code. His investigation revealed the use of standard Linux system tools in unusual ways to decode and execute the backdoor's payload, demonstrating the attackers' deep understanding of Linux systems and build processes​ (Source: "500ms to midnight: XZ / liblzma backdoor" / Elastic Security Labs)

Reactions and Responses

The discovery of the security breach prompted introspection within the open source community, highlighting the risks faced by volunteers dedicated to maintaining essential software infrastructure. The incident served as a wake-up call for the industry to prioritize security measures and collaborative efforts.

Microsoft was among the first to respond, providing detailed guidance and tools through their products such as Microsoft Defender Antivirus and Defender for Endpoint to help organizations assess and mitigate their exposure to the vulnerability. Microsoft's efforts focused on enhancing their products to identify affected software and providing queries for security teams to evaluate the impact within their environments​.

Akamai provided an in-depth analysis of the backdoor, highlighting its potential for remote code execution rather than just an SSH authentication bypass as initially thought. They detailed the backdoor's complexity and the long-term strategy employed by the threat actor, who infiltrated the project over two years, gradually gaining more control. Akamai recommended downgrading to an uncompromised release of XZ Utils as a mitigation strategy and offered additional measures to limit the attack's impact. (Source: "XZ Utils Backdoor — Everything You Need to Know, and What You Can Do" / Akamai)

The Open Source Security Foundation (OpenSSF) also played a crucial role in responding to the incident. They emphasized the collaborative effort between different Linux distributions and the open-source community in quickly addressing and containing the compromised packages. OpenSSF highlighted the importance of vigilance within the open-source software ecosystem and urged the community to verify their software versions, especially those using experimental or bleeding-edge distributions​. (Source: xz Backdoor CVE-2024-3094 / OSSF)

Challenges in Open Source Ecosystem

The incident shed light on the challenges faced by tech companies in supporting open source initiatives financially and operationally. Calls for a more equitable distribution of resources and responsibilities underscored the importance of fostering a secure and resilient open source ecosystem.

The open-source ecosystem faces several challenges that stem from its very nature and structure. One of the main challenges is the security and systemic risk associated with open-source software (OSS), as highlighted by incidents like Log4Shell. The open-source community, including foundations like OpenSSF and the Linux Foundation, have been actively working to address these challenges by informing the public about security imperatives, launching initiatives like the Alpha-Omega project to improve security posture, and collaborating with government and industry to secure the ecosystem. These efforts also include the development of the Open Source Software Security Mobilization Plan, aimed at securing OSS production, improving vulnerability discovery and remediation, and shortening ecosystem patching response times.

The discovery of the Log4Shell vulnerability in late 2021 catalyzed a concerted effort among open source software (OSS) stakeholders to address cybersecurity risks.

The Linux Foundation, alongside the Open Source Security Foundation (OpenSSF), played pivotal roles in this endeavor, focusing on the security of OSS. OpenSSF, through projects like Alpha-Omega, sought to bolster OSS security by leveraging expert knowledge and automated tools.The urgency of enhancing OSS supply chain security was echoed in discussions among industry and government leaders.

OpenSSF's contributions, including responses to the National Institute of Standards and Technology's (NIST) Request For Information and testimonies in congressional hearings, highlighted the necessity for comprehensive security strategies. The Open Source Software Security Summit II exemplified these collaborative efforts, uniting stakeholders to tackle OSS security challenges.

Research and reports from the Linux Foundation and LF Research brought to light the cybersecurity vulnerabilities within OSS, proposing methodologies to fortify security protocols. Collaborations, such as with Snyk, provided valuable insights into overcoming OSS cybersecurity hurdles.

The Census II report, in particular, offered a detailed examination of OSS utilization across various sectors. Initiatives aimed at addressing the cybersecurity talent gap, like the White House Cyber Workforce and Education Summit, sought to empower the OSS community with the necessary skills and knowledge.Policy discussions, including those at DEF CON 30, delved into the potential of policy interventions to strengthen OSS and overall software security. These initiatives underscored the importance of maintaining the open and collaborative nature of OSS while implementing effective security measures.

Lessons Learned

The fortunate detection of the backdoor in XZ Utils served as a reminder of the fragility of digital systems and the critical role of vigilant individuals like Freund. The incident underscored the necessity for proactive measures, collaboration, and ongoing vigilance to safeguard open source software from malicious threats.

Here are some detailed lessons that can be learned.

Importance of Vigilance in Open Source Maintenance

• Long-term Threats: The attack demonstrated that threat actors are willing to play the long game, contributing to projects over years to build trust before introducing malicious code
• Single Points of Failure: Many open-source projects rely on a small number of maintainers, which can become single points of failure if they are compromised or overburdened
• Community Support: The incident underscores the need for greater community support and resources for open-source maintainers, including potentially more maintainers to share the workload and reduce the risk of burnout or oversight

Sophistication of Supply Chain Attacks

• Stealth and Complexity: The backdoor was introduced in a sophisticated manner, using obfuscation and multi-stage execution chains to avoid detection, highlighting the need for more thorough code and release review processes
• Social Engineering: The attacker used social engineering, including the creation of fake accounts, to pressure the original maintainer into granting them maintainer rights

Detection and Response

• Accidental Discovery: The backdoor was discovered by chance, which indicates a need for more proactive and systematic security reviews in the software development lifecycle
• Rapid Response: Once detected, the response from the community and industry was swift, with advisories and mitigation steps quickly disseminated

Security Best Practices

• Regular Audits: Regular and automated security audits of codebases and dependencies are essential to catch vulnerabilities and backdoors
• Defense in Depth: Implementing a defense-in-depth strategy, including network segmentation and limiting external SSH access, can reduce the blast radius of a potential compromise
• Credential Management: Strong credential management and zero-trust architectures can help mitigate the risk of compromised components within a system

Policy and Governance

• Open Source Governance: There is a need for better governance and oversight in open-source projects, potentially through foundations or community-led initiatives
• Incentivization: The open-source ecosystem may need to explore new ways to incentivize maintainers, including financial support or dedicated time from organizations that benefit from open-source software

Awareness and Education

• Community Awareness: Raising awareness about the risks associated with open-source software is crucial for both developers and users
• Education and Training: Educating developers on secure coding practices and the importance of security in the software supply chain can help prevent similar incidents

Collaboration and Sharing

• Information Sharing: Sharing information about threats and vulnerabilities within the community can help prevent the spread of backdoors and other security issues
• Collective Defense: Collaborative efforts, such as sharing detection rules and threat intelligence, can enhance the collective defense against sophisticated attacks