First of a two-part series
Systems security is a key concern for data center managers all over the globe. As the product manager for the server management portfolio at Vertiv, I meet with customers and industry leaders to clearly understand how to address these concerns in a streamlined and cost effective manner. I have invited one of these industry leaders to provide his perspectives on the IPMI protocol and considerations for leveraging this protocol in a secure manner.
In this first blog, security researcher HD Moore will discuss the IPMI protocol and security concerns with base board management controllers (BMC). HD is best known as the founder of Metasploit, the foremost open source exploit development platform. He has spent the last 20 years auditing software, writing exploits, building products, and helping organizations secure their critical infrastructure. In his current venture, Special Circumstances, LLC, HD continues his mission to help organizations succeed through business advisory services, software development, security research, and penetration testing.
By: HD Moore, Special Circumstances, LLC
Much has been written about the security of baseboard management controllers (BMCs). You may know them as embedded service processors or by a product name such as iDRAC, iLO or IMM; these ubiquitous embedded controllers are designed to provide out-of-band access to server hardware. These devices implement the Intelligent Platform Management Interface (IPMI) protocol, a vendor-agnostic standard for monitoring and managing servers, even when they are powered off. BMCs are a mainstay of data centers, hosting providers, and difficult to reach sites across the world. Without remote access to these servers, problems can take much longer to resolve, but with remote access comes the risk of attack.
The vulnerabilities of Baseboard Management Controllers are well documented
Attacks against BMCs started to come to widespread attention in 2013; when J. Alex Halderman and team identified numerous implementation flaws in the Supermicro/ATEN BMC. These flaws would allow an attacker to compromise and obtain persistent access to the BMC. Subsequently, Dan Farmer, best known as the co-author of the original SATAN security scanner, authored a devastating analysis of the IPMI protocol, identifying vulnerabilities in the specification itself, that result in authentication bypass and password exposure. Since then, a number of other researchers, including myself, have discovered even more implementation-specific flaws across multiple BMCs, many of which allow unauthenticated access to the system. These vulnerabilities can be exploited using off-the-shelf tools and have become a staple for many hackers and security engineers alike.
A malicious attack could have disastrous consequences
Although awareness of BMC vulnerabilities has increased, the same could not be said for understanding of what is exposed after a successful compromise. BMCs offer a lot more than a power switch; popular products provide full KVM access to the server and support for virtual boot media. With a standard rescue disk, an attacker can gain full access to connected hard drives, networks, serial ports, and peripherals. As a result, a compromise of the BMC should always be considered a compromise of the server, and that is just the beginning.
In addition to being managed over the network, BMCs also expose a control channel to the server through an internal I2C bus. Through this bus, the server can issue unauthenticated IPMI commands and push firmware updates to the BMC itself. This means that any compromise of the server should also be considered a compromise of the BMC, as an attacker can add backdoor user accounts, change settings, and push modified firmware from within the server operating system. To make things worse, the process of updating the BMC through the management interface can be subverted by a malicious firmware image on the BMC itself, essentially faking out the server and convincing it that the update took place when it didn’t. Any compromise of a server that includes a BMC can result in the BMC becoming a permanently attached backdoor.
The co-dependency between the security of the BMC and the server OS is problematic for many reasons. Servers are often wiped and redeployed in completely different environments. Used servers are often purchased from auctions. Cloud services are both a consumer and producer of used servers, many of which include a BMC component. Refurbishment processes that reset the BIOS will have no effect on a BMC flashed with malicious firmware. For Supermicro motherboards public tools can be found for easily creating modified BMC firmware images. Even ignoring BMC backdoors, sensitive information can be stored in the non-volatile memory of the BMC, which would be exposed to the next user (or hacker) with access to this system. This problem is complicated enough that the US-CERT recommends physical destruction of server motherboards to avoid it.
There is a time for open doors; and a time to close them.
HD has given up a lot to think about in the above post but the news is not all bad. The open nature of the IPMI protocol makes it possible for each vendor to deliver a customized offering to complement their systems. There is great benefit in being able to access the core functions of a system, even when that system is off. However, that access should only be available to authorized users. There are many solutions that help manage and control access to embedded service processors; and the Avocent Universal Management Gateway (UMG) is designed from the ground up for this purpose.
The Avocent Universal Management Gateway is the first converged management appliance that helps data center managers take control of their multi-vendor, multi-platform environments. Using the Avocent Universal Management Gateway, embedded service processors can be configured into an out-of-band service processor network, effectively removing your service processors from public access. This is a first step in managing access to your infrastructure.
In this first post, HD Moore has provided a thorough analysis of the risks inherent in the design of the BMC. And I have given you a brief introduction into how you can begin to mitigate these risks with the Avocent Universal Management Gateway. In part two of this series, HD will describe some of the challenges in securing BMCs and highlight best practices that can be used to minimize these risks. I will build on HD’s guidance with an overview of how to use the Avocent Universal Management Gateway as a tool to increase security, control access and streamline management.
