Thousands of Internet-connected servers sold by the world’s biggest manufacturers can be remotely backdoored by exploiting critical vulnerabilities—some more than a decade old—that lurk deep inside system motherboards, according to research presented Wednesday.

Baseboard management controllers are miniature computers that are embedded into the motherboards of virtually every enterprise server. The microcontrollers, typically abbreviated as BMCs, run with their own operating system firmware, network stack, and IP address. Administrators rely on them to monitor the physical status of large fleets of servers and to perform a variety of tasks, including rebooting machines, installing updates, and even reinstalling operating systems. BMCs provide what’s known as “lights out” and “out-of-band” management because they work even when servers they’re attached to are turned off or are unresponsive.

A “pervasive, under-monitored, under-patched parallel attack surface”

Researchers have warned since at least 2013 that BMCs present a golden opportunity for hackers looking for ways to gain deep and persistent access to datacenters. The chief culprit was IPMI, the protocol that allows BMCs to operate independently of servers and to perform administrative tasks. Vulnerabilities in this firmware made it possible for attackers to remotely execute malicious code on the controllers and, from there, infect the servers they manage.

Research presented Wednesday at the Black Hat security conference in Las Vegas shows that little has changed since then. HD Moore, a firmware security expert and the CEO and founder of security firm runZero, uncovered more than a dozen new vulnerabilities in BMCs sold by HPE, Supermicro, Avocent, Huawei, Lenovo, Dell, and others. Moore has also found that some of the weaknesses he warned of in 2013 remain active, despite measures intended to fix them.

“The end result is a pervasive, under-monitored, under-patched parallel attack surface that is both Internet-exposed and widespread inside corporate networks, and is much more exploitable than many folks realize,” Moore wrote in an email ahead of his talk.

To highlight and quantify the threat, Moore oversaw two large-scale scans. One scanned Internet-connected BMCs at large, while the other internally surveyed the devices inside corporate networks. The external scan found more than 86,000 BMCs that exposed a management service to the public. More than 54 percent of those devices contained one or more critical vulnerabilities. As many as 75,000 of them remained vulnerable to CVE-2013-4786, a vulnerability in the IPMI 2.0 authentication protocol that enables off-line cracking of administrator-level BMC account passwords. The internal scan of 126,761 BMCs, meanwhile, found that nearly 29 percent of them had one or more critical vulnerabilities.

A vulnerability list that grows by the day

The number of new vulnerabilities Moore has discovered in the lead-up to his has grown by the day, making it hard to provide a specific number. Because the researcher is keeping vulnerability details confidential until the BMC makers have time to patch them, he’s also not at liberty to disclose many of them individually. Generally speaking, some of the bug classes are:

1. Flaws in the IPMI authentication handshake. Hackers can alter the prescribed sequence of massage exchanges in a way that bypasses authentication requirements. This gives an attacker a limited toehold into the BMC. The attacker can then gain administrative access by exploiting other vulnerabilities. Affected products include HPE iLO, Supermicro, OpenBMC, and OpenBMC-derived products from H3C and Nvidia.

2. A failure of IPMI to enforce integrity and encryption protections in-session. “The device decides whether to authenticate and decrypt each packet from that attacker’s own header, and not from the algorithms the session negotiated, so an unsigned, unencrypted command is accepted on a secured session,” Moore said. A proof-of-concept exploit Moore developed uses such bugs to “chain otherwise-unexploitable issues into full sessions.” Affected vendors include HPE, Supermicro, and Intel (legacy).

3. Predictable session identifiers. Session tokens are generated from counters or from a clock rather than secure random sources. This allows an attacker to predict and take over another user’s live BMC session across both the IPMI service and browser-based KVM consoles. The two most significant bugs are both present in Supermicro systems.

4. Pre-authentication memory corruptions. A length-validation error in the management SSH service is reachable before authentication and can be driven to execute malicious code. Moore found the vulnerabilities in HPE iLO systems.

5. The existence of unsigned or attacker-controllable firmware and unenforced configuration integrity. An authenticated administrator can install a persistent implant or replace the key used to verify firmware. These can be chained to separate authentication bypasses and privilege escalation vulnerabilities. Affected vendors include Supermicro, H3C, and Dell.

6. The use of secrets recoverable from firmware as live credentials. Keys and constants that can be extracted from public firmware can be used to authenticate to, or decrypt traffic from, BMCs. Affected vendors include Supermicro, OpenBMC, Huawei, and Dell.

7. Default and factory-random credentials that can be compromised by hash disclosure made possible through CVE-2013-4786. Frequently, devices continue to use default credentials. Even when the credentials have been changed before shipping, the small keyspaces of factory-randomized passwords make them recoverable in offline cracking attacks. Affected vendors include HPE, Supermicro, and Dell. HPE was the worst (eight digits or alphanum), and Supermicro and Dell use slightly longer defaults, which increase the cost of the attack and may delay the cleartext recovery by hours or days, depending on available compute.