In what I hope will be the debut of many in a series to come, I want to share my experience and musings as a frequent traveler in the world of hosting providers. Sometimes it will be just curious anomalies, of little interest to but a few. But perhaps some tales told here will ultimately prove valuable to others looking for insight even if to obscure scenarios. Maybe especially to obscure scenarios.
This inaugural article will focus on the unusual and curious operational behavior of precomputed SSH host keys in default system installs by LightNode.
Background
LightNode is the server hosting brand of Kaopu Cloud, a global network services provider based in China. They have been providing KVM-based hosting for approximately four years as of this writing. A compelling feature of LightNode’s offerings is the rather unique set of data center markets they operate in. Otherwise, most of the capabilities the LightNode platform has are similar to many global hosting providers. The one notable exception is they do not currently offer native IPv6 addressing and connectivity.
BGP routes covering LightNode prefixes are announced by AS 138915. This ASN is currently listed in the Spamhaus ASN-DROP. Spamhaus describes ASN-DROP this way:
“[ASN-DROP entries are] associated with the worst of the worst behavior. These are ASNs that our researchers wouldn’t recommend engaging with and are highly likely to announce or supply transit to IP ranges associated with malicious behavior. From networks hosting botnet command and control systems, to “bulletproof” networks selling connectivity/hosting to cyber criminals, to hardcore spammers, and more.”
Despite appearing in ASN-DROP, LightNode is relatively popular and seemingly widely used by many. At the time of this writing the originating AS announces the equivalent of about three /16 IPv4 prefixes or roughly 180 thousand addresses. Far fewer than the biggest cloud providers, and about an order of magnitude less than Linode (AS 63949) has for example, but still more than what most .edu’s hold.
To setup a new system through LightNode’s user interface you choose a location, system type, system version if applicable, and minimum resource requirements to name a few configurable options.
A system type may be a vanilla operating system image such as Ubuntu, Debian, AlmaLinux, Arch Linux, CentOS, FreeBSD, Rocky Linux, or Windows. For most OS types there are usually a few different versions to choose from. There are over a dozen application-specific system types if you want a preconfigured Docker or WordPress system for example. This process is straightforward and would seem familiar to anyone who has had experience using other hosting providers. There is nothing particularly special about this process. However, there is one very specific, default configuration detail on most if not all the LightNode installed systems I’ve not seen replicated by any other hosting provider.
Analysis
Each of LightNode’s Unix-based system options will install an OpenSSH
server listening on TCP port 22 by default. You have the option of
specifying access to the SSH service with a public key or password.
Normally, the SSH server host keys are generated as part of the package
installation process using the ssh-keygen utility. This would result
in one-time unique pairs of keys for each and every system. This is
where LightNode seems to do something very different than every other
hosting provider I’m aware of.
I tested a number of configurations and performed a survey of LightNode SSH servers listening on TCP port 22 to support this claim. The table below shows the number of TCP port 22 SSH listeners, distinct host keys, and distinct key types I found:
Property Count Listeners 30,183 Keys 1228 Key types 5
We would not expect to see so few keys compared to the number of TCP port 22
listeners. Most modern Unix-based systems would have two or three different
and unique keys per system, so in theory we’d expect see roughly 60,000 to
90,000 keys. Instead, on average we see at least 25 addresses for every key.
In fact it is a worse than that, because 1228 represents all key types and most
hosts use three key types on average. If we consider only ssh-rsa keys,
the most common type, this is what emerges:
Property Count Listeners 29,776 Keys 478
The number of listeners that have an ssh-rsa host key is 98.5% of the original count, but the number of unique keys is now 39% of the original. This is is significant amount of SSH host key duplication across LightNode’s infrastructure.
This leads us to ask at least one more question. If there are 478 distinct ssh-rsa host keys, how many times are each of those keys used? Surely LightNode doesn’t have 478 different system types does it? No, it is something less than that. We might not know exactly what the unique count is given that some templates may been retired, but we can examine keys by popularity. A table of statistics starts to get a little messy so I’ll simply tell you that the most popular ssh-rsa key is associated with LightNode’s Ubuntu 22.04 template. It is seen on over 10,000 Listener addresses! About one in three LightNode systems with a public SSH listener on TCP port run Ubuntu 22.04 and all use the same SSH host key.
So LightNode appears to have precomputed SSH host keys for each of the
unique system templates it offers as an install option. For example,
all LightNode Debian 12 installations, by default, will use the same SSH
host key pairs that you’ll find stored in /etc/ssh/ssh_host_*key*
files. Debian 11 will use another set of keys, but common to all Debian
11 installations. And so on. It is unclear if this is a failure of
entropy or if ssh-keygen is just never run, but the outcome is the same.
The host keys are predictable and shared with other hosts built from the
same system template configuration.
I can think of two scenarios where this anomaly might purposely occur in practice. One, some hosts have multiple addresses assigned, and each address has the same SSH server daemon active and willing to accept connections across all interfaces. In this case you’d see what looks like a multi-homed system. Honeypots may exhibit this behavior for example. Two, each listener address may be a different system, but host keys are being generated from the exact same entropy source for ssh-keygen, or are set to use precomputed ssh host keys. You might find this in certain enterprise or private environments as a way to simplify key management. It is this second scenario that LightNode appears to fall into.
It is unclear why LightNode sets up the SSH listener this way for public, independent systems. Perhaps the platform designer thought this would speed-up system setup and VM creation time? Should we care about this? Is it really bad? Could this be something more insidious? What, if anything, do we do?
Closing thoughts
Exposure of private encryption keys is a bad thing, and potentially catastrophic. However, when it comes to SSH server host keys, the damage is not as great as it might be compared to other public key systems. SSH server host keys are used to “identify” the system. An impersonation attack is feasible here. That is, if someone hijacked the SSH server IP address and used the same SSH host keys, an unwitting client would not receive an authentication warning, because from the perspective of the SSH protocol, it looks like the same system. A hijack, or MitM attack, of this sort might then be able to capture and decrypt communications from there on out. However, even if past traffic was captured and the SSH server host keys are known, a separate session key is used for traffic encryption that cannot be easily derived without direct access to an end system at the time of operation. In the immortal words of the The Hitchhiker’s Guide to the Galaxy: Don’t Panic.
Nevertheless, this scenario is less than ideal. I tried to contact LightNode support about this. I also tried to reach out to CNCERT/CC, the national computer network emergency response technical team / coordination center of China. A colleague also tried to communicate with a CNCERT contact on my behalf. These attempts have been met with silence. I’d guess this may help explain why the ASN is listed on the Spamhaus ASN-DROP. Perhaps, as another colleague once put it, the difference in our native languages can act as a natural firewall between us.
Technically LightNode or someone with the ability to hijack LightNode addresses could impersonate SSH servers there. I’m inclined to accept this is a relatively low probability threat, and likely an unintentional vulnerability on LightNode’s part. If you’re a LightNode customer, it would be wise to setup new host keys. If you or anyone you know has any ability to convince LightNode to alter this practice, I’d be happy to hear about it.