Defeating Hackers and Malware With Disorder

Entropy — the measure of disorder or randomness — isn’t always desirable in the world of IT security. Kinda, sorta patching your IT systems sometimes, for example, would be a bad thing. At times, though, entropy can be a powerful tool, as in the case of well-chosen passwords that are difficult to crack. A fast-growing SIEM (security incident event managment) company called Vigilant is using entropy in an innovative way that warrants a closer look: Its anomaly-detection service identifies malicious threats based on entropy.

First, a quick primer: Entropy, often measured in bits, is the technical measurement of the randomness of the next piece of data in a string. If you see a sequence of letters with a clear pattern, such as “ABABABABA,” you would logically predict the next letter in the series will be B. Because the answer is fairly certain, the entropy would be 0. If you’re flipping a coin, the predicted outcome, heads or tails, is considered to be 1 bit of entropy. If a native English speaker is shown a sequence of standard English text and is asked to predict the next letter, he or she could guess it with great accuracy. English text is considered to have an entropy of 0.6 to 1.5 bits.

One more example: A truly random 8-character password, using every possible character on a standard keyboard, could have 52 bits of entropy. Unfortunately, most people use common words as part of their passwords, so most have only 18 bits of entropy. Thus, a password cracker doesn’t need to work through every permutation, just the most likely ones in the range of 2¹⁸, which is far easier than 2⁵². (Read Appendix A and Table A.1 of NIST Special Publication 800-63 for all the nerdy calculations and details.)

In the realm of IT security, antispam services have long used entropy and its link to anomaly detection to sniff out unwanted messages. If the service detects a single email account sending out messages to thousands of seemingly random and unconnected addresses, the provider will usually examine the message for other indicators that it’s spam. Entropy and its link in anomaly detection isn’t new, but Vigilant has extended it further to fighting malware.

Malware often generates random file names when writing Trojans to the system directory. When I perform light forensic analysis on a Windows machine, my first clue that it’s been compromised is one or more very strange-looking file names, such as vx3kjngq.exe or bb9[qamz.exe, in the Windows/System32 folder. Many malware programs use nonrandom names, but enough use randomly generated names that high-entropy filenames are a fairly good — if only rudimentary — indicator of something bad. (If you want to perform an experiment on your system, look for high-entropy file names in unexpected places, such as Windows/System32 or a root folder. Not every extremely weird name is a sign of maliciousness — but it’s a consistent characteristic.)

Vigilant has figured out that the average domain name has from 2.5 to 3.9 bits of entropy per byte. The company’s analysis algorithms flag DNS domains with more than 4.0 bits of entropy per byte; it also looks for less frequently occurring top-level domains, such as .biz and .info, with less entropy.

Vigilant’s services also looks for high-entropy file names and network connections to unexpected locations. But perhaps my favorite Vigilant check is for high entropy within HTTP content. Most HTTP content is close to the English language (or whatever native language is used) and should have low entropy. Encryption, on the other hand, has — or should have — high entropy. In fact, good crypto should have such high entropy that its encrypted ciphertext is indistinguishable from “noise.” Vigilant knows that advanced persistent threats often send out victims’ data in encrypted form using HTTP versus the normally expected encrypted HTTPS. If Vigilant’s service sees an outbound data stream using HTTP with high entropy, it raises a red flag.

Again, Vigilant isn’t the first company to use randomness in its anomaly detection, but it’s the first company of which I am aware that looks for, measures, and alerts of entropy characteristics. Sure, the company does all the traditional anomaly detection, but I like the fact that Vigilant uses mathematics and expected probabilities to add in another type of measurement.