Who’s in the Sandbox? The Labyrinths of Malware Analysis and the Eternal Game of Cat and Mouse

We used to just toss a file into VirusTotal, see some red, and think, 'Okay, this is malicious,' and go about our day. If we were feeling extra curious, we’d run strings to see if 'http://bad-domain.com' popped up. But what if that file comes back 0/72? Or worse, what if it's programmed to execute only when it hits your specific corporate network? No one deals with those old-school viruses that trigger signature-based security anymore; the trend has fully shifted toward context-aware and anti-analysis structures.

Hey everyone, Sedat here. While sipping my coffee at my desk at Payten today, I wanted to talk about how we examine those sneaky payloads we've been seeing lately, how they drive analysts crazy, and how we survive this labyrinth. If you're ready, let’s take a trip inside that suspicious .exe or .js file.

Static Analysis: Love at First Sight, or a Trap?

You should never run a file before taking an X-ray of it. Static analysis is the investigation of a file without executing its code. But there’s a big boss waiting for us here: Entropy. If a file’s entropy is above 7.5, you can bet there’s some packing or encryption going on. In other words, the attacker has hidden the real code behind a packer.

For example, when we look at the headers of a PE (Portable Executable) file, we might see traces of common packers like UPX. But in the field, things aren't always that easy. When we encounter custom crypters, the strings output of the file remains nothing but a pile of meaningless characters.

We can try to understand how 'messy' a file is with this simple Python script:

import math

def calculate_entropy(data):
    if not data:
        return 0
    entropy = 0
    # Calculate the probability of each byte in the file
    for x in range(256):
        p_x = float(data.count(x)) / len(data)
        if p_x > 0:
            entropy += - p_x * math.log(p_x, 2)
    return entropy

# defanged_path = "sample_suspicious_file.bin"
# entropy_value = calculate_entropy(open(defanged_path, "rb").read())
# print(f"File Entropy: {entropy_value}")

If this value is around 7.9, we are 99% sure there’s something hidden inside that file.

Dynamic Analysis: Playtime in the Sandbox

If we come back empty-handed from static analysis (which usually happens), it’s time to run the file in a controlled environment—a 'Sandbox.' But beware! Modern malware is very 'smart' now. The first thing they do is check: 'Am I in a virtual machine?'

Check out what a simple anti-VM check looks like in pseudo-code:

// Representative anti-VM check
void check_environment() {
    if (IsDebuggerPresent()) {
        // Throw an error and exit, show the analyst nothing
        exit(0);
    }

    // Check disk size (Sandboxes usually use small disks)
    // Or check for specific MAC addresses (e.g., 08:00:27 for VirtualBox)
}