0x55 0x48 0x89 0xE5 – This isn't just a random sequence of bytes. In the x86_64 world, this is that sacred moment where a function says, "I'm here, I'm starting." It’s the signature of a stack frame being born. When you’re staring at a binary without a single line of source code and you see those famous push rbp; mov rbp, rsp instructions, you’ve officially taken your first step into the labyrinth.
Hey man, let’s change the "vibe" today and get our hands a little dirty in the world of Reverse Engineering (RE). You know, that world we all love but which occasionally makes us want to pull our hair out. In Red Teaming, it’s not always as simple as finding a public exploit and typing whoami. Sometimes you run into a custom protocol, a weird licensing check, or some heavily obfuscated malware, and you’re left with no choice but to ask: "What the hell is actually going on inside this thing?"
Close Your Eyes and See the Matrix
When you compile a beautiful C program, all those readable if-else blocks, switch-case structures, and fancy function names get chewed up and turned into machine code. Once the linker is done, we’re left with a binary that the CPU understands perfectly, but we (under normal circumstances) can’t make heads or tails of.
Reverse engineering is the art of trying to rewind that process. But look, it's not like restoring a file from the "Recycle Bin." It’s more like trying to turn ground beef back into a cow. Is it impossible? No. It just takes a lot of patience and the right set of tools.
We usually have two main paths: Static Analysis and Dynamic Analysis.
In static analysis, we don't touch the code; we just look. It’s like studying a painting. We fire up tools like IDA Pro, Ghidra, or Binary Ninja and analyze the Control Flow Graphs (CFG). In dynamic analysis, we actually run the code. We get inside its veins using x64dbg or GDB (with that legendary GEF plugin) and watch the registers change in real-time.
A Simple Scenario: Sledgehammer or Lockpick?
Let's say we have a piece of C code. We don't have the source, just the compiled binary:
#include <stdio.h>
#include <string.h>
int check_password(char *input) {
if (strcmp(input, "P4yt3n_R3d_T34m_2024") == 0) {
return 1;
}
return 0;
}
int main() {
char password[64];
printf("Sifreyi gir kanka: ");
scanf("%63s", password);
if (check_password(password)) {
printf("Sistem acildi, iceridesin!\n");
} else {
printf("Yanlis kapi, baska kapiya.\n");
}
return 0;
}
If you compile this (gcc crackme.c -o crackme) and give it to someone, they could probably find the password just by running the strings command. But real life isn't that merciful. Passwords get hashed, XORed, or put through some bizarre custom algorithms.
Now, let's look at how this looks on the Assembly (x64) side. The check_password function will likely transform into something like this:
0000000000001149 <check_password>:
1149: 55 push rbp
114a: 48 89 e5 mov rbp,rsp
See those first two lines? That's our prologue. We're saving the previous base pointer and setting up a new one for this function's local variables. From here on, it’s a game of following the data. We’d see the input string being moved into rdi, the hardcoded password (or its reference) being moved into rsi, and then a call to strcmp.
If you're doing a Red Team engagement and you need to bypass this check, you don't necessarily need to know the password. You could just patch the binary—change a jne (jump if not equal) to a je (jump if equal), or just force rax to be 1 before the function returns. That’s the beauty of RE; you're the one rewriting the rules of the game.