OpenRCE

About Articles Book Store Distributed RCE Downloads Event Calendar Forums Live Discussion Reference Library RSS Feeds Search Users What's New

Customize Theme

Flag: Tornado! Hurricane!

Created: Wednesday, August 6 2008 18:08.03 CDT

Printer Friendly ...

Part 2: Introduction to Optimization

Author: RolfRolles

# Views: 5888

Basically, VMProtect bytecode and the IR differ from x86 assembly language in four ways:

1)  It's a stack machine;
2)  The IR contains "temporary variables";
3)  It contains what I've called a "scratch" area, upon which computations are performed rather than in the registers;
4)  In the case of VMProtect Ultra, it's obfuscated (or rather, "de-optimized") in certain ways.

It turns out that removing these four aspects from the IR is sufficient preparation for compilation into sensible x86 code.  We accomplish this via standard compiler optimizations applied locally to each basic block.  In general, there are a few main compiler optimizations used in this process.  The first one is "constant propagation".  Consider the following C code.

int x = 1;

function(x);

Clearly x will always be 1 when function is invoked: it is defined on the first line, and is not re-defined before it is used in the following line (the definition in line one "reaches" the use in line two; alternatively, the path between the two is "definition-clear" with respect to x). Thus, the code can be safely transformed into "function(1)". If the first line is the only definition of the variable x, then we can replace all uses of x with the integer 1. If the second line is the only use of the variable x, then we can eliminate the variable.

The next is "constant folding". Consider the following C code.

int x = 1024;

function(x*1024);

By the above remarks, we know we can transform the second line into "function(1024*1024);". It would be silly to generate code that actually performed this multiplication at run-time: the value is known at compile-time, and should be computed then. We can replace the second line with "function(1048576);", and in general we can replace any binary operation performed upon constant values with the computed result of that operation.

Similar to constant propagation is "copy propagation", as in the below.

void f(int i)

{

  int j = i;

  g(j);

}

The variable j is merely a copy of the variable i, and so the variable i can be substituted in for j until the point where either is redefined. Lacking redefinitions, j can be eliminated entirely.

The next optimization is "dead-store elimination". Consider the following C code.

int y = 2;

y = 1;

The definition of the variable y on line one is immediately un-done by the one on line two. Therefore, there is no reason to actually assign the value 2 to the variable; the first store to y is a "dead store", and can be eliminated by the standard liveness-based compiler optimization known as "dead-store elimination", or more generally "dead-code elimination". Here's an example from the VMProtect IR.

ecx = DWORD Scratch:[Dword(44)]
ecx = DWORD Scratch:[Dword(20)]

After dead-store-eliminating the first instruction, it turns out no other instructions use Scratch:[Dword(44)], and so its previous definition can be eliminated as well.

There are 28,212 total registered users.

Recently Created Topics
Reverse Engineering ...	Jan/23
Career: DoD Agency I...	Jan/22
"Disappearing&q...	Jan/17
Career: Software Sec...	Jan/11
Where is the call st...	Jan/07
IDA Pro 6.1 Breakpoi...	Jan/01
How to create data s...	Dec/30
can i search all mod...	Dec/23
IDA symbol table exp...	Dec/20
An anti-attach trick	Dec/17

Recent Forum Posts
Reverse Engineering ...	NirIzr
"Disappearing&q...	NirIzr
Reverse Engineering ...	charlie
"Disappearing&q...	charlie
An anti-attach trick	Bass
An anti-attach trick	waleeda...
An anti-attach trick	Bass
An anti-attach trick	waleeda...
An anti-attach trick	Bass
Looking for value in...	NirIzr

Recent Blog Entries
	Ludwig	Feb/04
chi on sale
	Ludwig	Feb/04
Monster In The Vicinity Of ...
	Ludwig	Feb/04
Supra footwear Online
	waleedassar	Jan/31
Yet Another Anti-Debug Trick
	RolfRolles	Jan/22
Finding Bugs in VMs with a ...
More ...

Recent Blog Comments
	waleedassar on:	Feb/01
Yet Another Anti-Debug Trick
	NirIzr on:	Jan/31
Yet Another Anti-Debug Trick
	jackchen on:	Jan/10
nike mercurial vapor iii
	waleedassar on:	Dec/27
A new Anti-Olly trick.
	PeterFerrie on:	Dec/27
A new Anti-Olly trick.
More ...

Imagery

SoySauce Blueprint
Jun 6, 2008

[+] expand

View Gallery (11) / Submit