Flag: Tornado! Hurricane!

Blogs >> RolfRolles's Blog

Created: Wednesday, August 6 2008 18:08.03 CDT  
Printer Friendly ...
Part 2: Introduction to Optimization
Author: RolfRolles # Views: 10071

Basically, VMProtect bytecode and the IR differ from x86 assembly language in four ways:

1)  It's a stack machine;
2)  The IR contains "temporary variables";
3)  It contains what I've called a "scratch" area, upon which computations are performed rather than in the registers;
4)  In the case of VMProtect Ultra, it's obfuscated (or rather, "de-optimized") in certain ways.

It turns out that removing these four aspects from the IR is sufficient preparation for compilation into sensible x86 code.  We accomplish this via standard compiler optimizations applied locally to each basic block.  In general, there are a few main compiler optimizations used in this process.  The first one is "constant propagation".  Consider the following C code.

int x = 1;
function(x);


Clearly x will always be 1 when function is invoked:  it is defined on the first line, and is not re-defined before it is used in the following line (the definition in line one "reaches" the use in line two; alternatively, the path between the two is "definition-clear" with respect to x).  Thus, the code can be safely transformed into "function(1)".  If the first line is the only definition of the variable x, then we can replace all uses of x with the integer 1.  If the second line is the only use of the variable x, then we can eliminate the variable.

The next is "constant folding".  Consider the following C code.

int x = 1024;
function(x*1024);


By the above remarks, we know we can transform the second line into "function(1024*1024);".  It would be silly to generate code that actually performed this multiplication at run-time:  the value is known at compile-time, and should be computed then.  We can replace the second line with "function(1048576);", and in general we can replace any binary operation performed upon constant values with the computed result of that operation.

Similar to constant propagation is "copy propagation", as in the below.

void f(int i)
{
  int j = i;
  g(j);
}


The variable j is merely a copy of the variable i, and so the variable i can be substituted in for j until the point where either is redefined.  Lacking redefinitions, j can be eliminated entirely.

The next optimization is "dead-store elimination".  Consider the following C code.

int y = 2;
y = 1;


The definition of the variable y on line one is immediately un-done by the one on line two.  Therefore, there is no reason to actually assign the value 2 to the variable; the first store to y is a "dead store", and can be eliminated by the standard liveness-based compiler optimization known as "dead-store elimination", or more generally "dead-code elimination".  Here's an example from the VMProtect IR.

ecx = DWORD Scratch:[Dword(44)]
ecx = DWORD Scratch:[Dword(20)]

After dead-store-eliminating the first instruction, it turns out no other instructions use Scratch:[Dword(44)], and so its previous definition can be eliminated as well.




Add New Comment
Comment:









There are 31,193 total registered users.


Recently Created Topics
Robbinhood ransomwar...
Feb/23
Information on the t...
Feb/08
Information on the m...
Feb/07
Order Finax, Fincar ...
Feb/07
Information on the m...
Feb/07
Order Proscar (Finas...
Feb/07
Order Proscar, Finax...
Feb/07
Order Finasteride, F...
Feb/07
How to view IDA Pro'...
Nov/02
reverse MC9S12DG128
Oct/07


Recent Forum Posts
Finding the procedur...
rolEYder
Question about debbu...
rolEYder
Looking for an advan...
tthtlc
Looking for an advan...
tthtlc
Looking for an advan...
clightning
Identify RVA data in...
sohlow
let 'IDAPython' impo...
sohlow
How to find specific...
hackgreti
Problem with ollydbg
sh3dow
How can I write olly...
sh3dow


Recent Blog Entries
nieo
Mar/22
Android Application Reversing

halsten
Mar/14
Breaking IonCUBE VM

oleavr
Oct/24
Anatomy of a code tracer

hasherezade
Sep/24
IAT Patcher - new tool for ...

oleavr
Aug/27
CryptoShark: code tracer ba...

More ...


Recent Blog Comments
ComPuer on:
May/14
Android Application Reversing

nieo on:
Mar/22
IAT Patcher - new tool for ...

djnemo on:
Nov/17
Kernel debugger vs user mod...

acel on:
Nov/14
Kernel debugger vs user mod...

pedram on:
Dec/21
frida.github.io: scriptable...

More ...


Imagery
SoySauce Blueprint
Jun 6, 2008

[+] expand

View Gallery (11) / Submit