Making C compiler generate obfuscated code
Dennis Yurichev (modest) <dennisconusinfo> Tuesday, December 7 2010 08:55.43 CST


A customer of mine asked whether it is possible to protect his software from reverse engineering. I didn't found any C/C++ compiler which was able to produce obfuscated code making it hard to reverse engineer and complicate the use of such tools as Hex-Rays Decompiler, so I made a little attempt to hack Tiny C compiler's codegenerator.

I patched it so it produces a lot of random noise code between effective code. Of course, resulting code will work much slower. But in real life, we can obfuscate only critical parts of code containing algorithms we don't want to be easily leaked. Of course, it is virtually impossible to protect any code from reverse engineering, but it is possible to make it much more difficult.

Example: simple function:


int a (int a, int b)
{
return a + b * 4;
};


On output...


a               proc near

var_CD500B      = byte ptr -0CD500Bh
arg_0           = dword ptr  8
arg_4           = dword ptr  0Ch
arg_1D364BDE    = byte ptr  1D364BE6h

                nop
                push    ebp
                mov     ebp, esp
                sub     esp, 0
                nop
                xor     eax, ebx
                mov     eax, 99B7A34Ah
                mov     eax, 0EC06E7ACh
                lea     edx, [esi+63h]
                mov     ebx, [ebp+arg_0]
                and     ebx, ebx

loc_800001F:
                lea     ebx, [ebp+arg_1D364BDE]
                mov     ebx, 9EF81F3Eh
                lea     eax, [ebx+3Eh]
                lea     ecx, [esi]
                mov     eax, 0FD6D5D47h
                sub     ebx, edx
                lea     ecx, [ebp+var_CD500B]
                lea     ecx, [eax]
                mov     eax, [ebp+arg_4] ; *
                shl     eax, 2          ; *
                mov     ecx, eax
                adc     ecx, edx
                mov     ecx, [ebp+arg_0]
                adc     ecx, ecx
                sub     edx, ecx
                sub     edx, eax
                lea     ebx, [esp+ecx*8]
                mov     ecx, 29262C66h
                mov     ebx, 0CC18D2C4h
                mov     ebx, 0FDB56490h
                mov     ecx, 9E709D5Eh
                mov     ecx, 73805EBFh
                mov     ecx, eax
                or      ecx, eax
                mov     ebx, 7339AD0Eh
                mov     edx, 2CA8725Ah
                lea     edx, [edi+esi*8]
                mov     ebx, 87684A89h
                mov     ebx, 52A74759h
                xor     edx, edx
                jnz     short loc_800001F
                mov     ebx, 0CCA90613h
                sub     ecx, eax
                mov     ecx, 0C6699FDh
                mov     ebx, 0A8B272A1h
                mov     ebx, eax
                sbb     ebx, ebx
                mov     ecx, [ebp+arg_0] ; *
                add     ecx, eax        ; *
                or      edx, ebx
                mov     edx, 47257B14h
                mov     edx, ecx
                add     edx, edx
                mov     eax, 9E3E878Ah
                mov     ebx, 0DAB5E429h
                mov     edx, 0ABFDB94Eh
                adc     eax, ebx
                add     edx, ebx
                lea     edx, [ebx+75A1EF29h]
                or      edx, edx
                mov     eax, ecx        ; *
                jmp     $+5
                leave
                pop     ebx
                jmp     ebx
a               endp


(effective code marked with asterisk)

One funny thing is that now the compiler uses random number generator. Almost all good computer programs contain at least one random-number generator. (fortune file in plan 9 OS).

Here is also my crackme I created for testing. It was eventually reversed, though.

For those who are interested:
Patch for Tiny C version 0.9.25
Full source code patched
Tiny C 0.9.25 patched win32 executables

Comments
sparkrisp Posted: Tuesday, December 21 2010 08:40.06 CST
Excellent work! testing now...

thanks.

Genius Posted: Friday, December 24 2010 05:17.12 CST
Interesting work, however there's a key always for a lock , but it's matter to make it difficult to reverse a software .
thanks for this interesting work .

palaniyappan Posted: Saturday, January 8 2011 06:53.38 CST
No binary in this world is Non reversable if reaches a right hand..

prot0man Posted: Sunday, February 27 2011 11:19.17 CST
Good job! One thing I did notice is that it'd be really easy to write an IDA script that detects the real behavior of the function by starting at the end of the function, and looking at all registers / calls that affect eax (basically taint analysis). After all instructions are discovered that actually have an impact on eax, it would significantly limit the noise output of your tweaked compiler.

Though I'm sure your solution fulfilled your customers requirements, if you were ever looking to make the output even more difficult to reverse engineer, you could steal a few pages from vmprotect's (or other packers). One way you could make the generated code harder to reverse engineer is throw in function calls that modify memory that registers point to (between operations and on return). This of course could be trvially unraveled as well, but adding more depth in terms of calls in a function would certainly make the output more hideous, and the code to prune noise instructions more complex.

Sirmabus Posted: Wednesday, March 30 2011 04:29.49 CDT
Run it in IDA Pro if you have HexRays.
HexRays can do a good job of taking obfuscated ASM and making it readable C.
Just to test anyhow..

igorsk Posted: Friday, April 1 2011 08:30.04 CDT
Out of curiosity I tried this function in Hex-Rays. I had to manually reduce the stack frame size to make the decompiler happy, but the result was nice:

int __cdecl a(int a1, int a2)
{
  return 4 * a2 + a1;
}

dongi Posted: Monday, June 6 2011 13:54.11 CDT
Prior Art :-)

http://www.openrce.org/forums/posts/1295