OpenRCE

While analyzing a Tdss-packed sample, I stumbled across an interesting use of the CreatePipe API function for Anti-Emulation purposes. The idea is very simple: detecting and thwarting Emulating Sandboxes without valid implementation of the pipe subsystem.

How does it work? Well, the very first thing the malware did was to create an unnamed pipe using the CreatePipe function:

.text:004014CA push 0
.text:004014CC push 0
.text:004014CE lea eax, [ebp-18h]
.text:004014D1 push eax
.text:004014D2 lea eax, [ebp-14h]
.text:004014D5 push eax
.text:004014D6 call ds:CreatePipe
.text:004014D6

Now that the pipe has been created, the malware will proceed in writing to it:

.text:004013F1 push 0 ; [0x12fe78] = 0x00000000
.text:004013F3 lea eax, [ebp-1Ch]
.text:004013F6 push eax ; [0x12fe74] = 0x0012FFA0
.text:004013F7 push 0Ch ; [0x12fe70] = 0x0000000C
.text:004013F9 lea eax, [ebp-10h]
.text:004013FC push eax ; [0x12fe6c] = 0x0012FFAC�
.text:004013FD push dword ptr [ebp-18h] ; [0x12fe68] = 0x00012000
.text:00401400 call ds:WriteFile ; WriteFile
.text:00401400

If we look closely, we see that the malware wrote 0xC bytes to the pipe:

WriteBuffer:
.stack:0012FFAC dd 0FC7E251Eh [WriteBuffer + 0x0]
.stack:0012FFB0 dd 0A6B546CBh [WriteBuffer + 0x4]
.stack:0012FFB4 dd 401599h � �[WriteBuffer + 0x8]
.stack:0012FFB8 dd 4015A0h � �[WriteBuffer + 0xC]

Now comes the interesting part: after some junk operations, the malware will read again from the pipe, and will store the read data at offset +4 from the beginning of the buffer that was initially written to the pipe:

.text:00401509 push 0 ; [0x12fe78] = 0x00000000
.text:0040150B lea eax, [ebp-1Ch]
.text:0040150E push eax ; [0x12fe74] = 0x0012FFA0
.text:0040150F push 19h ; [0x12fe70] = 0x00000019
.text:00401511 lea eax, [ebp-0Ch]
.text:00401514 push eax ; [0x12fe6c] = 0x0012FFB0
.text:00401515 push dword ptr [ebp-14h] ; [0x12fe68] = 0x00011000
.text:00401518 call ds:ReadFile ; ReadFile

The outcome of this write-read combination from the pipe is rather obvious: part of the content of WriteBuffer gets shifted of 4 bytes, overwriting the initial content.

Why does the malware does these kinds of operations? Well, we discover it very quickly: after another bunch of useless instructions, the malware uses the offset stored at WriteBuffer + 0xC to continue the execution:

.text:00401487 call dword ptr [ebp-4] ; [0x12FFB8] WriteBuffer + 0xC

and, if we focus on the called address, we see that it is exactly one of the locations that gets overwritten by the shifted content. In fact:

[WriteBuffer + 0xC]:
.stack:0012FFB8 dd 4015A0h --> .stack:0012FFB8 dd 401599h

So the calls destination address has been changed from 0x4015A0 to 0x401599. If we quickly inspect the two locations, we immediately notice that the overwritten offset points to garbage:

.text:004015A0 jl short near ptr loc_40152E+5
.text:004015A2 inc edx
.text:004015A3 int 3 ; Trap to Debugger
.text:004015A4 mov ch, 9Dh
.text:004015A6 inc edx
.text:004015A7 int 3 ; Trap to Debugger
.text:004015A8 adc [edi+536FCC42h], esp
.text:004015AE pop edi
.text:004015AF pusha
.text:004015B0 dec ebp
.text:004015B1 rcl dword ptr [ebp+5E895EA6h], 0BAh
.text:004015B8 outsd
.text:004015B9 xchg eax, ebx
.text:004015BA mov al, ss:0ACAE1BF0h
.text:004015C0 ...

while the overwriting one points to the valid code that will allow us to continue the execution:

.text:00401599 xor edx, edi
.text:0040159B jmp loc_401057

The description above can be visually explained by the following diagram:

It is obvious how such a trick can cause problems even to Emulating Sandboxes that do feature extensive re-implementation of Windows APIs. In fact, unless you have a solid implementation of the pipe subsystem in an Emulating Sandbox, this code cannot be emulated, as the original wrong offset wont be overwritten and the execution would break right after the call.

It is also interesting to note that using Pipes is somewhat more effective than using other equivalent mechanisms (for example, the same trick could be used with files). The main reason for this is that implementation of pipe-related functions is not usually a priority for emulator developers, and such implementation may even be neglected given the huge amount of other APIs that need implementation.