Topic created on: September 4, 2006 05:57 CDT by GynvaelColdwind .

Hi ;>

Recently I was codding a disassembler, and found something interesing. Mainly, the way that disassemblers treat REP/REPNE prefixes differs in every disassembler. It looks like it differs also in CPU -emulators- like VirtualPC.

If a real-life CPU executes the following code:
REP REP REP ... (15xREP total) ... REP MOVSB, an "Illegal Instruction" exception is generated. If there are 14xREP total, the exception is not generated. Well. It looks like VirtualPC 2004 does not generate the exception even if there are 15xREP. So this can be used to detect if the application is runned from VPC ;>

Could anyone check this on QEMU or Bochs, or on different build of VirtualPC ? VMWare behaves just like the real CPU.

proof of concept code follows:

binary:
http://www.openrce.org/repositories/users/GynvaelColdwind/vpcredpill.exe

source:

--vpcredpill.asm---
; masm32
; research & code by gynvael.coldwind//vx
; special thx to ReWolf (even more research ;>) & vul7ur3 (testing)
.386
.model flat, stdcall
option casemap :none   ; case sensitive

include \masm32\include\windows.inc
include \masm32\include\user32.inc
include \masm32\include\kernel32.inc

includelib \masm32\lib\user32.lib
includelib \masm32\lib\kernel32.lib

.code

start:
  ; some strings
  jmp @F
    szDlgTitle    db "VirtualPC 2004 RedPill (by gynvael.coldwind//vx)",0
    szMsgOFF      db "VirtualPC was NOT detected",0
    szMsgON       db "VirtualPC DETECTED!",0
  @@:

  ; SEH
  xor eax, eax
  push offset detected
  db 064h ; FS
  push dword ptr [eax]
  db 064h ; FS
  mov dword ptr [eax], esp

  ; teh RedPill
  mov esi, esp
  mov edi, esp
  mov ecx, 1

  ; This is REP REP REP REP ... REP movsb
  ; 15 * REP generate 'Invalid Instruction' exception on real CPU (tested on both Intel and AMD)
  ; Microsoft Virtual PC 2004 does NOT generate this exception.
  db 0F3h,0F3h,0F3h,0F3h,0F3h,0F3h,0F3h,0F3h,0F3h,0F3h,0F3h,0F3h,0F3h,0F3h,0F3h;
  movsb    

  ; was found!
  invoke MessageBox, 0, ADDR szMsgON, ADDR szDlgTitle, MB_OK
  invoke ExitProcess, 0

detected:
  invoke MessageBox, 0, ADDR szMsgOFF, ADDR szDlgTitle, MB_OK
  invoke ExitProcess, 0
  
end start