📚 OpenRCE is preserved as a read-only archive. Launched at RECon Montreal in 2005. Registration and posting are disabled.








Flag: Tornado! Hurricane!

 Forums >>  Brainstorms - General  >>  [C++] Hooking member functions with static member functions, or non-member functions

Topic created on: June 5, 2009 11:54 CDT by phantal .

  There's a program I'm messing around with to get more experience with hooking, and the problems that can arise.  The target makes very heavy use of classes, member functions, virtual functions, etc.  In this instance, the target is always compiled in Visual Studio 2005, so I don't need to worry about cross-platform semantics & the like.  I have access to the source code, and symbols, so I have a lot of freedom to figure out how things work.

  The target function is a non-virtual member function using the __thiscall calling convention, so this should be passed in ECX.  Here's a sample of the code:


#include <windows.h>
#include <iostream>
#include <string>
#include <sstream>
#include "detours.h"

#define getECX(p) _asm {mov p, ECX}
#define setECX(p) _asm {mov ECX, p}

typedef int (__stdcall *IIPtrType)(int param1);

IIPtrType REAL_originalFunction = 0;

int DETOUR_originalFunction (int result) {
  unsigned int pThis;
  getECX(pthis);

  std::stringstream msg;
  std::string *dataptr = (std::string *)(*(unsigned int *)(pThis + Off_1) + Off_2);
  std::string data(dataptr->c_str());

  /*...*/
}


  The this object has a member at offset Off_1, which has an std::string at offset Off_2.  It's a very simple situation, but I'm running into a few issues and I'm trying to understand what the ultimate cause is.

  The first problem is not a huge deal (I think).  The debugger shows pThis as having value 99d, but when I check the value in memory at ebp+4 it has the correct value of what is in ECX when the function is entered.  Additionally, when pThis is used elsewhere, the correct value does get used, so I'm convinced it's just an issue with the debugger.  Any idea what may be at issue here?

  The second problem is more serious.  The disassembly looks like this:


int DETOUR_originalFunction (int result) {
020C1050  sub         esp,0ACh
020C1056  mov         eax,dword ptr [___security_cookie (20C5008h)]
020C105B  xor         eax,esp
020C105D  mov         dword ptr [esp+0A8h],eax
020C1064  push        esi  
  unsigned int pThis;
  getECX (pThis);
020C1065  mov         dword ptr [esp+4],ecx
  std::stringstream msg;
020C1069  push        1    
020C106B  push        3    
020C106D  lea         ecx,[esp+10h]
020C1071  call        dword ptr [__imp_std::basic_stringstream<char,std::char_traits<char>,std::allocator<char> >::basic_stringstream<char,std::char_traits<char>,std::allocator<char> > (20C3064h)]
  std::string *methodPtr = (std::string *)(*(unsigned int *)(pThis + Off_1) + Off_2);
020C1077  mov         eax,dword ptr [esp+4]
020C107B  mov         eax,dword ptr [eax+0A4h]
020C1081  add         eax,0DCh


  If I check the location pointed at by (this + Off_1) before the stringstream constructor is called, there is a non-null value there.  After the constructor executes, it zeros that memory.  Therefore, when the instruction:

mov eax,dword ptr [eax+0A4h]

... executes, it loads zeros into eax, then adds 0xDC to that, then tries to treat memory location 0x000000DC as an std::string (which, of course, results in an access violation).

  Any comments?

-Brian

  cod     June 6, 2009 02:53.59 CDT
The __thiscall calling convention is used on member functions and is the default calling convention used by C++ member functions that do not use variable arguments. Under __thiscall, the callee cleans the stack, which is impossible for vararg functions. Arguments are pushed on the stack from right to left, with the this pointer being passed via register ECX, and not on the stack, on the x86 architecture.

Are you sure that program use __thiscall call convention? By default VC++ use __cdecl call convention.. and you have not specified a modifier for "DETOUR_originalFunction". For 2nd question, it would be bettor to have the IDB to check

  phantal     June 8, 2009 11:31.38 CDT
  You're right that VC++ uses __cdecl by default for non-member functions, but to quote from MSDN:

The __thiscall calling convention is used on member functions and is the default calling convention used by C++ member functions that do not use variable arguments.

  For clarity I probably should explicitly give a modifier for the detour function, but it's using __cdecl since it isn't a non-static member function.

  cod     June 9, 2009 01:46.58 CDT
You declare the pointer to REAL_Function how __stdcall, and declare your detour function how __cdecl... But what the convention call used by VC++ to invoke the member? __cdecl? __stdcall, __thiscall ? The signature of detour function must be the same of original function to avoid stack corruption and parameter corruption.
In __thiscall "this" is an implicit parameter into ECX register, but in __stdcall is the first parameter into stack, and ecx is undefined (it will have the value of this because maybe the caller will use a __thiscall convention for method)

  phantal     June 9, 2009 10:52.49 CDT
  Ahh, I see what you're saying.  I didn't make the connection (and should have).  To answer your question, I chose __stdcall because __thiscall (based on my read of this article on thunks) appears to be __stdcall with this passed in ECX.  The mistake I made was to not pay close enough attention to the calling convention for my detour function (as you pointed out).

  Thank you, I'll try that out when I get a moment and see whether that helps with the constructor clobbering memory issue.

-Brian

  phantal     June 10, 2009 11:45.36 CDT
  Found the culprit.  You were right in that I needed to specify the calling convention, however, of the two problems I mentioned earlier, the 2nd was caused by me trusting the documentation (silly, I know).

  The documentation says this is passed in ECX, but, at least in this instance, attaching to the process with windbg and the symbols for the dll revealed that the EDI register actually holds the pointer to this.

  I can only speculate that it's an optimization.

  The first problem I was able to resolve by using an inline function.  I haven't found evidence to support my conclusion, but I believe macros that use __asm can fool the debugger.

-Brian

  cod     June 10, 2009 22:50.57 CDT
what's your compiler? maybe today I can do some test..

  cod     June 11, 2009 06:07.22 CDT
Hi, I tried with my VC++ 2008 to create an hook on a method..

The problem are been:
1) I can't create a function with __thiscall modifier;
2) RTC_CheckESP
3) ECX value

Normally, VC create a function prolog to initialize the portion of stack for function (or it's necessary to disable runtime check in project settings). However the Detour_Method is a normal function, not a member function.. and compiler don't make a backup of ECX (it use ecx in REP STOSD to initialize stack portion). Using "naked" the prolog and epilog are removed...


__declspec( naked ) void __stdcall Detour_Method(int param1)
{
DWORD This, _ESI;
__asm { mov _ESI, esi}
__asm { mov This, ecx};
member(This, param1);
__asm { mov ESI, _ESI }
__asm { ret 4 };
}

I create a backup of "ESI" because it's used by RTC_CheckEsp to compare the ESP register after call ..

This is only a test and it can be written in a better way..

  phantal     June 11, 2009 09:59.17 CDT
  Correct, you can't give a __thiscall convention to a non-member function, or static member.  However, as I said earlier __thiscall is essentially __stdcall with this passed in a register.

  I'm using vs2005 for this.  All evidence I've found says this is in the edi register, whereas the documentation says it should be in ecx.  In truth, ecx does not contain the this pointer when this function is called, but edi does.

-Brian

  phantal     June 12, 2009 12:49.23 CDT
  One more comment, I figured out something else yesterday.  It appears that the documentation is right only in that this is passed in ecx only when a non-member function calls a member function on the object in question.  If the member function then calls another member function, all bets are off.

-Brian

  cod     June 13, 2009 02:03.58 CDT
But public members, or protected/private function members? Or the compiler create two entry point for same method?

  phantal     June 15, 2009 15:47.02 CDT
  I'll give a snippet of code to give you an idea of what I mean:



class someClass {
  int memberFunction_1 (int a);
  int memberFunction_2 (int a);
};

int someClass::memberFunction_1 (int a) {
// code
  this->memberFunction_2 (a*a);
    // since memberFunction_2 is called by memberFunction_1,
    // cannot guaruntee that, upon entry to memberFunction_2,
    // ECX will contain the this pointer
// more code
}

int someClass::memberFunction_2 (int a) {
// code
    // cannot guaruntee that ECX will contain the this pointer
    // once this function is called
}

int main (int argc, char* argv[]) {
  someClass myobject;

  myobject.memberFunction1 (3);
}


  I have no documentation to support my hypothesis, only anecdotal evidence.  When I detour functions that are usually called by a non-member function, ECX contains this, but member funtions usually called by other member functions don't always have a register that contains this, and when it does happen to be in a register, it is not always in ECX.

-Brian

Note: Registration is required to post to the forums.

There are 31,328 total registered users.


Recently Created Topics
[help] Unpacking VMP...
Mar/12
Reverse Engineering ...
Jul/06
let 'IDAPython' impo...
Sep/24
set 'IDAPython' as t...
Sep/24
GuessType return une...
Sep/20
About retrieving the...
Sep/07
How to find specific...
Aug/15
How to get data depe...
Jul/07
Identify RVA data in...
May/06
Question about memor...
Dec/12


Recent Forum Posts
Finding the procedur...
rolEYder
Question about debbu...
rolEYder
Identify RVA data in...
sohlow
let 'IDAPython' impo...
sohlow
How to find specific...
hackgreti
Problem with ollydbg
sh3dow
How can I write olly...
sh3dow
New LoadMAP plugin v...
mefisto...
Intel pin in loaded ...
djnemo
OOP_RE tool available?
Bl4ckm4n


Recent Blog Entries
halsten
Mar/14
Breaking IonCUBE VM

oleavr
Oct/24
Anatomy of a code tracer

hasherezade
Sep/24
IAT Patcher - new tool for ...

oleavr
Aug/27
CryptoShark: code tracer ba...

oleavr
Jun/25
Build a debugger in 5 minutes

More ...


Recent Blog Comments
nieo on:
Mar/22
IAT Patcher - new tool for ...

djnemo on:
Nov/17
Kernel debugger vs user mod...

acel on:
Nov/14
Kernel debugger vs user mod...

pedram on:
Dec/21
frida.github.io: scriptable...

capadleman on:
Jun/19
Using NtCreateThreadEx for ...

More ...


Imagery
SoySauce Blueprint
Jun 6, 2008

[+] expand

View Gallery (11) / Submit