OpenRCE

📚 OpenRCE is preserved as a read-only archive. Launched at RECon Montreal in 2005. Registration and posting are disabled.

About Articles Book Store Distributed RCE Downloads Event Calendar Forums Live Discussion Reference Library RSS Feeds Search Users What's New

Customize Theme

Flag: Tornado! Hurricane!

Created: Tuesday, December 26 2006 04:16.05 CST

Modified: Wednesday, January 3 2007 05:42.22 CST

Printer Friendly ...

The Things They Didn't Tell You About the Debugging APIs

Author: Piotr

# Views: 3154

After checking out all this DebugApis tutorials arround, i found that they forgot to teach people that filehandles given by the system to the debugger should be closed. This is not a big problem when your debugger dies together with an application, the problems comes when you are using it to debugging many files. In such situations your process handles count will fly like a skyrocket, which in the end will lead to a system unstabillity and a 100% resource consumption. Such sample situation is illustrated here > image <.

I heard from some guys, that those handles should be closed by the system (ContinueDebugEvent() and continuing the termination event), whats more funny its seems they are not :)

So how to obtain handles that should be closed? Lets assume we are working in DEBUG_ONLY_THIS_PROCESS mode (we dont care about its childs).

Firstly when a LOAD_DLL_DEBUG_EVENT occurs, the debugger receives a handle to the loaded DLL. You should store all those handles somewhere in memory (for example in your HANDLE list). Secondly (if the handle doesnt close automaticly) take care of CREATE_PROCESS_DEBUG_EVENT, it gives the debugger a handle to debugged file. And also all the thread handles received by CREATE_THREAD_DEBUG_EVENT should be also stored in the list.

The pseudo code should look like that:

--- SNIP ---
DEBUG_EVENT de;
...
case CREATE_THREAD_DEBUG_EVENT:
save_handle(de.u.CreateThread.hThread);
break;

case CREATE_PROCESS_DEBUG_EVENT:
save_handle(de.u.CreateProcessInfo.hFile);
break;

case LOAD_DLL_DEBUG_EVENT:
save_handle(de.u.LoadDll.hFile);
break;
--- SNIP ---

So now when to release those handles? Microsoft advices to close thread handles when EXIT_THREAD_DEBUG_EVENT occurs, close dll handles when UNLOAD_DLL_DEBUG_EVENT occurs, and so on. But if you are lazy i think u can release all the handles on a list when EXIT_PROCESS_DEBUG_EVENT occurs. To do that, simply use CloseHandle for every entry in the list. I've been monitoring my little proggy for few hours and i havent seen that its handles count changed. Of course, every time you end the debuggin process u need to close also pi.hProcess and pi.hThread (obtained from CreateProcess) - thats obvious.

In the end, there are some cases when u need to terminate the debugging program not in clean way, the worst thing u can do here it is using the TerminateProcess API, for couple of reasons:
- it does not free the process resources correctly
- may lead to somekind of deadlocks
- and there is no damn synchronization

Instead of using TerminateProcess, you should consider sending WM_QUIT message (if the process has a window) or if that fails you should try reverting its EIP to ExitProcess. This can be achieved by directly changing the thread context (SetThreadContext) or by creating remote thread which address will point directly to ExitProcess API.

The little problem with WM_QUIT message is that you must find a window handle only by knowing the PID. Thats really easy task, and the algorithm looks like follows:
1) Enum all windows (EnumWindows)
2) In the enum windows procedure, you need to use GetWindowThreadProcessId API which will give you process id by the HWND.
3) Every time you will get the PID from point nr.2 you should compare it with the pid you got from CreateProcess API.
4) If those are the same, you use PostMessage to send
WM_QUIT message.

Thats all :) Thank you for watching!

Blog Comments

AlexIonescu	Posted: Thursday, December 28 2006 19:10.04 CST
You could've taken a look at my articles on the frontpage, they talk about this :)

Piotr	Posted: Friday, December 29 2006 04:06.00 CST
Hi AlexIonescu :) Now when i saw them i can say they could help me with some troubles earlier :) Too bad im so damn lazy and i havent checked them long time ago, anyway nice reading, good stuff :)

There are 31,328 total registered users.

Recently Created Topics
[help] Unpacking VMP...	Mar/12
Reverse Engineering ...	Jul/06
let 'IDAPython' impo...	Sep/24
set 'IDAPython' as t...	Sep/24
GuessType return une...	Sep/20
About retrieving the...	Sep/07
How to find specific...	Aug/15
How to get data depe...	Jul/07
Identify RVA data in...	May/06
Question about memor...	Dec/12

Recent Forum Posts
Finding the procedur...	rolEYder
Question about debbu...	rolEYder
Identify RVA data in...	sohlow
let 'IDAPython' impo...	sohlow
How to find specific...	hackgreti
Problem with ollydbg	sh3dow
How can I write olly...	sh3dow
New LoadMAP plugin v...	mefisto...
Intel pin in loaded ...	djnemo
OOP_RE tool available?	Bl4ckm4n

Recent Blog Entries
	halsten	Mar/14
Breaking IonCUBE VM
	oleavr	Oct/24
Anatomy of a code tracer
	hasherezade	Sep/24
IAT Patcher - new tool for ...
	oleavr	Aug/27
CryptoShark: code tracer ba...
	oleavr	Jun/25
Build a debugger in 5 minutes
More ...

Recent Blog Comments
	nieo on:	Mar/22
IAT Patcher - new tool for ...
	djnemo on:	Nov/17
Kernel debugger vs user mod...
	acel on:	Nov/14
Kernel debugger vs user mod...
	pedram on:	Dec/21
frida.github.io: scriptable...
	capadleman on:	Jun/19
Using NtCreateThreadEx for ...
More ...

Imagery

SoySauce Blueprint
Jun 6, 2008

[+] expand

View Gallery (11) / Submit