The Things They Didn't Tell You About the Debugging APIs
Piotr Bania (Piotr) <baniapiotrgmailcom> Tuesday, December 26 2006 04:16.05 CST


After checking out all this DebugApis tutorials arround, i found that they forgot to teach people that filehandles given by the system to the debugger should be closed. This is not a big problem when your debugger dies together with an application, the problems comes when you are using it to debugging many files. In such situations your process handles count will fly like a skyrocket, which in the end will lead to a system unstabillity and a 100% resource consumption. Such sample situation is illustrated here > image <.

I heard from some guys, that those handles should be closed by the system (ContinueDebugEvent() and continuing the termination event), whats more funny its seems they are not :)

So how to obtain handles that should be closed? Lets assume we are working in DEBUG_ONLY_THIS_PROCESS mode (we dont care about its childs).

Firstly when a LOAD_DLL_DEBUG_EVENT occurs, the debugger receives a handle to the loaded DLL. You should store all those handles somewhere in memory (for example in your HANDLE list). Secondly (if the handle doesnt close automaticly) take care of CREATE_PROCESS_DEBUG_EVENT, it gives the debugger a handle to debugged file. And also all the thread handles received by CREATE_THREAD_DEBUG_EVENT should be also stored in the list.

The pseudo code should look like that:

--- SNIP ---
DEBUG_EVENT de;
...
case CREATE_THREAD_DEBUG_EVENT:
save_handle(de.u.CreateThread.hThread);
break;

case CREATE_PROCESS_DEBUG_EVENT:
save_handle(de.u.CreateProcessInfo.hFile);
break;

case LOAD_DLL_DEBUG_EVENT:
save_handle(de.u.LoadDll.hFile);
break;
--- SNIP ---

So now when to release those handles? Microsoft advices to close thread handles when EXIT_THREAD_DEBUG_EVENT occurs, close dll handles when UNLOAD_DLL_DEBUG_EVENT occurs, and so on. But if you are lazy i think u can release all the handles on a list when EXIT_PROCESS_DEBUG_EVENT occurs. To do that, simply use CloseHandle for every entry in the list. I've been monitoring my little proggy for few hours and i havent seen that its handles count changed. Of course, every time you end the debuggin process u need to close also pi.hProcess and pi.hThread (obtained from CreateProcess) - thats obvious.


In the end, there are some cases when u need to terminate the debugging program not in clean way, the worst thing u can do here it is using the TerminateProcess API, for couple of reasons:
- it does not free the process resources correctly
- may lead to somekind of deadlocks
- and there is no damn synchronization

Instead of using TerminateProcess, you should consider sending WM_QUIT message (if the process has a window) or if that fails you should try reverting its EIP to ExitProcess. This can be achieved by directly changing the thread context (SetThreadContext) or by creating remote thread which address will point directly to ExitProcess API.

The little problem with WM_QUIT message is that you must find a window handle only by knowing the PID. Thats really easy task, and the algorithm looks like follows:
1) Enum all windows (EnumWindows)
2) In the enum windows procedure, you need to use GetWindowThreadProcessId API which will give you process id by the HWND.
3) Every time you will get the PID from point nr.2 you should compare it with the pid you got from CreateProcess API.
4) If those are the same, you use PostMessage to send
WM_QUIT message.

Thats all :) Thank you for watching!

Comments
AlexIonescu Posted: Thursday, December 28 2006 19:10.04 CST
You could've taken a look at my articles on the frontpage, they talk about this :)

Piotr Posted: Friday, December 29 2006 04:06.00 CST
Hi AlexIonescu :)

Now when i saw them i can say they could help me with some troubles earlier :) Too bad im so damn lazy and i havent checked them long time ago, anyway nice reading, good stuff :)