Overview

Process Stalking is a term coined to describe the combined process of run-time profiling, state mapping and tracing. Consisting of a series of tools and scripts the goal of a successful stalk is to provide the reverse engineer with an enjoyable interface to filtered, meaningful, run-time block-level trace data.

The Process Stalking suite is broken into three main components; an IDA Pro plug-in, a stand alone tracing tool and a series of Python scripts for instrumenting intermediary and GML graph files. The generated GML graph definitions were designed for usage with a freely available interactive graph visualization tool.

IDA Pro Plug-in

To install the IDA plug-in, copy the appropriate version of process_stalker.plw to the plugins folder in your top-level IDA Pro install directory. A successful install will generate the following message in the IDA log window upon restart:

[*] pStalker> Process Stalker - Profiler
[*] pStalker> Pedram Amini XXXXXXXXXXXXXXXXXXXXXX
[*] pStalker> Compiled on Jul 5 2005

The IDA plug-in is used for pre-processing target binaries and generates the following outputs:

• Individual function-level GML graphs
• Cross references file, listing all inter-function calls.
• Break point list, with entries for each basic block within the binary.
To start the plug-in, wait for IDA's auto analysis to finish and either launch it with the hot key Alt + 5 or from the Edit -> Plugins menu. The following dialog should appear:


Process Stalker IDA plug-in dialog

The default options should be sufficient in most cases however here is a description of what they are:

Enable Instruction Colors: Controls whether or not the plug-in should generate colored instructions. When dealing with large graphs, disabling instruction-level coloring can assist in performance gain.
Enable Comments: Enable or disable the inclusion of IDA comments in the generated graphs.
Allow Self Loops: The orthogonal layout is not available in graphs that contain self loops.
Once the desired options have been selected hit OK and select a target directory to output generated files to. Do not worry about the filename entry, the filename is stripped so anything will suffice. The plug-in can take some time when analyzing the binary and will output the following messages in the IDA log window:

[*] pStalker> Profile analysis 25% complete.
[*] pStalker> Profile analysis 50% complete.
[*] pStalker> Profile analysis 75% complete.
[*] pStalker> Profile analysis 100% complete.

Graphs are generated in the GML language specifically for GDE Community Edition, a freeware GML viewer available for download from www.oreas.com. Graphs are interactive, editable, sport instruction level coloring for easy reading and can be displayed with a number of layout algorithms such as hierarchical, orthogonal, symmetric, circular, etc... Entry points are highlighted in green, true/false branches are colored green/red accordingly and implicit edges are colored blue.

Process Stalker - Tracer

The Process Stalker stand alone tracer utility requires no installation. The executable, process_stalker.exe, is a command line utility used to attach to or load a process:

usage:
  process_stalker <-a pid | -l filename | -la filename args>

options:
  [-b bp list]   specify the breakpoint list for the main module.
  [-r recorder]  enter a recorder (0-9) from trace initiation.
  [--one-time]   disable breakpoint restoration.
  [--no-regs]    disable register enumeration / dereferencing.

-a pid: Attach to process with process id 'pid'.
-l filename: Load a process with no command line options.
-la filename args: Load a process and pass in the arguments specified by 'args'.
-b breakpoint list: Load and parse the specified breakpoint list for the main image. Note that you don't need to specify this argument for subsequently loaded DLLs.
-r recorder: Start recording immediately into the specified recorder. Valid arguments are 0 through 9. Recording output is saved to [pid | filename].[recorder #]
--one-time: If specified, disable breakpoint restores. Specifying this option can result in a significant performance gain.
--no-regs: If specified, disable register enumeration and dereferencing. Specifying this option can result in a significant performance gain. Register recording output is saved to [pid | filename]-regs.[recorder #]
Pertinent breakpoint lists must exist in the directory Process Stalker is being executed from. An initial breakpoint list needs to be specified on the command line for the main image only. As the target process loads modules, Process Stalker will examine the current directory for MODULE_NAME.bpl. If found, the breakpoint list will be automatically loaded and parsed. The following output demonstrates the start of a successful session of Process Stalker on PID 864, Hyper Terminal:

$ process_stalker -a 864 --one-time

process stalker
pedram amini XXXXXXXXXXXXXXXXXXXXXX
compiled on Jul 5 2005
target: 864

processing breakpoints for module HYPERTRM.dll at 67441000
done. 10568 of 10577 breakpoints set.

initial break, tid = 0794.

commands: [h]   this screen                     [m] module list
          [0-9] enter recorder modes            [x] stop recording
          [v]   toggle verbosity
          [d]   detach (XP/2003 only)           [q] quit/close

004d68b2 T:00000a74 [bp] 6747D041 mov edi,edi
004d68b2 T:00000a74 [bp] 6747D05C cmp [ebp+0C],0x1
004d68b2 T:00000a74 [bp] 6747D0AB cmp [ebp+0C],esi
004d68bc T:00000a74 [bp] 6747D0EB xor eax,eax
...

As breakpoints are hit Process Stalker will generate a log entry to STDOUT. The format of the entry is as follows. The first field, 004d68bc, displays the current tick count as reported by Windows. The second field, T:00000a74, displays the thread id that this breakpoint was reached in. The third field, [bp], displays the recorder that this entry was written to. Recording is off by default and is denoted by the tag '[bp]', otherwise the recording number would appear in the tag. The fourth field, 6747D0EB, displays the address of the breakpoint. The final field, xor eax,eax, displays the disassembly of the instructions at the breakpoint address.

Hotkeys can be used to control Process Stalker at any point while it is running. The numeric keys, [0-9], signal to Process Stalker that it should start recording in the specified recorder slot. The recorder file name is specified by the attached pid or loaded filename and appended with the extension .[recorder #]. Recorder files are appended to and not overwritten. To close a recorder, strike the [x] hotkey. The [v] hotkey toggles breakpoint-level output logging. Disable logging to for increased performance. The [m] hotkey, displays a list of all loaded modules and a list of modules that are currently being monitored:

---------- MODULE LIST ----------
module 01001000 HYPERTRM.EXE
module 7c801000 KERNEL32.dll
module 7c901000 ntdll.dll
module 7c9c1000 SHELL32.dll
...

stalking:
67441000 - 67480000 [0003f000] HYPERTRM.dll
---------- MODULE LIST ----------

Use the [q] hotkey to close any open recordings as well as the debuggee. Alternatively, on systems that support detaching (Windows XP and 2003) use the [d] hotkey to close any open recordings, reset all breakpoints and detach from the debuggee.

Python Utilities

A number of Python utilities are provided for the instrumentation of breakpoint lists, recordings, statistics and graphs. Note, the utilities that deal with recordings expect processed recordings (see ps_process_recording). A description and sample usage of each utility follows (alphabetical order):

ps_add_register_metadata.py
Process Stalker Graph Register Metadata Importer
This utility is provided to add recorded register metadata to the specified graph. Example usage:

ps_add_register_metadata <recording-regs> <GML file> [rebase address]

ps_add_register_metadata 2360-regs.0 in.gml > out.gml

The output graph is rendered with live register data and shows the immediate 32-bit register values. Registers that are determined to be potential pointers into stack or heap space are dereferenced and displayed. If an ASCII or UNICODE string is detected then it will be displayed in place of the 4-byte aligned data.

ps_bp_filter.py
Process Stalker Breakpoint Filter
This utility is provided as a means to filter generated breakpoint lists. Breakpoint lists should be piped in via standard input and can be filtered in one of two ways:

function - Filter breakpoints that aren't on a function, ie: we don't care for block-level detail.
function list - Filter breakpoints that belong to any function as defined by a user-supplied function list.

The "in/out" modifier specifies whether or not the functions in the function list are to be filtered into the output list or out of the input list. The output of this file must be redirected into the new breakpoint list. Example usage:

ps_bp_filter <in bpl> <out bpl> <functions|<func1>:[func2]:[...] <in | out>>

ps_bp_filter input.bpl filtered.bpl functions
ps_bp_filter input.bpl filtered.bpl 00001234:deadbeef in

This utility is used frequently for fine tuning our process stalking to only interesting areas of code. Consider the following when stalking a process with a GUI. Take a recording of the target process while heavily interacting with the GUI. Use the ps_recording_to_list utility with a 'module' argument to extract the function list from the recording. Use ps_bp_filter to remove these uninteresting nodes from the breakpoint list.

ps_gde_fixup.py
Process Stalker GDE-Generated GML Fixer
Oreas GDE will sometimes save GML files that it cannot parse. When this occurs use this script to parse and re-render the broken GML description. Example usage:

ps_gde_fixup <in.gml>

ps_gde_fixup gde-generated.gml

ps_graph_cat.py
Process Stalker Graph Concatenate
This utility is provided to combine multiple function graphs into one larger and optionally interconnected graph. Nodes are clustered together by function. Example usage:

ps_graph_cat [-x <xrefs file> -b <base addr>] <file_1> [file_2] ...

ps_graph_cat.py -x dll.xrefs -b 46001000 1.gml 2.gml > out.gml

The cross-references file name and the base address are frequently used variables so it helps to add them to your environment prior to starting your work. Note that there is a significant performance decrease visualizing and interacting with large interconnected graphs.

ps_graph_highlight.py
Process Stalker Graph Highlighter
This utility is used to highlight potentially interesting graph nodes. We define interesting nodes as nodes containing any of the following:

• Instruction level loops (REP MOVS, etc.)
• Call to string manipulation API (wstrcpy, strcat, sprintf, etc.)
• Call to memory allocation and manipulation API (malloc, LocalAlloc, etc.)
• Interupt instructions (INT 80, etc)

Command line options are made available to control whether or not "all" or just "hit" interesting nodes should be highlighted as well as controlling the highlight conditions (reps, ints, str, mem, all). Example usage:

ps_graph_highlight [--nodes hit,missed,all] [--reps,--ints,--str,--mem] <GML>

ps_graph_highlight --nodes hit in.gml              > graph-highlighted.gml ps_graph_highlight --nodes all --reps --str in.gml > graph-highlighted.gml

ps_idc_gen.py
Process Stalker IDC Script Generator
This utility will parse a GML file and enumerate all function names and user comments. An IDC script will be generated that can be used to import changes made directly within the graph editor back to the IDA database. Example usage:

ps_idc_gen <file 1> [file 2] [...]

ps_idc_gen.py *.gml > import_changes.idc

ps_process_recording.py
Process Stalker Recording Post Processor
This utility is provided to process a recording to add function offsets to breakpoint addresses and optionally rebase the recording addresses. This utility must be applied to a recording before it can be processed any further. Example Usage:

ps_process_recording <recording> [base address]

ps_process_recording recording.0

The above command would generate a separate file for each encountered thread found the recording. Filenames are generated in the form: recording.0.thread_id-processed. This utility expects the appropriate .bpl breakpoint lists to exist in the current directory.

ps_recording_to_list.py
Process Stalker Recording-Function List Generator
This utility is provided as a helper to process a recording into a function list that can be later utilized in a call to the breakpoint filter utility. It optionally takes a second command line option to filter only functions belonging to a specified module. Example usage:

ps_recording_to_list <processed recording> [module]

ps_recording_to_list recording.0.processed

The optional 'module' argument must be used if the output of this utility is destined for use as an argument to ps_bp_filter.

ps_recursive_view.py
Process Stalker Recursive Graph Viewer
This utility is provided to combine multiple function graphs into one larger and interconnected graph. Given a single graph (function) this utility will recursively enumerate all inter-module function calls. Functions are clustered together. Example usage:

ps_recursive_view <GML file> <xrefs file> <base address>

ps_recursive_view graph.gml xrefs DEADBEEF > out.gml

The same caveat applies as above, large interconnected graphs tend to render very slow. If you need to visualize large graphs use a faster layout algorithm like circular as opposed to hierarchical or orthogonal.

ps_state_mapper.py
Process Stalker State Mapper
This utility constructs a "state mapping" from multiple recorder files. The visualization is function-level and clusters nodes together by recorder state. Example usage:

ps_state_mapper <xrefs file> <base address> <recorder0 recorder1 [...]>

ps_state_mapper module.xrefs DEADBEEF recorder.0 recorder.1 > out.gml

One interesting use of this visualization is for charting the maturity of a developing fuzzer. A state mapper view can be generated on a weekly basis for example and compared to see if the fuzzer is getting "deeper" into the target process space.

ps_view_recording_funcs.py
Process Stalker Recording Function Viewer
This utility is provided as a means to visualize each function that was "hit" within a recording. Every basic block for each hit function will be displayed. Basic blocks that were "hit" will be highlighted. Example usage:

ps_view_recording_funcs [-f function] [-x <xrefs file> -b <base addr>] <recording>

ps_view_recording_funcs recording.0.processed             > out.gml
ps_view_recording_funcs recording.0.processed -f deadbeef > out.gml

Specify the optional 'function' argument to view the hit-graph of only that function. Note, this utility expects the .gml function graphs to exist in the current directory.

ps_view_recording_stats.py
Process Stalker Recording Statistics
This utility will generate basic statistics on a recording including the per function hit count and the function to function transition time. Example usage:

ps_view_recording_stats <recording> [sort]

ps_view_recording_stats recording.0.processed
ps_view_recording_stats recording.0.processed sort

Specify the optional 'sort' argument to sort the output by count/time as opposed to address.

ps_view_recording_trace.py
Process Stalker Recording Trace Viewer
This utility is provided as a means to visualize the results of a recorder file in sequential order. It will "fold" repeat sequences and cluster them together with a repeat count. Example usage:

ps_view_recording_trace <recording>

ps_view_recording_trace recording.0.processed > out.gml

The generated graph, while not pretty, is useful for locating run-time logic driven loops. Note, this utility expects the .gml function graphs to exist in the current directory.

Python Instrumentation API

The bundled Python utilities will provide adequate instrumentation functionality for the majority of users. To appease the needs of "power" users an API was developed to abstract tedious tasks such as file parsing and graph rendering. The API is inline documented via Epydoc and exports the following modules/classes:

gml
• gml_node
• gml_edge
• gml_cluster
• gml_graph
ps_parsers
• bpl_parser
• recording_parser
• register_metadata_parser
• xrefs_parser
All the bundled Python utilities are built against this API and are good working examples to use as starting points for creating new utilities. More detailed documentation is available in the 'ps_api_docs' folder:

Epydoc Generated API Reference

Sample Graphs

A plethora of visualizations can be generated using a combination of filtering and graph instrumentation with the above described utilities. To help familiarize new users, the following examples and graph screenshot excerpts have been compiled. The following graph excerpt demonstrates the inclusion and highlighting of "chunked" function code:



Nodes highlighted in yellow represent chunked segments.

Chunked functions are generally seen in Microsoft optimize compiled binaries. At compile time, the Microsoft compiler will combine and extract similar and "less likely to be executed" code from various functions and store them outside the contiguous function space. Function fragments are displayed as the color maroon in the IDA navigation bar. The standard IDA grapher does not support visualization of function chunks and will simply display them as dead end red blocks. With the introduction of IDA 4.7 came chunked function support in the SDK but the grapher does not utilize it. To the security engineer, "less likely to be executed" code is interesting, especially as common security handling routines are classified as such. The process stalking suite supports chunked functions in both tracing and visualization.

The following graph excerpt demonstrates basic block level "hit" highlighting and could have been constructed for example after a run-time trace using the utilities ps_process_recording -> ps_view_recording_funcs:



Nodes highlighted in red represent run-time hits.

A common difficulty that arises when conducting an assembly level code audit is the determination of where to start looking. Most binaries contain a number of functions ranging from the low end of ~500 well into the thousands on the high end. Researchers commonly start with library routines known to process user input, such as recv(), and manually trace through potentially traversed code. Using the process stalking suite we are able to quickly and visually determine which functions are involved in processing specified inputs and equally as important which of the underlying basic blocks within that function were traversed.

Even still, the generated visualization of a process trace can be time consuming to comb through. While already much faster then the traditional method, the ps_graph_highlight utility can be applied to immediately differentiate potentially interesting blocks of code:



Nodes highlighted in purple represent potentially "interesting" nodes.

The ps_graph_highlight utility is easily extensible, customizable and controllable via command line options. The analyst can specify whether or not to highlight all potentially interesting nodes, only those that are potentially interesting and were traversed during the trace or only those that are potentially interesting and not traversed during the trace. In the above graph excerpt an inline string copy is immediately detected and highlighted. Three nodes earlier in the graph, the branch condition can be analyzed to determine how the supplied user input can be modified to alter the flow of control to reach this potentially vulnerable block.

Highlighted static disassembly allows researchers to immediately view code execution paths. To better determine the "type" of data being handled by specific nodes Process Stalker can be directed to save and render register metadata:


Register values are displayed with automated "smart" dereferencing and string detection.

The nodes from the above graph were rendered with live register data and show the immediate 32-bit register values. Registers that are determined to be potential pointers into stack or heap space are dereferenced and displayed. If an ASCII or UNICODE string is detected then it will be displayed in place of the 4-byte aligned data.

A number of layout algorithms are available to the user, each with it's own benefits. The following graph excerpt shows a number of functions displayed in a single graph with hierarchical layout applied:



Low-res excerpt from graph concatenation of multiple graphs.

The above graph may have been generated with ps_view_recording_funcs or ps_graph_cat. Again, there are a multitude of approaches leaving flexibility in the hands of the researcher. As mentioned above the red nodes represent basic blocks that were "hit" during process tracing. The excerpt is from a much larger graph containing at least twenty functions. While it is not recommended viewing such a large graph concatenation as interconnected, the researcher could do so:



Low-res excerpt from interconnected graph concatenation of the same multiple graphs from above.

The above generated graph is very complex and presented only as a demo. Further filtering could and should have been applied to reduce the function set necessary to visualize. Alternatively in cases where such large function coverage is unavoidable, function groups should be created and analyzed separately. With extremely large interconnected graphs the circular layout provides a fast visualization:



Circular layout view. Circular layouts are fast in comparison to hierarchical and orthogonal.

Though not as useful it certainly is pretty and may be useful in manually spotting and filtering frequently referenced (high edge count) yet unimportant functions and blocks. As a final example consider the following generated cluster orthogonal view:



The cluster orthogonal view is used to visualize state maps and group function nodes together.

Utilities that display multiple combined functions, such as pg_graph_cat, group nodes into clusters by their containing function. This view is useful for visually separating functions on screen while allowing the researcher to see the relationships between them. The explorer panel in GDE constructs collapsible labeled trees allowing for quick search and access to functions and their underlying nodes.