📚 OpenRCE is preserved as a read-only archive. Launched at RECon Montreal in 2005. Registration and posting are disabled.








Flag: Tornado! Hurricane!

Blogs >> mandingo's Blog

Created: Tuesday, December 4 2007 05:50.48 CST Modified: Tuesday, December 4 2007 06:20.39 CST
Printer Friendly ...
Fredet v1.3
Author: mandingo # Views: 4364

Description

Fredet means "Flexible Regular Expresion Data Extraction Tool".

The purpose of this tool is to facilitate the extraction of information contained in files using advanced regular expressions.

Fredet allows nest and combine regular expressions, define fields to facilitate the extraction information to others, generate new "checks" by simply editing a text file (config.xml), and so on.

Requirements

To work properly, the current version of "Fredet" needs:

    * A Perl interpreter installed
    * Curl binary for http requets

Help


fredet v1.3 - "Flexible Regular Expresion Data Extraction Tool"
(C) Copyleft 2007, created by Mandingo # http://www.yoire.com

Available checks (edit config.xml for checks management):

  comments             find comments in HTML file
  jpg                  extract jpg files to disk
  emails               find email addresses
  werrors              Typical Web Errors
  winfo                Typical Web Info
  asf                  action script dangerous functions
  title                find titles in HTML files
  dotnet               find .net error messages in HTML files
  words                extract words from files
  ips                  find local IP addresses
  paths                find local paths
  dbs                  find database error messages
  links                find links in files
  dlinks               find dynamic links in files
  comments             find comments in HTML files

Usage:

   fredet [[options] [<check>] [<format>] [<file|dir|url>] [<postData>]]

Options:

   -f                  print filename before each line
   -s                  read data from standard input (STDIN)
   -O                  display matching offsets
   -w                  sliding window mode (experimental)
   -x                  XML output mode
   -m <regexp>         match this <regexp> (<check> won't be used)
   -c <config_file>    config file (def. config.xml)
   -b <block_size>     block size used for reading files (def. 5242880 bytes)
   -o <out_file_fmt>   dump results to disk using <out_file_fmt> format string

Examples:

   fredet
   fredet emails
   fredet emails example.txt
   fredet emails name example.txt
   fredet emails '$0;$1' example.txt
   fredet words http://www.google.com
   fredet -m '\d+' example.txt
   fredet -o 'file_$count.jpg' jpg image.bin


Usage examples

Example 1, getting more information about a "check":

./fredet.pl emails

Details:

  description         find email addresses
  match[1].regexp     (?-xism:(\w+?)@([^\.]+)\.\w+(?=[<>\'\"\s]))
  match[1].display    found email address: $0
  field.email         $0
  field.name          $1
  field.domain        $2

Usage:

   fredet.pl emails <file>
   fredet.pl emails [field1] [field2] [...] <file>
   fredet.pl emails ['<format>'] <file>

Examples:

   fredet.pl emails example.txt
   fredet.pl emails name example.txt
   fredet.pl emails '$0;$1' example.txt

Where:

    * description: check name
    * match[1].regexp: this check n�1 regular expresion
    * match[1].display: format used (optional) to display this regexp results
    * email: this name will be assigned to the first field of the regular expression
    * name: name for the second field
    * domain: name for the third field


Example 2a, extracting email addresses present in "example.txt" file:

./fredet.pl emails example.txt
found email address: [email protected]

./fredet.pl emails email example.txt
[email protected]

./fredet.pl emails '$0;$1;$2' example.txt
[email protected];j0hn;foo-ar


Where "example.txt" file has the following lines:

Try our wargames at <!--comment-->http://www.yoire.com, and enjoy it  
invalid@email
//this is a comment
<!--172.18.1.2,[email protected] c:windows
...

The definition of this "check" is stored inside "config.xml":

    <check name="emails" description="find email addresses">
                <match display="found email address: $0">(\w+?)@([^\.]+)\.\w+(?=[<>\'\"\s])</match>
                <field name="email" index="0"/>
                <field name="name" index="1"/>
                <field name="domain" index="2"/>
    </check>


Example 2b, same as before using pipes:

cat example.txt|./fredet.pl -s emails
found email address: [email protected]


Example 2c, "fredet"+"find" to process multiple files at once:

find . -exec ./fredet.pl -f emails \{\} \;
./example.txt:found email address: [email protected]


Note: the "-f" option will shown the name of the opened file for each result.


Example 3, using regular expresions from command line:

./fredet.pl -m '\d+' example.txt
172
18
1
2
0
192
168
1
2


Example 4a, working with URLs insead of files:

./fredet.pl words http://www.google.com
HTML
HEAD
meta
http
equiv
content
...


It's possible to send POST data after the URL; it's recommended to enclose it between quotes.

Example 4b, URL+POST:

./fredet.pl words http://www.google.com 'var1=param1&var2=param2'
html
head
meta
http
equiv
content
...


Example 5, working with large files:

From version 1.2, it's possible to work with large binary files. The default block size is 1MB, but it's possible to change it using the new parameter -b

sudo ./fredet.pl -b 524288 ips /dev/mem
192.168.0.0
10.0.0.0
172.16.0.0
172.18.1.130
172.18.1.239
172.16.176.102
172.174.35.4
192.26.10.2
10.46.174.27
...


Example 6, extracting all "jpg" files from a binary image:

./fredet.pl -o 'image_$count.jpg' jpg 8-jpeg-search.dd
image_0001.jpg
image_0002.jpg
image_0003.jpg
image_0004.jpg
image_0005.jpg
image_0006.jpg
image_0007.jpg


Config.xml

All the checks are configured inside this file. This is the basic format of a check:

<check name="check name" description="'check' description">
    <match[ modifiers="modifiers"][display="format"][output="output_file"]>regexp</match>
    [more "match" definitions]
    [<field name="field name" index="num1"/>]
    [<field name="field name" index="num2"/>]
    [more "field" definitions]
</check>


Next are real examples. This first example extracts the words (whose length >=3) of a file:

<check name="words" description="extract words from files">
    <match modifiers="i" display="$1">([a-z]{3,}?)\</match>
</check>


This example may help to extract the dynamic links inside a downloaded HTML page:

<check name="dlinks" description="find dynamic links in files">
    <match display="$1 $2">(\w+)=[\"\']?(https*://.+\?.+?=.+?(?=[,\s\"\'<>]))</match>
    <match display="txt $0">(?!=[\"\'])(https*://.+\?.+?=.+?(?=[,\s<>\"\']))</match>
</check>


Where:

    * "display" is an optional parameter that allows us to specify the output format for this regexp
    * "modifiers" may be added to the regular expression; for example, "i" makes the regexp "case-insensitive."


Download

Download Fredet v1.3


Blog Comments
c1de0x Posted: Wednesday, December 5 2007 01:13.35 CST
Dude....

Please publish changelogs... It's difficult to tell what changes from version to version!



Add New Comment
Comment:









There are 31,328 total registered users.


Recently Created Topics
[help] Unpacking VMP...
Mar/12
Reverse Engineering ...
Jul/06
let 'IDAPython' impo...
Sep/24
set 'IDAPython' as t...
Sep/24
GuessType return une...
Sep/20
About retrieving the...
Sep/07
How to find specific...
Aug/15
How to get data depe...
Jul/07
Identify RVA data in...
May/06
Question about memor...
Dec/12


Recent Forum Posts
Finding the procedur...
rolEYder
Question about debbu...
rolEYder
Identify RVA data in...
sohlow
let 'IDAPython' impo...
sohlow
How to find specific...
hackgreti
Problem with ollydbg
sh3dow
How can I write olly...
sh3dow
New LoadMAP plugin v...
mefisto...
Intel pin in loaded ...
djnemo
OOP_RE tool available?
Bl4ckm4n


Recent Blog Entries
halsten
Mar/14
Breaking IonCUBE VM

oleavr
Oct/24
Anatomy of a code tracer

hasherezade
Sep/24
IAT Patcher - new tool for ...

oleavr
Aug/27
CryptoShark: code tracer ba...

oleavr
Jun/25
Build a debugger in 5 minutes

More ...


Recent Blog Comments
nieo on:
Mar/22
IAT Patcher - new tool for ...

djnemo on:
Nov/17
Kernel debugger vs user mod...

acel on:
Nov/14
Kernel debugger vs user mod...

pedram on:
Dec/21
frida.github.io: scriptable...

capadleman on:
Jun/19
Using NtCreateThreadEx for ...

More ...


Imagery
SoySauce Blueprint
Jun 6, 2008

[+] expand

View Gallery (11) / Submit