OpenRCE: Blog

Fredet v1.3

Tue, 04 Dec 2007 05:50:48 -0600

Description

Fredet means "Flexible Regular Expresion Data Extraction Tool".

The purpose of this tool is to facilitate the extraction of information contained in files using advanced regular expressions.

Fredet allows nest and combine regular expressions, define fields to facilitate the extraction information to others, generate new "checks" by simply editing a text file (config.xml), and so on.

Requirements

To work properly, the current version of "Fredet" needs:

* A Perl interpreter installed
* Curl binary for http requets

Help



fredet v1.3 - "Flexible Regular Expresion Data Extraction Tool"

(C) Copyleft 2007, created by Mandingo # http://www.yoire.com



Available checks (edit config.xml for checks management):



  comments             find comments in HTML file

  jpg                  extract jpg files to disk

  emails               find email addresses

  werrors              Typical Web Errors

  winfo                Typical Web Info

  asf                  action script dangerous functions

  title                find titles in HTML files

  dotnet               find .net error messages in HTML files

  words                extract words from files

  ips                  find local IP addresses

  paths                find local paths

  dbs                  find database error messages

  links                find links in files

  dlinks               find dynamic links in files

  comments             find comments in HTML files



Usage:



   fredet [[options] [<check>] [<format>] [<file|dir|url>] [<postData>]]



Options:



   -f                  print filename before each line

   -s                  read data from standard input (STDIN)

   -O                  display matching offsets

   -w                  sliding window mode (experimental)

   -x                  XML output mode

   -m <regexp>         match this <regexp> (<check> won't be used)

   -c <config_file>    config file (def. config.xml)

   -b <block_size>     block size used for reading files (def. 5242880 bytes)

   -o <out_file_fmt>   dump results to disk using <out_file_fmt> format string



Examples:



   fredet

   fredet emails

   fredet emails example.txt

   fredet emails name example.txt

   fredet emails '$0;$1' example.txt

   fredet words http://www.google.com

   fredet -m '\d+' example.txt

   fredet -o 'file_$count.jpg' jpg image.bin

Usage examples

Example 1, getting more information about a "check":



./fredet.pl emails



Details:



  description         find email addresses

  match[1].regexp     (?-xism:(\w+?)@([^\.]+)\.\w+(?=[<>\'\"\s]))

  match[1].display    found email address: $0

  field.email         $0

  field.name          $1

  field.domain        $2



Usage:



   fredet.pl emails <file>

   fredet.pl emails [field1] [field2] [...] <file>

   fredet.pl emails ['<format>'] <file>



Examples:



   fredet.pl emails example.txt

   fredet.pl emails name example.txt

   fredet.pl emails '$0;$1' example.txt



Where:



    * description: check name

    * match[1].regexp: this check nº1 regular expresion

    * match[1].display: format used (optional) to display this regexp results

    * email: this name will be assigned to the first field of the regular expression

    * name: name for the second field

    * domain: name for the third field

Example 2a, extracting email addresses present in "example.txt" file:



./fredet.pl emails example.txt

found email address: j0hn@foo-ar.us



./fredet.pl emails email example.txt

j0hn@foo-ar.us



./fredet.pl emails '$0;$1;$2' example.txt

j0hn@foo-ar.us;j0hn;foo-ar

Where "example.txt" file has the following lines:



Try our wargames at <!--comment-->http://www.yoire.com, and enjoy it  

invalid@email

//this is a comment

<!--172.18.1.2,j0hn@foo-ar.us c:windows

...



The definition of this "check" is stored inside "config.xml":



    <check name="emails" description="find email addresses">

                <match display="found email address: $0">(\w+?)@([^\.]+)\.\w+(?=[<>\'\"\s])</match>

                <field name="email" index="0"/>

                <field name="name" index="1"/>

                <field name="domain" index="2"/>

    </check>

Example 2b, same as before using pipes:



cat example.txt|./fredet.pl -s emails

found email address: j0hn@foo-ar.us

Example 2c, "fredet"+"find" to process multiple files at once:



find . -exec ./fredet.pl -f emails \{\} \;

./example.txt:found email address: j0hn@foo-ar.us

Note: the "-f" option will shown the name of the opened file for each result.

Example 3, using regular expresions from command line:



./fredet.pl -m '\d+' example.txt

172

18

1

2

0

192

168

1

2

Example 4a, working with URLs insead of files:



./fredet.pl words http://www.google.com

HTML

HEAD

meta

http

equiv

content

...

It's possible to send POST data after the URL; it's recommended to enclose it between quotes.

Example 4b, URL+POST:



./fredet.pl words http://www.google.com 'var1=param1&var2=param2'

html

head

meta

http

equiv

content

...

Example 5, working with large files:

From version 1.2, it's possible to work with large binary files. The default block size is 1MB, but it's possible to change it using the new parameter -b



sudo ./fredet.pl -b 524288 ips /dev/mem

192.168.0.0

10.0.0.0

172.16.0.0

172.18.1.130

172.18.1.239

172.16.176.102

172.174.35.4

192.26.10.2

10.46.174.27

...

Example 6, extracting all "jpg" files from a binary image:



./fredet.pl -o 'image_$count.jpg' jpg 8-jpeg-search.dd

image_0001.jpg

image_0002.jpg

image_0003.jpg

image_0004.jpg

image_0005.jpg

image_0006.jpg

image_0007.jpg

Config.xml

All the checks are configured inside this file. This is the basic format of a check:



<check name="check name" description="'check' description">

    <match[ modifiers="modifiers"][display="format"][output="output_file"]>regexp</match>

    [more "match" definitions]

    [<field name="field name" index="num1"/>]

    [<field name="field name" index="num2"/>]

    [more "field" definitions]

</check>

Next are real examples. This first example extracts the words (whose length >=3) of a file:



<check name="words" description="extract words from files">

    <match modifiers="i" display="$1">([a-z]{3,}?)\</match>

</check>

This example may help to extract the dynamic links inside a downloaded HTML page:



<check name="dlinks" description="find dynamic links in files">

    <match display="$1 $2">(\w+)=[\"\']?(https*://.+\?.+?=.+?(?=[,\s\"\'<>]))</match>

    <match display="txt $0">(?!=[\"\'])(https*://.+\?.+?=.+?(?=[,\s<>\"\']))</match>

</check>

Where:

* "display" is an optional parameter that allows us to specify the output format for this regexp
* "modifiers" may be added to the regular expression; for example, "i" makes the regexp "case-insensitive."

Download

Download Fredet v1.3

Fredet v1.2

Thu, 29 Nov 2007 06:02:57 -0600

Description
Fredet means "Flexible Regular Expresion Data Extraction Tool".

The purpose of this tool is to facilitate the extraction of information contained in files using advanced regular expressions.

Fredet allows nest and combine regular expressions, define fields to facilitate the extraction information to others, generate new "checks" by simply editing a text file (config.xml), and so on.

Requirements
To work properly, the current version of "Fredet" needs:

* A Unix / Linux or Windows operating system
* with a Perl interpreter and "File:: Basename" installed

Help



fredet v1.2 - "Flexible Regular Expresion Data Extraction Tool"

(C) Copyleft 2007, created by Mandingo # http://www.yoire.com



Available checks (edit config.xml for checks management):



  werrors              Typical Web Errors

  winfo                Typical Web Info

  asf                  action script dangerous functions

  title                find titles in HTML files

  dotnet               find .net error messages in HTML files

  words                extract words from files

  emails               find email addresses

  ips                  find local IP addresses

  paths                find local paths

  dbs                  find database error messages

  links                find links in files

  dlinks               find dynamic links in files

  comments             find comments in HTML file



Usage:



   fredet.pl [options] [[<check>] [<format>] <file>]]



Options:



   -f                  print filename before each line

   -s                  read data from standard input (STDIN)

   -m <regexp>         match this <regexp> (<check> won't be used)

   -b <block_size>     block size used for reading files (def. 1048576 bytes)



Examples:



   fredet.pl

   fredet.pl emails

   fredet.pl emails example.txt

   fredet.pl emails name example.txt

   fredet.pl emails '$0;$1' example.txt

   fredet.pl words http://www.google.com

   fredet.pl -m '\d+' example.txt

Usage examples

Example 1, getting more information about a "check":



./fredet.pl emails



Details:



  description         find email addresses

  match[1].regexp     (?-xism:(\w+?)@([^\.]+)\.\w+(?=[<>\'\"\s]))

  match[1].display    found email address: $0

  field.email         $0

  field.name          $1

  field.domain        $2



Usage:



   fredet.pl emails <file>

   fredet.pl emails [field1] [field2] [...] <file>

   fredet.pl emails ['<format>'] <file>



Examples:



   fredet.pl emails example.txt

   fredet.pl emails name example.txt

   fredet.pl emails '$0;$1' example.txt



Where:



    * description: check name

    * match[1].regexp: this check nº1 regular expresion

    * match[1].display: format used (optional) to display this regexp results

    * email: this name will be assigned to the first field of the regular expression

    * name: name for the second field

    * domain: name for the third field

Example 2a, extracting email addresses present in "example.txt" file:



./fredet.pl emails example.txt

found email address: j0hn@foo-ar.us



./fredet.pl emails email example.txt

j0hn@foo-ar.us



./fredet.pl emails '$0;$1;$2' example.txt

j0hn@foo-ar.us;j0hn;foo-ar

Where "example.txt" file has the following lines:



Try our wargames at <!--comment-->http://www.yoire.com, and enjoy it  

invalid@email

//this is a comment

<!--172.18.1.2,j0hn@foo-ar.us c:windows

...

The definition of this "check" is stored inside "config.xml":



    <check name="emails" description="find email addresses">

                <match display="found email address: $0">(\w+?)@([^\.]+)\.\w+(?=[<>\'\"\s])</match>

                <field name="email" index="0"/>

                <field name="name" index="1"/>

                <field name="domain" index="2"/>

    </check>

Example 2b, same as before using pipes:



cat example.txt|./fredet.pl -s emails

found email address: j0hn@foo-ar.us

Example 2c, "fredet"+"find" to process multiple files at once:



find . -exec ./fredet.pl -f emails {} ;

./example.txt:found email address: j0hn@foo-ar.us

Note: the "-f" option will shown the name of the opened file for each result.

Example 3, using regular expresions from command line:



./fredet.pl -m '\d+' example.txt

172

18

1

2

0

192

168

1

2

Example 4a, working with URLs insead of files:



./fredet.pl words http://www.google.com

HTML

HEAD

meta

http

equiv

content

...



It's possible to send POST data after the URL; it's recommended to enclose it between quotes.

Example 4b, URL+POST:



./fredet.pl words http://www.google.com 'var1=param1&var2=param2'

html

head

meta

http

equiv

content

...

Example 5, working with large files:

From version 1.2, it's possible to work with large binary files. The default block size is 1MB, but it's possible to change it using the new parameter -b



sudo ./fredet.pl -b 524288 ips /dev/mem

192.168.0.0

10.0.0.0

172.16.0.0

172.18.1.130

172.18.1.239

172.16.176.102

172.174.35.4

192.26.10.2

10.46.174.27

...

Config.xml

All the checks are configured inside this file. This is the basic format of a check:



<check name="check name" description="'check' description">

    <match[ modifiers="modifiers"][display="format"]>regexp</match>

    [more "match" definitions]

    [<field name="field name" index="num1"/>]

    [<field name="field name" index="num2"/>]

    [more "field" definitions]



</check>

Next are real examples. This first example extracts the words (whose length >=3) of a file:



<check name="words" description="extract words from files">

    <match modifiers="i" display="$1">([a-z]{3,}?)\w</match>

</check>

This example may help to extract the dynamic links inside a downloaded HTML page:



<check name="dlinks" description="find dynamic links in files">

    <match display="$1 $2">(\w+)=[\"\']?(https*://.+\?.+?=.+?(?=[,\s\"\'<>]))</match>

    <match display="txt $0">(?!=[\"\'])(https*://.+\?.+?=.+?(?=[,\s<>\"\']))</match>

</check>

Fredet v1.1

Tue, 27 Nov 2007 05:52:17 -0600

Fredet means "Flexible Regular Expresion Data Extraction Tool".

The purpose of this tool is to facilitate the extraction of information contained in files using
advanced regular expressions.

Fredet allows nest and combine regular expressions, define fields to facilitate the extraction information
to others, generate new "checks" by simply editing a text file (config.xml), and so on.

Requirements

To work properly, the current version of "Fredet" needs:

* A Unix / Linux or Windows operating system
* with a Perl interpreter and "File:: Basename" installed

Help



fredet v1.1 - "Flexible Regular Expresion Data Extraction Tool"

(C) Copyleft 2007, created by Mandingo # http://www.yoire.com



Available checks (edit config.xml for checks management):



  asf                  action script dangerous functions

  words                extract words from files

  emails               find email addresses

  ips                  find local IP addresses

  paths                find local paths

  dbs                  find database error messages

  links                find links in files

  dlinks               find dynamic links in files

  comments             find comments in HTML files



Usage:



   fredet.pl [options] [[<check>] [<format>] <file>]]



Options:



   -f                  print filename before each line

   -s                  read data from standard input (STDIN)

   -m <regexp>         match this <regexp> (<check> won't be used)



Examples:



   fredet.pl

   fredet.pl emails

   fredet.pl emails example.txt

   fredet.pl emails name example.txt

   fredet.pl emails '$0;$1' example.txt

Usage examples

* Example 1, getting more information about a "check":



./fredet.pl emails



Details:



  description         find email addresses

  match[1].regexp     (?-xism:(\w+?)@([^\.]+).\w+(?=[<>\'\"\s]))

  match[1].display    found email address: $0

  field.email         $0

  field.name          $1

  field.domain        $2



Usage:



   fredet.pl emails <file>

   fredet.pl emails [field1] [field2] [...] <file>

   fredet.pl emails ['<format>'] <file>



Examples:



   fredet.pl emails example.txt

   fredet.pl emails name example.txt

   fredet.pl emails '$0;$1' example.txt

Where:

    * description: check name
    * match[1].regexp: this check nº1 regular expresion
    * match[1].display: format used (optional) to display this regexp results
    * email: this name will be assigned to the first field of the regular expression
    * name: name for the second field
    * domain: name for the third field

* Example 2a, extracting email addresses present in "example.txt" file:



./fredet.pl emails example.txt

found email address: j0hn@foo-ar.us



./fredet.pl emails email example.txt

j0hn@foo-ar.us



./fredet.pl emails '$0;$1;$2' example.txt

j0hn@foo-ar.us;j0hn;foo-ar

Where "example.txt" file has the following lines:



Try our wargames at <!--comment-->http://www.yoire.com, and enjoy it  

invalid@email

//this is a comment

<!--172.18.1.2,j0hn@foo-ar.us c:windows

...

The definition of this "check" is stored inside "config.xml":



    <check name="emails" description="find email addresses">

                <match display="found email address: $0">(\w+?)@([^\.]+)\.w+(?=[<>\'\"\s])</match>

                <field name="email" index="0"/>

                <field name="name" index="1"/>

                <field name="domain" index="2"/>

    </check>

* Example 2b, same as before using pipes:



cat example.txt|./fredet.pl -s emails

found email address: j0hn@foo-ar.us

* Example 2c, "fredet"+"find" to process multiple files at once:



find . -exec ./fredet.pl -f emails {} ;

./example.txt:found email address: j0hn@foo-ar.us

Note: the "-f" option will shown the name of the opened file for each result.

* Example 3, using regular expresions from command line:



./fredet.pl -m 'd+' example.txt

172

18

1

2

0

192

168

1

2

Config.xml

All the checks are configured inside this file. This is the basic format of a check:



<check name="check name" description="'check' description">

    <match[ modifiers="modifiers"][display="format"]>regexp</match>

    [more "match" definitions]

    [<field name="field name" index="num1"/>]

    [<field name="field name" index="num2"/>]

    [more "field" definitions]

</check>

Next are real examples. This first example extracts the words (whose length >=3) of a file:



<check name="words" description="extract words from files">

    <match modifiers="i" display="$1">([a-z]{3,}?)\w</match>

</check>

This example may help to extract the dynamic links inside a downloaded HTML page:



<check name="dlinks" description="find dynamic links in files">

    <match display="$1 $2">(\w+)=[\"\']?(https*://.+\?.+?=.+?(?=[,\s\"\'<>]))</match>

    <match display="txt $0">(?!=[\"\'])(https*://.+\?.+?=.+?(?=[,\s<>\"\']))</match>

</check>

XSS Check v1.0

Tue, 27 Nov 2007 04:17:40 -0600

XSS_Check may allow us to detect potentially "Cross-Site Scripting" (XSS) vulnerable URLs.

Help

The current version of "XSS_Check" showns the following help:



xss_check.pl v1.0 (C) Copyleft 2007 - By Mandingo at http://www.yoire.com     



Usage: perl xss_check.pl <url|-i <file>> [postData] [OPTIONS]



Overview:



XSS_Check performs various Cross-Site Scripting (XSS) against provided URLs in order to detect this kind of vulnerability.



It's possible to pass an URL as first parameter in commandline, or a list of URLs using the option -i <file>                                           

 



Options:           



    -x server      Check XSS using a proxy server

    -c cookie      Use this cookie on connections

    -o file        Output results to XML file

    -i file        Read URLs from file

    -v	           Verbose output

    -h	           This help

Usage examples

Example 1, detecting XSS in a single URL:



./xss_check.pl http://testasp.acunetix.com/Search.asp?tfsearch=



[f] http://testasp.acunetix.com/Search.asp?tfsearch="'>XSS_HERE

[f] http://testasp.acunetix.com/Search.asp?tfsearch=<H1>XSS_HERE</H1>

Example 2, detecting XSS in a single URL, the option "-v" will shown all the requests done:



./xss_check.pl http://testasp.acunetix.com/Search.asp?tfsearch= -v



[f] http://testasp.acunetix.com/Search.asp?tfsearch="'>XSS_HERE

[-] http://testasp.acunetix.com/Search.asp 'tfsearch="'>XSS_HERE'

[-] http://testasp.acunetix.com/Search.asp?tfsearch="'%253EXSS_HERE

[-] http://testasp.acunetix.com/Search.asp 'tfsearch="'%253EXSS_HERE'

[f] http://testasp.acunetix.com/Search.asp?tfsearch=<H1>XSS_HERE</H1>

[-] http://testasp.acunetix.com/Search.asp 'tfsearch=<H1>XSS_HERE</H1>'

[-] http://testasp.acunetix.com/Search.asp?tfsearch=%253CH1%253EXSS_HERE%253C/H1%253E

[-] http://testasp.acunetix.com/Search.asp 'tfsearch=%253CH1%253EXSS_HERE%253C/H1%253E'

Example 3: Detecting XSS using an input file with URLs



./xss_check.pl -i checks.txt



[f] http://testphp.acunetix.com/guestbook.php 'text=<H1>XSS_HERE</H1>&submit=add message&name=anonymous user'

[f] http://testphp.acunetix.com/guestbook.php 'name=anonymous user<H1>XSS_HERE</H1>&text=&submit=add message'

[f] http://testphp.acunetix.com/showimage.php?file=.%2Fpictures%2F1.jpg<H1>XSS_HERE</H1>

[f] http://testphp.acunetix.com/showimage.php?file=.%2Fpictures%2F1.jpg<H1>XSS_HERE</H1>&size=160

[f] http://testphp.acunetix.com/userinfo.php 'uname="'>XSS_HERE&pass='

[f] http://testphp.acunetix.com/userinfo.php 'uname=<H1>XSS_HERE</H1>&pass='

Where "checks.txt" is a text file with the following URLs:



http://testphp.acunetix.com/artists.php?artist=1

http://testphp.acunetix.com/comment.php?aid=1

http://testphp.acunetix.com/guestbook.php 'text=&submit=add message&name=anonymous user'

http://testphp.acunetix.com/listproducts.php?cat=1

http://testphp.acunetix.com/product.php?pic=1

http://testphp.acunetix.com/redir.php?r=http://www.eclectasy.com/Fractal-Explorer/index.html

http://testphp.acunetix.com/search.php?test=query

http://testphp.acunetix.com/search.php?test=query 'gobutton=go&searchfor='

http://testphp.acunetix.com/showimage.php?file=.%2Fpictures%2F1.jpg

http://testphp.acunetix.com/showimage.php?file=.%2Fpictures%2F1.jpg&size=160

http://testphp.acunetix.com/userinfo.php 'uname=&pass=

Note: The information that should be sent using POST must be enclose between simple quotes.

Download

Download XSS_Check.pl