Difference between pages "Upcoming events" and "Carver 2.0 Planning Page"

From ForensicsWiki
(Difference between pages)
Jump to: navigation, search
(Calls For Papers)
 
(Validator Construction)
 
Line 1: Line 1:
<b>PLEASE READ BEFORE YOU EDIT THE LISTS BELOW</b><br>
+
This page is for planning Carver 2.0.
Events should be posted in the correct section, and in date order.  An event should NEVER be listed in more than one section (i.e. Ongoing/Continuous events should not be listed in Scheduled Training).  When events begin the same day, events of a longer length should be listed first.  New postings of events with the same date(s) as other events should be added after events already in the list. If a provider offers the same event at several locations simultaneously, the listing should have a single (ONE) entry in the list with the date(s) and ALL locations for the event. Please use three-letter month abbreviations (i.e. Sep, NOT Sept. or September), use two digit dates (i.e. Jan 01 NOT Jan 1), and use date ranges rather than listing every date during an event(i.e. Jan 02-05, NOT Jan 02, 03, 04, 05).<br>
+
<i>Some events may be <u>limited</u> to <b>Law Enforcement Only</b> or to a specific audience. Such restrictions should be noted when known.</i>
+
  
This is a BY DATE listing of upcoming events relevant to [[digital forensics]].  It is not an all inclusive list, but includes most well-known activities. Some events may duplicate events on the generic [[conferences]] page, but entries in this list have specific dates and locations for the upcoming event.
+
Please, do not delete text (ideas) here. Use something like this:
  
This listing is divided into four sections (described as follows):<br>
+
<pre>
<ol><li><b><u>Calls For Papers</u></b> - Calls for papers for either Journals or for Conferences, relevant to Digital Forensics (Name, Closing Date, URL)</li><br>
+
<s>bad idea</s>
<li><b><u>Conferences</u></b> - Conferences relevant for Digital Forensics (Name, Date, Location, URL)</li><br>
+
:: good idea
<li><b><u>On-Going / Continuous Training</u></b> - Training opportunities that are either always available online/distance learning format (start anytime) or that are offered the same time every month (Name, date-if applicable, URL)</li><br>
+
</pre>
<li><b><u>[[Scheduled Training Courses]]</u></b> - Training Classes/Courses that are scheduled for specific dates/locations.  This would include online (or distance learning format) courses which begin on specific dates, instead of the "start anytime" courses listed in the previous section. (Provider, URL) (''note: this has been moved to its own page.'')<br></li></ol>
+
  
== Calls For Papers ==
+
This will look like:
Please help us keep this up-to-date with deadlines for upcoming conferences that would be appropriate for forensic research.
+
  
{| border="0" cellpadding="2" cellspacing="2" align="top"
+
<s>bad idea</s>
|- style="background:#bfbfbf; font-weight: bold"
+
:: good idea
! width="30%|Title
+
! width="15%"|Due Date
+
! width="15%"|Notification Date
+
! width="40%"|Website
+
|-
+
|USENIX Security Symposium 2010
+
|Feb 05, 2010
+
|Jul 05, 2010
+
|http://www.usenix.org/events/sec10/cfp/
+
|-
+
|7th International Symposium on Risk Management and Cyber-Informatics: RMCI 2010
+
|Feb 10, 2010
+
|Mar 03, 2010
+
|http://www.iiis2010.org/wmsci/Contents/CallForPapers-RMCI-2010.pdf
+
|-
+
|Thirtieth Annual International Cryptology Conference
+
|Feb 18, 2010
+
|Apr 30, 2010
+
|http://www.iacr.org/conferences/crypto2010/cfp.php
+
|-
+
|2010 Conference on Digital Forensics, Security and Law
+
|Feb 19, 2010
+
|
+
|http://www.digitalforensics-conference.org/callforpapers.htm
+
|-
+
|Digital Forensic Research Workshop (DFRWS) 2010
+
|Feb 28, 2010
+
|Apr 05, 2010
+
|http://dfrws.org/2010/cfp.shtml
+
|-
+
|Blackhat Europe 2010
+
|Mar 01, 2010
+
|
+
|http://blackhat.com/html/bh-eu-10/registration/bh-eu-10-cfp.html
+
|-
+
|20th Virus Bulletin International Conference
+
|Mar 05, 2010
+
|
+
|http://www.virusbtn.com/conference/vb2010/call/index
+
|-
+
|European Symposium on Research in Computer Security
+
|Apr 01, 2010
+
|Jun 10, 2010
+
|http://www.esorics2010.org/index.php?option=com_content&view=article&id=1&Itemid=3
+
|-
+
|ACM Computer and Communications Security Conference
+
|Apr 17, 2010
+
|Jun 21, 2010
+
|http://www.sigsac.org/ccs/CCS2010/cfp.shtml
+
|-
+
|2010 IEEE International Conference on Technologies for Homeland Security
+
|Apr 24, 2010
+
|
+
|http://ieee-hst.org/
+
|-
+
|2nd International ICST Conference on Digital Forensics & Cyber Crime (ICDF2C)
+
|May 01, 2010
+
|Jun 15, 2010
+
|http://www.d-forensics.org/callforpapers.shtml
+
|-
+
|2nd International Workshop on Security in Cloud Computing (SCC'2010)
+
|May 01, 2010
+
|Jun 07, 2010
+
|http://bingweb.binghamton.edu/~ychen/SCC2010.htm
+
|-
+
|}
+
  
See also [http://www.wikicfp.com/cfp/servlet/tool.search?q=forensics WikiCFP 'Forensics']
+
= License =
  
== Conferences ==
+
BSD-3.
{| border="0" cellpadding="2" cellspacing="2" align="top"
+
|- style="background:#bfbfbf; font-weight: bold"
+
! width="40%"|Title
+
! width="20%"|Date/Location
+
! width="40%"|Website
+
|-
+
|DoD Cyber Crime Conference
+
|Jan 22-29<br>St. Louis, MO
+
|http://www.dodcybercrime.com/10CC/
+
|-
+
|ShmooCon VI
+
|Feb 05-07<br>Washington, DC
+
|http://www.shmoocon.org
+
|-
+
|International Conference on Technical and Legal Aspects of the e-Society
+
|Feb 10-15<br>St. Maarten, Netherlands Antilles
+
|http://www.iaria.org/conferences2010/CYBERLAWS10.html
+
|-
+
|Third International Workshop on Digital Forensics
+
|Feb 15-18<br>Krakow, Poland
+
|http://www.ares-conference.eu/conf/index.php/workshops/wsdf
+
|-
+
|American Academy of Forensic Sciences Annual Meeting
+
|Feb. 22-27<br>Seattle, WA
+
|http://www.aafs.org/default.asp?section_id=meetings&page_id=aafs_annual_meeting
+
|-
+
|17th Network and IT Security Conference
+
|Feb 38-Mar 03<br>San Diego, CA
+
|http://www.isoc.org/isoc/conferences/ndss/10/
+
|-
+
|RSA Conference 2010
+
|Mar 01-05<br>San Francisco, CA
+
|http://www.rsaconference.com/2010/usa/index.htm
+
|-
+
|CanSecWest 2010
+
|Mar 22-26<br>Vancouver, British Columbia, Canada
+
|http://cansecwest.com/index.html
+
|-
+
|Blackhat Europe 2010
+
|Apr 12-15<br>Barcelona, Spain
+
|http://blackhat.com/html/bh-eu-10/bh-eu-10-home.html
+
|-
+
|31st IEEE Symposium on Security and Privacy
+
|May 16-19<br>Oakland, CA
+
|http://oakland31.cs.virginia.edu/
+
|-
+
|AusCERT Asia Pacific Information Security Conference
+
|May 16-21<br>Kenmore Hills, Queensland, Australia
+
|http://conference.auscert.org.au/conf2010/index.html
+
|-
+
|Conference on Digital Forensics, Security and Law 2010
+
|May 19-21<br>St. Paul, MN
+
|http://www.digitalforensics-conference.org/index.htm
+
|-
+
|Blackhat Abu Dhabi 2010
+
|May 30-Jun 02<br>Abu Dhabi, UAE
+
|http://blackhat.com/html/events.html
+
|-
+
|Techno-Security 2010
+
|Jun 06-09<br>Myrtle Beach, SC
+
|http://www.thetrainingco.com/html/Security_Conference_2010.html
+
|-
+
|7th International Symposium on Risk Management and Cyber-Informatics
+
|Jun 29-Jul 02<br>Orlando, FL
+
|http://www.2010iiisconferences.org/RMCI
+
|-
+
|Blackhat USA 2010
+
|Jul 24-29<br>Las Vegas, NV
+
|http://blackhat.com/html/events.html
+
|-
+
|Digital Forensic Research Workshop (DFRWS) 2010
+
|Aug 02-04<br>Portland, OR
+
|http://dfrws.org/2010/
+
|-
+
|19th USENIX Security Symposium
+
|Aug 11-13(br>Washington, DC
+
|http://www.usenix.org/events/sec10/
+
|-
+
|30th International Cryptology Conference
+
|Aug 15-19<Santa Barbara, CA
+
|http://www.iacr.org/conferences/crypto2010/
+
|-
+
|2nd International Workshop on Security in Cloud Computing (SCC'2010)
+
|Sep 13-16<br>San Diego, CA
+
|http://bingweb.binghamton.edu/~ychen/SCC2010.htm
+
|-
+
|2010 HTCIA International Training Conference & Exposition
+
|Sep 20-22<br>Atlanta, GA
+
|http://www.htciaconference.org/
+
|-
+
|VB2010 Fighting malware and spam
+
|Sep 29-Oct 01<br>Vancouver, BC, Canada
+
|http://www.virusbtn.com/conference/vb2010/
+
|-
+
|17th ACM Computer and Communications Security Conference
+
|Oct 04-08<br>Chicago, IL
+
|http://www.sigsac.org/ccs/CCS2010/
+
|-
+
|2nd International ICST Conference on Digital Forensics & Cyber Crime (ICDF2C)
+
|Oct 04-06<br>Abu Dhabi, UAE
+
|http://www.d-forensics.org/
+
|-
+
|Techno Forensics 2010
+
|Oct 25-26<br>Gaithersburg, MD
+
|http://www.techsec.com/html/TechnoForensics2010.html
+
|-
+
|2010 IEEE International Conference on Technologies for Homeland Security
+
|Nov 08-10<br>Waltham, MA
+
|http://ieee-hst.org/
+
|-
+
|IFIP Working Group 11.9 - Digital Forensics
+
|January 2011<br>Unknown
+
|http://www.ifip119.org/Conferences/
+
|-
+
|}
+
  
== On-going / Continuous Training ==
+
= OS =
{| border="0" cellpadding="2" cellspacing="2" align="top"
+
|- style="background:#bfbfbf; font-weight: bold"
+
! width="40%"|Title
+
! width="20%"|Date/Location
+
! width="40%"|Website
+
|-
+
|- style="background:pink;align:left"
+
! DISTANCE LEARNING
+
|-
+
|Basic Computer Examiner Course - Computer Forensic Training Online
+
|Distance Learning Format
+
|http://www.cftco.com
+
|-
+
|Linux Data Forensics Training
+
|Distance Learning Format
+
|http://www.crazytrain.com/training.html
+
|-
+
|SANS On-Demand Training
+
|Distance Learning Format
+
|http://www.sans.org/ondemand/?portal=69456f95660ade45be29c00b0c14aea1
+
|-
+
|Champlain College - CCE Course
+
|Online / Distance Learning Format
+
|http://extra.champlain.edu/cps/wdc/alliances/cce/landing/
+
|-
+
|Las Positas College
+
|Online Computer Forensics Courses
+
|http://www.laspositascollege.edu
+
|-
+
|- style="background:pink;align:left"
+
!RECURRING TRAINING
+
|-
+
|MaresWare Suite Training
+
|First full week every month<br>Atlanta, GA
+
|http://www.maresware.com/maresware/training/maresware.htm
+
|-
+
|Evidence Recovery for Windows Vista&trade;
+
|First full week every month<br>Brunswick, GA
+
|http://www.internetcrimes.net
+
|-
+
|Evidence Recovery for Windows Server&reg; 2003 R2
+
|Second full week every month<br>Brunswick, GA
+
|http://www.internetcrimes.net
+
|-
+
|Evidence Recovery for the Windows XP&trade; operating system
+
|Third full week every month<br>Brunswick, GA
+
|http://www.internetcrimes.net
+
|-
+
|Computer Forensics Training and CCE&trade; Testing for Litigation Support Professionals
+
|Third weekend of every month(Fri-Mon)<br>Dallas, TX
+
|http://www.md5group.com
+
|-
+
|}
+
  
==See Also==
+
Linux/FreeBSD/MacOS
* [[Scheduled Training Courses]]
+
: (shouldn't this just match what the underlying afflib & sleuthkit cover? [[User:RB|RB]])
==References==
+
:: Yes, but you need to test and validate on each. Question: Do we want to support windows? [[User:Simsong|Simsong]] 21:09, 30 October 2008 (UTC)
* [http://faculty.cs.tamu.edu/guofei/sec_conf_stat.htm Computer Security Conference Ranking and Statistic]
+
:: [[User:Joachim Metz|Joachim]] I think we would do wise to design with windows support from the start this will improve the platform independence from the start
* [http://www.kdnuggets.com/meetings/ Meetings and Conferences in Data Mining and Discovery]
+
 
* http://www.conferencealerts.com/data.htm Data Mining Conferences World-Wide]
+
= Requirements =
 +
 
 +
* [[User:Joachim Metz|Joachim]] A name for the tooling I propose coldcut
 +
 
 +
[[User:Joachim Metz|Joachim]] Could we do a MoSCoW evaluation of these.
 +
 
 +
* AFF and EWF file images supported from scratch. ([[User:Joachim Metz|Joachim]] I would like to have raw/split raw and device access as well)
 +
* [[User:Joachim Metz|Joachim]] volume/partition aware layer (what about carving unpartioned space)
 +
* File system aware layer.  
 +
** By default, files are not carved. (clarify: only identified? [[User:RB|RB]]; I guess that it operates like [[Selective file dumper]] [[User:.FUF|.FUF]] 07:00, 29 October 2008 (UTC))
 +
* Plug-in architecture for identification/validation.
 +
** [[User:Joachim Metz|Joachim]] support for multiple types of validators
 +
*** dedicated validator
 +
*** validator based on file library (i.e. we could specify/implement a file structure for these)
 +
*** configuration based validator (Can handle config files,like Revit07, to enter different file formats used by the carver.)
 +
* Ship with validators for:
 +
[[User:Joachim Metz|Joachim]] I think we should distinguish between file format validators and content validators
 +
** JPEG
 +
** PNG
 +
** GIF
 +
** MSOLE
 +
** ZIP
 +
** TAR (gz/bz2)
 +
 
 +
[[User:Joachim Metz|Joachim]] For a production carver we need at least the following formats
 +
** Grapical Images
 +
*** JPEG (the 3 different types with JFIF/EXIF support)
 +
*** PNG
 +
*** GIF
 +
*** BMP
 +
*** TIFF
 +
** Office documents
 +
*** OLE2 (Word/Excell content support)
 +
*** PDF
 +
*** Open Office/Office 2007 (ZIP+XML)
 +
** Archive files
 +
*** ZIP
 +
** E-mail files
 +
*** PFF (PST/OST)
 +
*** MBOX (text based format, base64 content support)
 +
** Audio/Video files
 +
*** MPEG
 +
*** MP2/MP3
 +
*** AVI
 +
*** ASF/WMV
 +
*** QuickTime
 +
** Printer spool files
 +
*** EMF (if I remember correctly)
 +
** Internet history files
 +
*** index.dat
 +
*** firefox (sqllite 3)
 +
** Other files
 +
*** thumbs.db
 +
 
 +
* Simple fragment recovery carving using gap carving.
 +
** [[User:Joachim Metz|Joachim]] have hook in for more advanced fragment recovery?
 +
* Recovering of individual ZIP sections and JPEG icons that are not sector aligned.
 +
** [[User:Joachim Metz|Joachim]] I would propose a generic fragment detection and recovery
 +
* Autonomous operation (some mode of operation should be completely non-interactive, requiring no human intervention to complete [[User:RB|RB]])
 +
** [[User:Joachim Metz|Joachim]] as much as possible, but allow to be overwritten by user
 +
* Tested on 500GB-sized images. Should be able to carve a 500GB image in roughly 50% longer than it takes to read the image.
 +
** Perhaps allocate a percentage budget per-validator (i.e. each validator adds N% to the carving time)
 +
** [[User:Joachim Metz|Joachim]] have multiple carving phases for precision/speed trade off?
 +
* Parallelizable
 +
** [[User:Joachim Metz|Joachim]] tunable for different architectures
 +
* Configuration:
 +
** Capability to parse some existing carvers' configuration files, either on-the-fly or as a one-way converter.
 +
** Disengage internal configuration structure from configuration files, create parsers that present the expected structure
 +
** [[User:Joachim Metz|Joachim]] The validator should deal with the file structure the carving algorithm should not know anything about the file structure (as in revit07 design)
 +
**  Either extend Scalpel/Foremost syntaxes for extended features or use a tertiary syntax ([[User:Joachim Metz|Joachim]] I would prefer a derivative of the revit07 configuration syntax which already has encountered some problems of dealing with defining file structure in a configuration file)
 +
* Can output audit.txt file.
 +
* [[User:Joachim Metz|Joachim]] Can output database with offset analysis values i.e. for visualization tooling
 +
* [[User:Joachim Metz|Joachim]] Can output debug log for debugging the algorithm/validation
 +
* Easy integration into ascription software.
 +
** [[User:Joachim Metz|Joachim]] I'm no native speaker what do you mean with "ascription software"?
 +
 
 +
= Ideas =
 +
* Use as much TSK if possible. Don't carry your own FS implementation the way photorec does.
 +
** [[User:Joachim Metz|Joachim]] using TSK as much as possible would not allow to add your own file system support (i.e. mobile phones, memory structures, cap files)
 +
I would propose wrapping TSK and using it as much as possible but allow to integrate own FS implementations.
 +
* Extracting/carving data from [[Thumbs.db]]? I've used [[foremost]] for it with some success. [[Vinetto]] has some critical bugs :( [[User:.FUF|.FUF]] 19:18, 28 October 2008 (UTC)
 +
* Carving data structures. For example, extract all TCP headers from image by defining TCP header structure and some fields (e.g. source port > 1024, dest port = 80). This will extract all data matching the pattern and write a file with other fields. Another example is carving INFO2 structures and URL activity records from index.dat [[User:.FUF|.FUF]] 20:51, 28 October 2008 (UTC)
 +
** This has the opportunity to be extended to the concept of "point at blob FOO and interpret it as BAR"
 +
 
 +
.FUF added:
 +
The main idea is to allow users to define structures, for example (in pascal-like form):
 +
 
 +
<pre>
 +
Field1: Byte = 123;
 +
SomeTextLength: DWORD;
 +
SomeText: string[SomeTextLength];
 +
Field4: Char = 'r';
 +
...
 +
</pre>
 +
 
 +
This will produce something like this:
 +
<pre>
 +
Field1 = 123
 +
SomeTextLength = 5
 +
SomeText = 'abcd1'
 +
Field4 = 'r'
 +
</pre>
 +
 
 +
(In text or raw forms.)
 +
 
 +
Opinions?
 +
 
 +
Opinion: Simple pattern identification like that may not suffice, I think Simson's original intent was not only to identify but to allow for validation routines (plugins, as the original wording was). As such, the format syntax would need to implement a large chunk of some programming language in order to be sufficiently flexible. [[User:RB|RB]]
 +
 
 +
=File System Awareness =
 +
==Background: Why be File System Aware?==
 +
Advantages of being FS aware:
 +
* You can pick up sector allocation sizes ([[User:Joachim Metz|Joachim]] do you mean file system block sizes?)
 +
* Some file systems may store things off sector boundaries. (ReiserFS with tail packing)
 +
* Increasingly file systems have compression (NTFS compression)
 +
* Carve just the sectors that are not in allocated files.
 +
 
 +
==Tasks that would be required==
 +
 
 +
==Discussion==
 +
:: As noted above, TSK should be utilized as much as possible, particularly the filesystem-aware portion.  If we want to identify filesystems outside of its supported set, it would be more worth our time to work on implementing them there than in the carver itself.  [[User:RB|RB]]
 +
 
 +
[[User:Joachim Metz|Joachim]] I would like to have the carver (recovery tool) also do recovery using file allocation data or remainders of file allocation data.
 +
 
 +
:::: I guess this tool operates like [[Selective file dumper]] and can recover files in both ways (or not?). Recovering files by using carving can recover files in situations where sleuthkit does nothing (e.g. file on NTFS was deleted using ntfs-3g, or filesystem was destroyed or just unknown). And we should build the list of filesystems supported by carver, not by TSK. [[User:.FUF|.FUF]] 07:08, 29 October 2008 (UTC)
 +
 
 +
:: This tool is still in the early planning stages (requirements discovery), hence few operational details (like precise modes of operation) have been fleshed out - those will and should come later.  The justification for strictly using TSK for the filesystem-sensitive approach is simple: TSK has good filesystem APIs, and it would be foolish to create yet another standalone, incompatible implementation of filesystem(foo) when time would be better spent improving those in TSK, aiding other methods of analysis as well.  This is the same reason individuals that have implemented several other carvers are participating: de-duplication of effort.  [[User:RB|RB]]
 +
 
 +
[[User:Joachim Metz|Joachim]]
 +
I would go as far to ask you all to look beyond the carver as a tool and look from the perspective of the carver as part of the forensic investigation process. In my eyes certain information needed/acquired by the carver could be also very useful investigative information i.e. what part of a hard disk contains empty sectors.
 +
 
 +
[[User:Joachim Metz|Joachim]]
 +
I'm missing a part on the page about the carving challenges (scenarios)
 +
* normal file (file structure, loose text based structure (more a content structure?))
 +
* fragmented file (the file entirely exist)
 +
* a file fragment (the file does not entirely exist)
 +
* intertwined file
 +
* encapsulated file (MPEG/network capture)
 +
* embedded file (JPEG thumbnail)
 +
 
 +
=Validator Construction=
 +
Options:
 +
* Write validators in C/C++
 +
* Have a scripting language for writing them (python? Perl?) our own?
 +
** [[User:Joachim Metz|Joachim]] use easy to embed programming languages i.e. Phyton or Lua
 +
* Use existing programs (libjpeg?) as plug-in validators?
 +
** [[User:Joachim Metz|Joachim]] define a file structure api for this
 +
 
 +
=Existing Code that we have=
 +
 
 +
[[User:Joachim Metz|Joachim]]
 +
* DFRWS2006/2007 carving challenge results
 +
* photorec
 +
* revit06 and revit07
 +
* s3/scarve
 +
 
 +
=Implementation Timeline=
 +
# gather the available resources/ideas/wishes/needs etc. (I guess we're in this phase)
 +
# start discussing a high level design (in terms of algorithm, facilities, information needed)
 +
## input formats facility
 +
## partition/volume facility
 +
## file system facility
 +
## file format facility
 +
## content facility
 +
## how to deal with fragment detection (do the validators allow for fragment detection?)
 +
## how to deal with recombination of fragments
 +
## do we want multiple carving phases in light of speed/precision tradeoffs
 +
# start detailing parts of the design
 +
## Discuss options for a grammar driven validator?
 +
## Hard-coded plug-ins?
 +
## Which exsisting code can we use?
 +
# start building/assembling parts of the tooling for a prototype
 +
## Implement simple file carving with validation.
 +
## Implement gap carving
 +
# Initial Release
 +
# Implement the ''threaded carving'' that [[User:.FUF|.FUF]] is describing above.

Revision as of 04:16, 31 October 2008

This page is for planning Carver 2.0.

Please, do not delete text (ideas) here. Use something like this:

<s>bad idea</s>
:: good idea

This will look like:

bad idea

good idea

License

BSD-3.

OS

Linux/FreeBSD/MacOS

(shouldn't this just match what the underlying afflib & sleuthkit cover? RB)
Yes, but you need to test and validate on each. Question: Do we want to support windows? Simsong 21:09, 30 October 2008 (UTC)
Joachim I think we would do wise to design with windows support from the start this will improve the platform independence from the start

Requirements

  • Joachim A name for the tooling I propose coldcut

Joachim Could we do a MoSCoW evaluation of these.

  • AFF and EWF file images supported from scratch. (Joachim I would like to have raw/split raw and device access as well)
  • Joachim volume/partition aware layer (what about carving unpartioned space)
  • File system aware layer.
    • By default, files are not carved. (clarify: only identified? RB; I guess that it operates like Selective file dumper .FUF 07:00, 29 October 2008 (UTC))
  • Plug-in architecture for identification/validation.
    • Joachim support for multiple types of validators
      • dedicated validator
      • validator based on file library (i.e. we could specify/implement a file structure for these)
      • configuration based validator (Can handle config files,like Revit07, to enter different file formats used by the carver.)
  • Ship with validators for:

Joachim I think we should distinguish between file format validators and content validators

    • JPEG
    • PNG
    • GIF
    • MSOLE
    • ZIP
    • TAR (gz/bz2)

Joachim For a production carver we need at least the following formats

    • Grapical Images
      • JPEG (the 3 different types with JFIF/EXIF support)
      • PNG
      • GIF
      • BMP
      • TIFF
    • Office documents
      • OLE2 (Word/Excell content support)
      • PDF
      • Open Office/Office 2007 (ZIP+XML)
    • Archive files
      • ZIP
    • E-mail files
      • PFF (PST/OST)
      • MBOX (text based format, base64 content support)
    • Audio/Video files
      • MPEG
      • MP2/MP3
      • AVI
      • ASF/WMV
      • QuickTime
    • Printer spool files
      • EMF (if I remember correctly)
    • Internet history files
      • index.dat
      • firefox (sqllite 3)
    • Other files
      • thumbs.db
  • Simple fragment recovery carving using gap carving.
    • Joachim have hook in for more advanced fragment recovery?
  • Recovering of individual ZIP sections and JPEG icons that are not sector aligned.
    • Joachim I would propose a generic fragment detection and recovery
  • Autonomous operation (some mode of operation should be completely non-interactive, requiring no human intervention to complete RB)
    • Joachim as much as possible, but allow to be overwritten by user
  • Tested on 500GB-sized images. Should be able to carve a 500GB image in roughly 50% longer than it takes to read the image.
    • Perhaps allocate a percentage budget per-validator (i.e. each validator adds N% to the carving time)
    • Joachim have multiple carving phases for precision/speed trade off?
  • Parallelizable
    • Joachim tunable for different architectures
  • Configuration:
    • Capability to parse some existing carvers' configuration files, either on-the-fly or as a one-way converter.
    • Disengage internal configuration structure from configuration files, create parsers that present the expected structure
    • Joachim The validator should deal with the file structure the carving algorithm should not know anything about the file structure (as in revit07 design)
    • Either extend Scalpel/Foremost syntaxes for extended features or use a tertiary syntax (Joachim I would prefer a derivative of the revit07 configuration syntax which already has encountered some problems of dealing with defining file structure in a configuration file)
  • Can output audit.txt file.
  • Joachim Can output database with offset analysis values i.e. for visualization tooling
  • Joachim Can output debug log for debugging the algorithm/validation
  • Easy integration into ascription software.
    • Joachim I'm no native speaker what do you mean with "ascription software"?

Ideas

  • Use as much TSK if possible. Don't carry your own FS implementation the way photorec does.
    • Joachim using TSK as much as possible would not allow to add your own file system support (i.e. mobile phones, memory structures, cap files)

I would propose wrapping TSK and using it as much as possible but allow to integrate own FS implementations.

  • Extracting/carving data from Thumbs.db? I've used foremost for it with some success. Vinetto has some critical bugs :( .FUF 19:18, 28 October 2008 (UTC)
  • Carving data structures. For example, extract all TCP headers from image by defining TCP header structure and some fields (e.g. source port > 1024, dest port = 80). This will extract all data matching the pattern and write a file with other fields. Another example is carving INFO2 structures and URL activity records from index.dat .FUF 20:51, 28 October 2008 (UTC)
    • This has the opportunity to be extended to the concept of "point at blob FOO and interpret it as BAR"

.FUF added: The main idea is to allow users to define structures, for example (in pascal-like form):

Field1: Byte = 123;
SomeTextLength: DWORD;
SomeText: string[SomeTextLength];
Field4: Char = 'r';
...

This will produce something like this:

Field1 = 123
SomeTextLength = 5
SomeText = 'abcd1'
Field4 = 'r'

(In text or raw forms.)

Opinions?

Opinion: Simple pattern identification like that may not suffice, I think Simson's original intent was not only to identify but to allow for validation routines (plugins, as the original wording was). As such, the format syntax would need to implement a large chunk of some programming language in order to be sufficiently flexible. RB

File System Awareness

Background: Why be File System Aware?

Advantages of being FS aware:

  • You can pick up sector allocation sizes (Joachim do you mean file system block sizes?)
  • Some file systems may store things off sector boundaries. (ReiserFS with tail packing)
  • Increasingly file systems have compression (NTFS compression)
  • Carve just the sectors that are not in allocated files.

Tasks that would be required

Discussion

As noted above, TSK should be utilized as much as possible, particularly the filesystem-aware portion. If we want to identify filesystems outside of its supported set, it would be more worth our time to work on implementing them there than in the carver itself. RB

Joachim I would like to have the carver (recovery tool) also do recovery using file allocation data or remainders of file allocation data.

I guess this tool operates like Selective file dumper and can recover files in both ways (or not?). Recovering files by using carving can recover files in situations where sleuthkit does nothing (e.g. file on NTFS was deleted using ntfs-3g, or filesystem was destroyed or just unknown). And we should build the list of filesystems supported by carver, not by TSK. .FUF 07:08, 29 October 2008 (UTC)
This tool is still in the early planning stages (requirements discovery), hence few operational details (like precise modes of operation) have been fleshed out - those will and should come later. The justification for strictly using TSK for the filesystem-sensitive approach is simple: TSK has good filesystem APIs, and it would be foolish to create yet another standalone, incompatible implementation of filesystem(foo) when time would be better spent improving those in TSK, aiding other methods of analysis as well. This is the same reason individuals that have implemented several other carvers are participating: de-duplication of effort. RB

Joachim I would go as far to ask you all to look beyond the carver as a tool and look from the perspective of the carver as part of the forensic investigation process. In my eyes certain information needed/acquired by the carver could be also very useful investigative information i.e. what part of a hard disk contains empty sectors.

Joachim I'm missing a part on the page about the carving challenges (scenarios)

  • normal file (file structure, loose text based structure (more a content structure?))
  • fragmented file (the file entirely exist)
  • a file fragment (the file does not entirely exist)
  • intertwined file
  • encapsulated file (MPEG/network capture)
  • embedded file (JPEG thumbnail)

Validator Construction

Options:

  • Write validators in C/C++
  • Have a scripting language for writing them (python? Perl?) our own?
    • Joachim use easy to embed programming languages i.e. Phyton or Lua
  • Use existing programs (libjpeg?) as plug-in validators?
    • Joachim define a file structure api for this

Existing Code that we have

Joachim

  • DFRWS2006/2007 carving challenge results
  • photorec
  • revit06 and revit07
  • s3/scarve

Implementation Timeline

  1. gather the available resources/ideas/wishes/needs etc. (I guess we're in this phase)
  2. start discussing a high level design (in terms of algorithm, facilities, information needed)
    1. input formats facility
    2. partition/volume facility
    3. file system facility
    4. file format facility
    5. content facility
    6. how to deal with fragment detection (do the validators allow for fragment detection?)
    7. how to deal with recombination of fragments
    8. do we want multiple carving phases in light of speed/precision tradeoffs
  3. start detailing parts of the design
    1. Discuss options for a grammar driven validator?
    2. Hard-coded plug-ins?
    3. Which exsisting code can we use?
  4. start building/assembling parts of the tooling for a prototype
    1. Implement simple file carving with validation.
    2. Implement gap carving
  5. Initial Release
  6. Implement the threaded carving that .FUF is describing above.