Difference between pages "Upcoming events" and "Carver 2.0 Planning Page"

From ForensicsWiki
(Difference between pages)
Jump to: navigation, search
(Calls For Papers)
 
(Ideas)
 
Line 1: Line 1:
Here is a BY DATE listing of '''upcoming conferences and training events''' that pertain to [[digital forensics]]. Some of these duplicate the generic [[conferences]], but have specific dates/locations for the upcoming conference/training event.
+
This page is for planning Carver 2.0.
  
<b> The Conference and Training List is provided by the American Academy of Forensic Sciences (AAFS) Digital and Multi-media Listserv</b>
+
Please, do not delete text (ideas) here. Use something like this:
<i> (Subscribe by sending an email to listserv@lists.mitre.org with message body containing SUBSCRIBE AAFS-DIGITAL-MULTIMEDIA-LIST)</i>
+
<b> Any requests for additions, deletions or corrections to this list should be sent by email to David Baker <i>(bakerd AT mitre.org)</i>. </b>
+
  
== Calls For Papers ==
+
<pre>
{| border="0" cellpadding="2" cellspacing="2" align="top"
+
<s>bad idea</s>
|- style="background:#bfbfbf; font-weight: bold"
+
:: good idea
! Title
+
</pre>
! Due Date
+
! Website
+
|-
+
|PacSec Applied Security Conference
+
|Jul 27, 2007
+
|http://www.pacsec.jp/speakers.html
+
|-
+
|Journal of Digital Forensic Practice
+
|Jul 31, 2007
+
|http://www.tandf.co.uk/journals/titles/15567281.asp
+
|-
+
|American Academy of Forensic Sciences 2008 Annual Meeting
+
|Aug 01, 2007
+
|http://www.aafs.org/abstracts/your_online_presentation_submiss.htm
+
|-
+
|Digital Forensic Forum Prague 2007
+
|Aug 31, 2007
+
|http://www.dff-prague.com/News/article/sid=17.html
+
|-
+
|Internet Investigations Training Program (IITP) Sep 24-28, FLETC
+
|Glynco, GA
+
|http://www.fletc.gov/training/programs/computer-financial-investigations/technology-investigation/internet-investigations-training-program-iitp/
+
|Limited to Law Enforcement
+
  
== Conferences ==
+
This will look like:
{| border="0" cellpadding="2" cellspacing="2" align="top"
+
|- style="background:#bfbfbf; font-weight: bold"
+
! Title
+
! Date/Location
+
! Website
+
|-
+
|SANSFire 2007
+
|Jul 25-Aug 03, Washington, DC
+
|http://www.sans.org/sansfire07/index.php?portal=e1bc55a0898bf3774408826c9368ae40
+
|-
+
|BlackHat Briefings
+
|Jul 28-Aug 02, Las Vegas, NV
+
|http://www.blackhat.com/html/bh-link/briefings.html
+
|-
+
|DefCon
+
|Aug 03-05, Las Vegas, NV
+
|http://www.defcon.org/
+
|-
+
|16th USENIX Security Symposium
+
|Aug 06-10, Boston, MA
+
|http://www.usenix.org/events/
+
|-
+
|GMU 2007 Symposium
+
|Aug 06-10, George Mason University, Fairfax, VA
+
|http://www.rcfg.org
+
|-
+
|[[Digital Forensic Research Workshop|Digital Forensic Research Workshop 2007]]
+
|Aug 13-15, Pittsburgh, PA
+
|http://www.dfrws.org/2007/index.html
+
|-
+
|HTCIA 2007 International Training Conference & Exposition
+
|Aug 27-29, San Diego, CA
+
|http://www.htcia-sd.org/htcia2007.html
+
|-
+
|Recent Advances in Intrusion Detection (RAID) 2007
+
|Sep 05-07, Gold Coast, Queensland, Australia
+
|http://www.isi.qut.edu.au/events/conferences/raid07
+
|-
+
|14th International Conference on Image Analysis and Processing (ICIAP 2007)
+
|Sep 10-14, Modena, Italy
+
|http://www.iciap2007.org
+
|-
+
|3rd International Conference on IT-Incident Management & IT-Forensics
+
|Sep 11-12, Stuttgart, Germany
+
|http://www.imf-conference.org/
+
|-
+
|ForenSec Canada 2007
+
|Sep 17-18, Regina, Saskatchewan, Canada
+
|http://www.csiservices.ca/events.html#ForenSec
+
|-
+
|SANS Network Security
+
|Sep 22-30, Las Vegas, NV
+
|http://www.sans.org/ns2007/?portal=69456f95660ade45be29c00b0c14aea1
+
|-
+
|Black and White Ball
+
|Sep 25-28, London, UK
+
|http://www.theblackandwhiteball.co.uk/
+
|-
+
|Wisconsin Association of Computer Crimes Investigators/Forensic Association of Computer Technologists
+
|Sep 26-28, Milwaukee, WI
+
|http://www.byteoutofcrime.org
+
|-
+
|6th Annual Internet Crimes Against Children National Conference
+
|Oct 15-18, San Jose, CA
+
|http://www.icactraining.org/website/registration.html
+
|-
+
|BlackHat Japan - Briefings
+
|Oct 23-26, Tokyo, Japan
+
|http://www.blackhat.com/html/bh-japan-07/bh-jp-07-main.html
+
|-
+
|Global Conference on Economic and High-Tech Crime (NW3C Membership Required)
+
|Oct 24-26, Crystal City, VA
+
|https://conference.nw3c.org/index.cfm
+
|-
+
|European Network Forensic and Security Conference 2007
+
|Oct 24-26,  Zuyd University, Heerlen, Netherlands
+
|http://www.enfsc2007.com/
+
|-
+
|Techno-Forensics Conference
+
|Oct 29 - 31, Rockville, MD
+
|http://www.techsec.com/html/TechnoForensics2007.html
+
|-
+
|DeepSec IDSC
+
|Nov 22-24, Vienna, Austria
+
|http://deepsec.net/
+
|-
+
|Digital Forensic Forum Prague 2007
+
|Nov 26-27, Prague, Czech Republic
+
|http://www.dff-prague.com/
+
|-
+
|PacSec Applied Security Conference
+
|Nov 29-30, Tokyo, Japan
+
|http://www.pacsec.jp/index.html
+
|-
+
|DoD Cyber Crime Conference 2008
+
|Jan 13-18, St. Louis, MO
+
|http://www.dodcybercrime.com/
+
|-
+
|AAFS Annual Meeting 2008
+
|Feb 18-23, Washington, DC
+
|http://aafs.org/default.asp?section_id=meetings&page_id=aafs_annual_meeting
+
|-
+
|CanSecWest Security Conference 2008
+
|Mar 19-21, Vanouver, BC, Canada
+
|http://cansecwest.com/
+
|-
+
|EuSecWest Security Conference 2008
+
|May 21-22, London, England
+
|http://eusecwest.com/
+
|-
+
|}
+
  
== On-going / Continuous Training ==
+
<s>bad idea</s>
{| border="0" cellpadding="2" cellspacing="2" align="top"
+
:: good idea
|- style="background:#bfbfbf; font-weight: bold"
+
! Title
+
! Date/Location or Venue
+
! Website
+
|-
+
|Basic Computer Examiner Course
+
|Computer Forensic Training Online
+
|http://www.cftco.com
+
|-
+
|MaresWare Suite Training
+
|First full week every month, Atlanta, GA
+
|http://www.maresware.com/maresware/training/maresware.htm
+
|-
+
|Linux Data Forensics Training
+
|Distance Learning Format
+
|http://www.crazytrain.com/training.html
+
|-
+
|Evidence Recovery for Windows Vista&trade;
+
|First full week every month, Brunswick, GA
+
|http://www.internetcrimes.net
+
|-
+
|Evidence Recovery for Windows Server&reg; 2003 R2
+
|Second full week every month, Brunswick, GA
+
|http://www.internetcrimes.net
+
|-
+
|Evidence Recovery for the Windows XP&trade; operating system
+
|Third full week every month, Brunswick, GA
+
|http://www.internetcrimes.net
+
|-
+
|SANS On-Demand Training
+
|Distance Learning Format
+
|http://www.sans.org/ondemand/?portal=69456f95660ade45be29c00b0c14aea1
+
|-
+
|}
+
  
== Scheduled Training Courses ==
+
= License =
{| border="0" cellpadding="2" cellspacing="2" align="top"
+
 
|- style="background:#bfbfbf; font-weight: bold"
+
BSD-3.
! Title
+
:: [[User:Joachim Metz|Joachim]] library based validators could require other licenses
! Date/Location
+
 
! Website
+
= OS =
! Limitation
+
 
|-
+
Linux/FreeBSD/MacOS
|Seized Computer Evidence Recovery Specialist (SCERS)
+
: Shouldn't this just match what the underlying afflib & sleuthkit cover? [[User:RB|RB]]
|Jul 16-27, FLETC, Glynco, GA
+
:: Yes, but you need to test and validate on each. Question: Do we want to support windows? [[User:Simsong|Simsong]] 21:09, 30 October 2008 (UTC)
|http://www.fletc.gov/training/programs/computer-financial-investigations/technology-investigation/
+
:: [[User:Joachim Metz|Joachim]] I think we would do wise to design with windows support from the start this will improve the platform independence from the start
|Limited to Law Enforcement
+
:::: Agreed; I would even settle at first for being able to run against Cygwin. Note that I don't even own or use a copy of Windows, but the vast majority of forensic investigators do. [[User:RB|RB]] 14:01, 31 October 2008 (UTC)
|-
+
:: [[User:Capibara|Rob J Meijer]] Leaning heavily on the autotools might be the way to go. I do however feel that support requirements for windows would not be essential. Being able to run from a virtual machine with the main storage mounted over cifs should however be tested and if possible tuned extensively.
|EnCase Enterprise v6 - Phase II
+
:::: [[User:Joachim Metz|Joachim]] You'll need more than autotools to do native Windows support i.e. file access, UTF-16 support, wrap some basic system functions or have them available otherwise
|Jul 16-19, Washington DC
+
 
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
= Name tooling =
|-
+
 
|SMART Windows Data Forensics
+
* [[User:Joachim Metz|Joachim]] A name for the tooling I propose coldcut
|Jul 16-18, Austin, TX
+
:: How about 'butcher'?  ;)  [[User:RB|RB]] 14:20, 31 October 2008 (UTC)
|http://asrdata.com/training/training2.html
+
:: [[User:Joachim Metz|Joachim]] cleaver ( scalpel on steroids ;-) )
|-
+
* I would like to propose Gouge or Chisel :-) [[User:Capibara|Rob J Meijer]]
|EnCase v6 Computer Forensics I
+
 
|Jul 17-20, Los Angeles, CA and Houston, TX
+
= Requirements =
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
 
|-
+
[[User:Joachim Metz|Joachim]] Could we do a MoSCoW evaluation of these.
|EnCase v6 Computer Forensics II
+
 
|Jul 17-20, Chicago, IL and United Kingdom
+
* AFF and EWF file images supported from scratch. ([[User:Joachim Metz|Joachim]] I would like to have raw/split raw and device access as well)
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
:: If we base our image i/o on afflib, we get all three with one interface. [[User:RB|RB]] Instead of letting the tools use afflib, better to write an afflib module for carvfs, and update the libewf module. The tool could than be oblivious of the file format. [[User:Capibara|Rob J Meijer]]
|-
+
:::: [[User:Joachim Metz|Joachim]] this layer should support multi threaded decompression of compressed image types, this speeds up IO
|EnCase v6 Advanced Computer Forensics
+
* [[User:Joachim Metz|Joachim]] volume/partition aware layer (what about carving unpartioned space)
|Jul 17-20, Washington DC
+
* File system aware layer. This could be or make use of tsk-cp.
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
** By default, files are not carved. (clarify: only identified? [[User:RB|RB]]; I guess that it operates like [[Selective file dumper]] [[User:.FUF|.FUF]] 07:00, 29 October 2008 (UTC)). Alternatively, the tool could use libcarvpath and output carvpaths or create a directory with symlinks to carvpaths that point into a carvfs mountpoint [[User:Capibara|Rob J Meijer]].
|-
+
* Plug-in architecture for identification/validation.
|EnCase v6 NTFS
+
** [[User:Joachim Metz|Joachim]] support for multiple types of validators
|Jul 17-20, Perth, Australia
+
*** dedicated validator
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
*** validator based on file library (i.e. we could specify/implement a file structure API for these)
|-
+
*** configuration based validator (Can handle config files,like Revit07, to enter different file formats used by the carver.)
|AccessData BootCamp
+
* Ship with validators for:
|Jul 17-19, Boise, ID
+
[[User:Joachim Metz|Joachim]] I think we should distinguish between file format validators and content validators
|http://www.accessdata.com/training
+
** JPEG
|-
+
** PNG
|Paraben Handheld Forensic Course
+
** GIF
|Jul 23-26, Potomac Falls, VA
+
** MSOLE
|http://www.paraben-training.com/
+
** ZIP
|-
+
** TAR (gz/bz2)
|EnCase v6 Advanced Computer Forensics
+
 
|Jul 24-27, Chicago, IL and United Kingdom
+
[[User:Joachim Metz|Joachim]] For a production carver we need at least the following formats
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
** Grapical Images
|-
+
*** JPEG (the 3 different types with JFIF/EXIF support)
|EnCase v6 Computer Forensics I
+
*** PNG
|Jul 24-27, United Kingdom
+
*** GIF
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
*** BMP
|-
+
*** TIFF
|EnCase v6 Computer Forensics II
+
** Office documents
|Jul 24-27, Houston, TX and Los Angeles, CA
+
*** OLE2 (Word/Excell content support)
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
*** PDF
|-
+
*** Open Office/Office 2007 (ZIP+XML)
|EnCase v6 Network Intrusion Investigations - Phase I
+
:: Extension validation? AFAIK, MS Office 2007 [[DOCX]] format uses plain ZIP (or not?), and carved files will (or not?) have .zip extension instead of DOCX. Is there any way to fix this (may be using the file list in zip)? [[User:.FUF|.FUF]] 20:25, 31 October 2008 (UTC)
|Jul 24-27, Washington DC
+
:: [[User:Joachim Metz|Joachim]] Addition: Office 2007 also has a binary file format which is also a ZIP-ed data
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
 
|-
+
** Archive files
|AccessData Windows Forensics
+
*** ZIP
|Jul 24-26, Albuquerque, NM
+
*** 7z
|http://www.accessdata.com/training
+
*** gzip
|-
+
*** bzip2
|Network Forensics and Investigations Workshop
+
*** tar
|Jul 25-27, Washington, DC
+
*** RAR
|http://www.strozllc.com/trainingcenter/
+
** E-mail files
|-
+
*** PFF (PST/OST)
|EnCase v6 Network Intrusion Investigations - Phase II
+
*** MBOX (text based format, base64 content support)
|Jul 30-Aug 02, Washington DC
+
** Audio/Video files
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
*** MPEG
|-
+
*** MP2/MP3
|First Responder to Digital Evidence Program (FRDE)
+
*** AVI
|Jul 31-Aug 02, FLETC, Glynco, GA
+
*** ASF/WMV
|http://www.fletc.gov/training/programs/computer-financial-investigations/technology-investigation/
+
*** QuickTime
|Limited to Law Enforcement
+
*** MKV
|-
+
** Printer spool files
|EnCase v6 Advanced Computer Forensics
+
*** EMF (if I remember correctly)
|Jul 31-Aug 03, Los Angeles, CA
+
** Internet history files
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
*** index.dat
|-
+
*** firefox (sqllite 3)
|EnCase v6 Computer Forensics I - Private Sector
+
** Other files
|Jul 31-Aug 03, Houston, TX
+
*** thumbs.db
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
*** pagefile?
|-
+
 
|EnCase Enterprise v6 - Phase I
+
* Simple fragment recovery carving using gap carving.
|Jul 31-Aug 03, United Kingdom
+
** [[User:Joachim Metz|Joachim]] have hook in for more advanced fragment recovery?
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
* Recovering of individual ZIP sections and JPEG icons that are not sector aligned.
|-
+
** [[User:Joachim Metz|Joachim]] I would propose a generic fragment detection and recovery
|EnCase v6 Computer Forensics II
+
* Autonomous operation (some mode of operation should be completely non-interactive, requiring no human intervention to complete [[User:RB|RB]])
|Jul 31-Aug 03, Ypsilanti, MI and Orlando, FL
+
** [[User:Joachim Metz|Joachim]] as much as possible, but allow to be overwritten by user
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
* Tested on 500GB-sized images. Should be able to carve a 500GB image in roughly 50% longer than it takes to read the image.
|-
+
** Perhaps allocate a percentage budget per-validator (i.e. each validator adds N% to the carving time) [[User:RB|RB]]
|EnCase v6 Advanced Internet Examinations
+
** [[User:Joachim Metz|Joachim]] have multiple carving phases for precision/speed trade off?
|Jul 31-Aug 03, Sydney, Australia
+
* Parallelizable
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
** [[User:Joachim Metz|Joachim]] tunable for different architectures
|-
+
* Configuration:
|Paraben Wireless Forensics
+
** Capability to parse some existing carvers' configuration files, either on-the-fly or as a one-way converter.
|Aug 01-03, Potomac Falls, VA
+
** Disengage internal configuration structure from configuration files, create parsers that present the expected structure
|http://www.paraben-training.com/
+
** [[User:Joachim Metz|Joachim]] The validator should deal with the file structure the carving algorithm should not know anything about the file structure (as in revit07 design)
|-
+
**  Either extend Scalpel/Foremost syntaxes for extended features or use a tertiary syntax ([[User:Joachim Metz|Joachim]] I would prefer a derivative of the revit07 configuration syntax which already has encountered some problems of dealing with defining file structure in a configuration file)
|SARC Steganography Examiner Training
+
* Can output audit.txt file.
|Aug 04-05, Fairfax, VA (RCFG/GMU Conference 2007)
+
* [[User:Joachim Metz|Joachim]] Can output database with offset analysis values i.e. for visualization tooling
|http://www.sarc-wv.com/training.aspx
+
* [[User:Joachim Metz|Joachim]] Can output debug log for debugging the algorithm/validation
|-
+
* Easy integration into ascription software.
|SMART for Linux
+
** [[User:Joachim Metz|Joachim]] I'm no native speaker what do you mean with "ascription software"?
|Aug 06-09, Austin, TX
+
:: I think this was another non-native requesting easy scriptability. [[User:RB|RB]] 14:20, 31 October 2008 (UTC)
|http://asrdata.com/training/training2.html
+
:::: [[User:Joachim Metz|Joachim]] that makes sense ;-)
|-
+
* [[User:Joachim Metz|Joachim]] When the tool output files the filenames should contain the offset in the input data (in hexadecimal?)
|Introduction to Cyber Crime
+
* [[User:Joachim Metz|Joachim]] Should the tool allow to export embedded files?
|Aug 06-08, Mississippi State University
+
* [[User:Joachim Metz|Joachim]] Should the tool allow to export fragments separately?
|http://www.security.cse.msstate.edu/ftc/schedule.php
+
* [[User:Mark Stam|Mark]] I really like the fact carved files are named after the physical or logical sector in which the file is found (photorec)
|Limited to Law Enforcement
+
* [[User:Mark Stam|Mark]] I personally use photorec often for carving files in the whole volume (not only unallocated clusters), so I can store information about all potential interesting files in MySQL
|-
+
* [[User:Mark Stam|Mark]] It would also be nice if the files can be hashed immediately (MD5) so looking for them in other tools (for example Encase) is a snap
|X-Ways Forensics
+
 
|Aug 06-08, Seattle, WA
+
= Ideas =
|http://www.x-ways.net/training/seattle.html
+
* Use as much TSK if possible. Don't carry your own FS implementation the way photorec does.
|-
+
:: [[User:Joachim Metz|Joachim]] using TSK as much as possible would not allow to add your own file system support (i.e. mobile phones, memory structures, cap files) I would propose wrapping TSK and using it as much as possible but allow to integrate own FS implementations.
|EnCase v6 FIM/Mobile Use of EE Live Forensics
+
* Extracting/carving data from [[Thumbs.db]]? I've used [[foremost]] for it with some success. [[Vinetto]] has some critical bugs :( [[User:.FUF|.FUF]] 19:18, 28 October 2008 (UTC)
|Aug 07-10, Chicago, IL
+
:: [[User:Joachim Metz|Joachim]] this poses an interesting addition to the carver do we want to support (let's call it) 'recursive in file carving' (for now) this is different from embedded files because there is a file system structure in the file and not just another file structure
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
 
|-
+
[[User:Capibara|Rob J Meijer]] :
|EnCase v6 Advanced Computer Forensics
+
* Use libcarvpath whenever possible and by default to avoid high storage requirements.
|Aug 07-10, Houston, TX
+
:: [[User:Joachim Metz|Joachim]] For easy deployment I would not opt for making an integral part of the tool solely dependant on a single external library or the library must be integrated in the package
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
::[[User:Capibara|Rob J Meijer]] Integrating libraries (libtsk,libaff.libewf,libcarvpath etc) is bad practice, autotools are your friend IMO.
|-
+
:: [[User:Joachim Metz|Joachim]] I'm not talking about integrating (shared) libraries. I'm talking about that an integral part of a tool should be part of it's package. Why can't the tool package contain shared or static libraries for local use? A far worse thing to do is to have a large set of dependencies. The tool package should contain the most necessary code. afflib/libewf support could be detected by the autotools a neat separation of functionality.
|EnCase Enterprise v6 - Phase I
+
* Dont stop with filesystem detection after the first match. Often if a partition is reused with a new FS and is not all that full yet, much of the old FS can still be valid. I have seen this with ext2/fat. The fact that you have identified a valid FS on a partition doesn't mean there isn't an(almost) valid second FS that would yield additional files. Identifying doubly allocated space might in some cases also be relevant.
|Aug 07-10, Los Angeles, CA
+
:: [[User:Joachim Metz|Joachim]] What your saying is that dealing with file system fragments should be part of the carving algorithm
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
* Allow use where filesystem based carving is done by other tool, and the tool is used as second stage on (sets of) unallocated block (pseudo) files and/or non FS partition (pseudo) files.
|-
+
:: [[User:Joachim Metz|Joachim]] I would not opt for this. The tool would be dependent on other tools and their data format, which makes the tool difficult to maintain. I would opt to integrate the functionality of having multiple recovery phases (stages) and allow the tooling to run the phases after one and other or separately.
|EnCase eDiscovery with v6
+
::[[User:Capibara|Rob J Meijer]] More generically, I feel a way should exist to communicate the 'left overs' a previous (non open, for example LE-only) tool left.
|Aug 07-10, Washington DC
+
:: [[User:Joachim Metz|Joachim]] I guess if the tool is designed to handle multiple phases it should store its data somewhere. So it should be possible to convert results of such non open tooling to the format required. However I would opt to design the recovery functionality of these non-open tools into open tools. And not to limit ourselves making translators due to the design of these non-open tools.
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
* Ability to be used as a library instead of a tool. Ability to access metadata true library, and thus the ability to set metadata from the carving modules. This would be extremely usefull for integrating the project into a framework like ocfa.
|-
+
:: [[User:Joachim Metz|Joachim]] I guess most of the code could be integrated into libraries, but I would not opt integrating tool functionality into a library
|EnCase v6 Network Intrusion Investigations - Phase II
+
* A wild idea that I hope at least one person will have a liking for: It might be very interesting to look at the possibilities of using a multi process style of module support and combine it with a least authority design. On platforms that support AppArmor (or similar) and uid based firewall rules, this could make for the first true POLA (principle of least authority) based forensic tool ever. POLA based forensics tools should make for a strong integrity guard against many anti forensics. Alternatively we could look at integrating a capability secure language (E?) for implementation of at least validation modules. I don't expect this idea to make it, but mentioning it I hope might spark off less strong alternatives that at least partially address the integrity + anti-forensics problem. If we can in some way introduce POLA to a wider forensics public, other tools might also pick up on it what would be great. 
|Aug 07-10, United Kingdom
+
* [[User:Mark Stam|Mark]] I think it would be very handy to have a CSV, TSV, XML or other delimited output (log)file with information about carved files. This output file can then be stored in a database or Excel sheet (report function)
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
 
|-
+
== Format syntax specification ==
|EnCase v6 Computer Forensics I - Private Sector
+
* Carving data structures. For example, extract all TCP headers from image by defining TCP header structure and some fields (e.g. source port > 1024, dest port = 80). This will extract all data matching the pattern and write a file with other fields. Another example is carving INFO2 structures and URL activity records from index.dat [[User:.FUF|.FUF]] 20:51, 28 October 2008 (UTC)
|Aug 07-10, Washington DC
+
** This has the opportunity to be extended to the concept of "point at blob FOO and interpret it as BAR"
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
.FUF added:
|-
+
The main idea is to allow users to define structures, for example (in pascal-like form):
|Forensics Tools and Techniques
+
 
|Aug 08-10, Mississippi State University
+
<pre>
|http://www.security.cse.msstate.edu/ftc/schedule.php
+
Field1: Byte = 123;
|Limited to Law Enforcement
+
SomeTextLength: DWORD;
|-
+
SomeText: string[SomeTextLength];
|File Systems Revealed
+
Field4: Char = 'r';
|Aug 09-10, Seattle, WA
+
...
|http://www.x-ways.net/training/seattle.html
+
</pre>
|-
+
 
|Search and Seizure of Computers and Electronic Evidence
+
This will produce something like this:
|Aug 09-10, Oxford, MS
+
<pre>
|http://www.security.cse.msstate.edu/ftc/schedule.php
+
Field1 = 123
|Limited to Law Enforcement
+
SomeTextLength = 5
|-
+
SomeText = 'abcd1'
|Macintosh Forensic Survival Course
+
Field4 = 'r'
|Aug 13-17, Fredricksburg, VA
+
</pre>
|http://www.phoenixdatagroup.com/cart/index.php
+
 
|-
+
(In text or raw forms.)
|EnCase Enterprise v6 - Phase II
+
 
|Aug 13-16, Los Angeles, CA
+
Opinions?
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
 
|-
+
Opinion: Simple pattern identification like that may not suffice, I think Simson's original intent was not only to identify but to allow for validation routines (plugins, as the original wording was). As such, the format syntax would need to implement a large chunk of some programming language in order to be sufficiently flexible. [[User:RB|RB]]
|X-Ways Forensics
+
 
|Aug 13-15, Long Beach, CA
+
[[User:Joachim Metz|Joachim]]
|http://www.x-ways.net/training/long_beach.html
+
In my option your example is too limited. Making the revit configuration I learned you'll need a near programming language to specify some file formats.
|-
+
A simple descriptive language is too limiting. I would also go for 2 bytes with endianess instead of using terminology like WORD and small integer, it's much more clear. The configuration also needs to deal with aspects like cardinality, required and optional structures.
|SMART Linux Data Forensics
+
:: This is simply data structures carving, see ideas above. Somebody (I cannot track so many changes per day) separated the original text. There is no need to count and join different structures. [[User:.FUF|.FUF]] 19:53, 31 October 2008 (UTC)
|Aug 13-15, Austin, TX
+
:::: [[User:Joachim Metz|Joachim]] This was probably me is the text back in it's original form?
|http://asrdata.com/training/training2.html
+
:::: I started it by moving your Revit07 comment to the validator/plugin section in [http://www.forensicswiki.org/index.php?title=Carver_2.0_Planning_Page&diff=prev&oldid=7583 this edit], since I was still at that point thinking operational configuration for that section, not parser configurations. [[User:RB|RB]]
|-
+
:::: [[User:Joachim Metz|Joachim]] I renamed the title to format syntax, clarity is important ;-)
|Network Forensics and Investigations Workshop
+
 
|Aug 13-15, Los Angeles, CA
+
Please take a look at the revit07 configuration. It's not there yet but goes a far way. Some things currently missing:
|http://www.strozllc.com/trainingcenter/
+
* bitwise alignment
|-
+
* handling encapsulated streams (MPEG/capture files)
|Paraben Cellular/GPS Signal Analysis
+
* handling content based formats (MBOX)
|Aug 13-14, Potomac Falls, VA
+
 
|http://www.paraben-training.com/
+
=Caving algorithm =
|-
+
[[User:Joachim Metz|Joachim]]
|Computer Network Investigations Training Program (CNITP)
+
* should we allow for multiple carving phases (runs/stages)?
|Aug 14-24, FLETC, Glynco, GA
+
:: I opt yes (separation of concern)
|http://www.fletc.gov/training/programs/computer-financial-investigations/technology-investigation/
+
* should we allow for multiple carving algorithms?
|Limited to Law Enforcement
+
:: I opt yes, this allows testing of different approaches
|-
+
* Should the algorithm try to do as much in 1 run over the input data? To reduce IO?
|EnCase v6 NTFS
+
:: I opt that the tool should allow for multiple and single run over the input data to minimize the IO or the CPU as bottleneck
|Aug 14-17, Chicago, IL
+
* Interaction between algorithm and validators
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
** does the algorithm passes data blocks to the validators?
|-
+
** does a validator need to maintain a state?
|EnCase v6 Computer Forensics II – Private Sector
+
** does a validator need to revert a state?
|Aug 14-17, Washington DC and United Kingdom
+
** How do we deal with embedded files and content validation? Do the validators call another validator?
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
* do we use the assumption that a data block can be used by a single file (with the exception of embedded/encapsulated files)?
|-
+
* Revit07 allows for multiple concurrent result files states to deal with fragmentation. One has the attribute of being active (the preferred) and the other passive. Do we want/need something similar? The algorithm adds block of input data (offsets) to these result files states.
|EnCase v6 Computer Forensics I
+
** if so what info would these result files states require (type, list of input data blocks)
|Aug 14-17, Houston, TX
+
* how do we deal with file system remainders?
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
** Can we abstract them and compare them against available file system information?
|-
+
* Do we carve file systems in files?
|AccessData Internet Forensics
+
:: I opt that at least the validator uses this information
|Aug 14-16, Austin, TX
+
 
|http://www.accessdata.com/training
+
==Caving scenarios ==
|-
+
[[User:Joachim Metz|Joachim]]
|File Systems Revealed
+
* normal file (file structure, loose text based structure (more a content structure?))
|Aug 16-17, Long Beach, CA
+
* fragmented file (the file entirely exist)
|http://www.x-ways.net/training/long_beach.html
+
* a file fragment (the file does not entirely exist)
|-
+
* intertwined file
|EnCase v6 Computer Forensics II
+
* encapsulated file (MPEG/network capture)
|Aug 21-24, Houston, TX
+
* embedded file (JPEG thumbnail)
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
* obfuscation ('encrypted' PFF) this also entails encryption and/or compression
|-
+
* file system in file
|EnCase v6 FIM/Mobile Use of EE Live Forensics
+
 
|Aug 21-24, Melbourne, Australia
+
=File System Awareness =
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
==Background: Why be File System Aware?==
|-
+
Advantages of being FS aware:
|EnCase v6 NTFS
+
* You can pick up sector allocation sizes
|Aug 21-24, Washington DC
+
:: [[User:Joachim Metz|Joachim]] do you mean file system block sizes?
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
* Some file systems may store things off sector boundaries. (ReiserFS with tail packing)
|-
+
* Increasingly file systems have compression (NTFS compression)
|EnCase v6 Computer Forensics I
+
* Carve just the sectors that are not in allocated files.
|Aug 21-24, Chicago, IL
+
 
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
==Tasks that would be required==
|-
+
 
|EnCase v6 Advanced Internet Examinations
+
==Discussion==
|Aug 21-24, Los Angeles, CA
+
:: As noted above, TSK should be utilized as much as possible, particularly the filesystem-aware portion. If we want to identify filesystems outside of its supported set, it would be more worth our time to work on implementing them there than in the carver itself.  [[User:RB|RB]]
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
 
|-
+
:::: I guess this tool operates like [[Selective file dumper]] and can recover files in both ways (or not?). Recovering files by using carving can recover files in situations where sleuthkit does nothing (e.g. file on NTFS was deleted using ntfs-3g, or filesystem was destroyed or just unknown). And we should build the list of filesystems supported by carver, not by TSK. [[User:.FUF|.FUF]] 07:08, 29 October 2008 (UTC)
|EnCase Enterprise v6 - Phase II
+
 
|Aug 21-24, United Kingdom
+
:: This tool is still in the early planning stages (requirements discovery), hence few operational details (like precise modes of operation) have been fleshed out - those will and should come later.  The justification for strictly using TSK for the filesystem-sensitive approach is simple: TSK has good filesystem APIs, and it would be foolish to create yet another standalone, incompatible implementation of filesystem(foo) when time would be better spent improving those in TSK, aiding other methods of analysis as well.  This is the same reason individuals that have implemented several other carvers are participating: de-duplication of effort. [[User:RB|RB]]
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
 
|-
+
[[User:Joachim Metz|Joachim]] I would like to have the carver (recovery tool) also do recovery using file allocation data or remainders of file allocation data.
|SARC Steganography Examiner Training
+
 
|Aug 24-25, San Diego, CA (HTCIA Conference 2007)
+
[[User:Joachim Metz|Joachim]]
|http://www.sarc-wv.com/training.aspx
+
I would go as far to ask you all to look beyond the carver as a tool and look from the perspective of the carver as part of the forensic investigation process. In my eyes certain information needed/acquired by the carver could be also very useful investigative information i.e. what part of a hard disk contains empty sectors.
|-
+
 
|EnCase v6 Computer Forensics II
+
=Supportive tooling=
|Aug 28-31, Los Angeles, CA
+
[[User:Joachim Metz|Joachim]]
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
* validator (definitions) tester (detest in revit07)
|-
+
* tool to make configuration based definitions
|EnCase v6 Computer Forensics I
+
* post carving validation
|Aug 28-31, Singapore
+
* the carver needs to provide support for fuse mount of carved files (carvfs)
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
 
|-
+
=Testing =
|EnCase v6 Computer Forensics I
+
[[User:Joachim Metz|Joachim]]
|Aug 28-31, Savannah, Georgia
+
* automated testing
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
* test data
|-
+
 
|EnCase v6 Advanced Internet Examinations
+
=Validator Construction=
|Aug 28-31, Washington DC
+
Options:
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
* Write validators in C/C++
|-
+
** [[User:Joachim Metz|Joachim]] you mean dedicated validators
|EnCase v6 Computer Forensics I - Private Sector
+
* Have a scripting language for writing them (python? Perl?) our own?
|Aug 28-31, Houston, TX
+
** [[User:Joachim Metz|Joachim]] use easy to embed programming languages i.e. Phyton or Lua
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
* Use existing programs (libjpeg?) as plug-in validators?
|-
+
** [[User:Joachim Metz|Joachim]] define a file structure api for this
|Helix Live Forensics and Incident Response Course
+
 
|Aug 28-30, Tennessee Bureau of Investigations - Nashville, TN
+
=Existing Code that we have=
|https://www.e-fense.com/register.php
+
[[User:Joachim Metz|Joachim]]
|-
+
Please add any missing links
|Paraben Cellular/GPS Signal Analysis
+
 
|Aug 30-31, Potomac Falls, VA
+
Documentation/Articles
|http://www.paraben-training.com/
+
* DFRWS2006/2007 carving challenge results
|-
+
* DFRWS2008 paper on carving
|SMART for Linux
+
 
|Sep 03-06, Austin, TX
+
Carvers
|http://asrdata.com/training/training2.html
+
* DFRWS2006/2007 carving challenge results
|-
+
* photorec (http://www.cgsecurity.org/wiki/PhotoRec)
|Paraben Handheld Forensic Course
+
* revit06 and revit07 (http://sourceforge.net/projects/revit/)
|Sep 04-07, Mississauga, Ontario, Canada
+
* s3/scarve
|http://www.paraben-training.com/
+
 
|-
+
Possible file structure validator libraries
|EnCase v6 Computer Forensics II – Private Sector
+
* divers existing file support libraries
|Sep 04-07, Houston, TX
+
* libole2 (inhouse experimental code of OLE2 support)
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
* libpff (alpha release for PFF (PST/OST) file support) (http://sourceforge.net/projects/libpff/)
|-
+
 
|EnCase v6 Computer Forensics I
+
Input support
|Sep 04-07, Melbourne, Australia and United Kingdom
+
* AFF (http://www.afflib.org/)
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
* EWF (http://sourceforge.net/projects/libewf/)
|-
+
* TSK device & raw & split raw (http://www.sleuthkit.org/)
|EnCase v6 FIM/Mobile Use of EE Live Forensics
+
 
|Sep 04-07, The Netherlands
+
Volume/Partition support
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
* disktype (http://disktype.sourceforge.net/)
|-
+
* testdisk (http://www.cgsecurity.org/wiki/TestDisk)
|EnCase v6 Computer Forensics II
+
* TSK
|Sep 04-07, Austin, TX
+
 
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
File system support
|-
+
* TSK
|AccessData BootCamp
+
* photorec FS code
|Sep 04-06, Mississippi State University
+
* implementations of FS in Linux/BSD
|http://www.security.cse.msstate.edu/ftc/schedule.php
+
 
|Limited to Law Enforcement
+
Content support
|-
+
 
|Paraben Advanced Cell Phone Forensics
+
Zero storage support
|Sep 10-12, San Diego, CA
+
* libcarvpath
|http://www.paraben-training.com/
+
* carvfs
|-
+
 
|Paraben E-Discovery: E-mail & Mobile E-mail Devices
+
POLA
|Sep 10-14, Potomac Falls, VA
+
* joe-e (java)
|http://www.paraben-training.com/
+
* Emily (ocaml)
|-
+
* the E language
|EnCase v6 Computer Forensics II
+
* AppArmor
|Sep 11-14, United Kingdom and Singapore
+
* iptables/ipfw
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
* minorfs
|-
+
* plash
|EnCase v6 Computer Forensics I
+
 
|Sep 11-14, Houston, TX and Washington, DC
+
=Implementation Timeline=
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
# gather the available resources/ideas/wishes/needs etc. (I guess we're in this phase)
|-
+
# start discussing a high level design (in terms of algorithm, facilities, information needed)
|EnCase Enterprise v6 - Phase I
+
## input formats facility
|Sep 11-14, Chicago, IL
+
## partition/volume facility
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
## file system facility
|-
+
## file format facility
|EnCase v6 Computer Forensics I - Private Sector
+
## content facility
|Sep 11-14, Los Angeles, CA
+
## how to deal with fragment detection (do the validators allow for fragment detection?)
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
## how to deal with recombination of fragments
|-
+
## do we want multiple carving phases in light of speed/precision tradeoffs
|First Responder to Digital Evidence Program (FRDE)
+
# start detailing parts of the design
|Sep 11-13, FLETC, Glynco, GA
+
## Discuss options for a grammar driven validator?
|http://www.fletc.gov/training/programs/computer-financial-investigations/technology-investigation/
+
## Hard-coded plug-ins?
|Limited to Law Enforcement
+
## Which existing code can we use?
|-
+
# start building/assembling parts of the tooling for a prototype
|AccessData Applied Decryption
+
## Implement simple file carving with validation.
|Sep 11-13, Dallas, TX
+
## Implement gap carving
|http://www.accessdata.com/training
+
# Initial Release
|-
+
# Implement the ''threaded carving'' that [[User:.FUF|.FUF]] is describing above.
|Paraben Advanced SIM Card Forensics
+
 
|Sep 13-14, San Diego, CA
+
[[User:Joachim Metz|Joachim]] Shouldn't multi threaded carving (MTC) not be part of the 1st version?
|http://www.paraben-training.com/
+
The MT approach makes for different design decisions
|-
+
|Enterprise Data Forensics
+
|Sep 17-19, Austin, TX
+
|http://asrdata.com/training/training2.html
+
|-
+
|Paraben Network Incident Response
+
|Sep 17-21, Potomac Falls, VA
+
|http://www.paraben-training.com/
+
|-
+
|EnCase v6 Computer Forensics II – Private Sector
+
|Sep 18-21, Los Angeles, CA
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Computer Forensics II
+
|Sep 18-21, Houston, TX and Leipzig, Germany
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Advanced Computer Forensics
+
|Sep 18-21, Sydney, Australia and United Kingdom
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Computer Forensics I
+
|Sep 18-21, Toronto, Canada
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|Paraben Cellular/GPS Signal Analysis
+
|Sep 20-21, San Diego, CA
+
|http://www.paraben-training.com/
+
|-
+
|Paraben Advanced Cell Phone Forensics
+
|Sep 24-26, Potomac Falls, VA
+
|http://www.paraben-training.com/
+
|-
+
|Introduction to Cyber Crime
+
|Sep 24-26, Jackson, Mississippi
+
|http://www.security.cse.msstate.edu/ftc/schedule.php
+
|Limited to Law Enforcement
+
|-
+
|Internet Investigations Training Program (IITP)
+
|Sep 24-28, Glynco, GA
+
|http://www.fletc.gov/training/programs/computer-financial-investigations/technology-investigation
+
|Limited to Law Enforcement
+
|-
+
|Macintosh Forensic Survival Course
+
|Sep 24-28, Santa Ana, CA
+
|http://www.phoenixdatagroup.com/cart/index.php
+
|-
+
|BlackBag Introductory MacIntosh Forensics
+
|Sep 24-28, Richmond, VA
+
|http://www.blackbagtech.com/products/training.htm
+
|Limited to Law Enforcement
+
|-
+
|EnCase v6 FIM/Mobile Use of EE Live Forensics
+
|Sep 25-28, United Kingdom
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Computer Forensics I - Private Sector
+
|Sep 25-28, Houston, TX
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Advanced Computer Forensics
+
|Sep 25-28, Los Angeles, CA
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Computer Forensics II
+
|Sep 25-28, Toronto, Ontario, Canada
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|AccessData Applied Decryption
+
|Sep 25-27, Chicago, IL
+
|http://www.accessdata.com/training
+
|-
+
|AccessData BootCamp
+
|Sep 25-27, Solna, SE
+
|http://www.accessdata.com/training
+
|-
+
|Forensics Tools and Techniques
+
|Sep 26-28, Jackson, Mississippi
+
|http://www.security.cse.msstate.edu/ftc/schedule.php
+
|Limited to Law Enforcement
+
|-
+
|Paraben Advanced SIM Card Forensics
+
|Sep 27-28, Potomac Falls, VA
+
|http://www.paraben-training.com/
+
|-
+
|Paraben Wireless Forensics
+
|Oct 01-03, San Diego, CA
+
|http://www.paraben-training.com/
+
|-
+
|SMART for Linux
+
|Oct 01-04, Austin, TX
+
|http://asrdata.com/training/training2.html
+
|-
+
|EnCase v6 Computer Forensics I - Private Sector
+
|Oct 02-05, Chicago, IL
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 FIM/Mobile Use of EE Live Forensics
+
|Oct 02-05, Los Angeles, CA, Washington, DC and Perth, Australia
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Computer Forensics II – Private Sector
+
|Oct 02-05, Houston, TX
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Computer Forensics I
+
|Oct 02-05, The Netherlands
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 EnScript Programming - Phase II
+
|Oct 02-05, Washington DC
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Advanced Internet Examinations
+
|Oct 02-05, Toronto, Canada
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|Paraben Cellular/GPS Signal Analysis
+
|Oct 04-05, Potomac Falls, VA
+
|http://www.paraben-training.com/
+
|-
+
|SMART Windows Data Forensics
+
|Oct 08-10, Austin, TX
+
|http://asrdata.com/training/training2.html
+
|-
+
|Paraben Handheld Forensic Course
+
|Oct 8-11, San Diego, CA
+
|http://www.paraben-training.com/
+
|-
+
|Paraben Handheld Forensic Course
+
|Oct 8-11, Potomac Falls, VA
+
|http://www.paraben-training.com/
+
|-
+
|EnCase v6 Network Intrusion Investigations - Phase I
+
|Oct 09-12, Los Angeles, CA and The Netherlands
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 FIM/Mobile Use of EE Live Forensics
+
|Oct 09-12, Sydney, Australia
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Computer Forensics I
+
|Oct 09-12, Houston, TX
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Advanced Internet Examinations
+
|Oct 09-12, Washington, DC and United Kingdom
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Advanced Computer Forensics
+
|Oct 09-12, Washington DC
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Computer Forensics II – Private Sector
+
|Oct 09-12, Chicago, IL
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|Digital Evidence Acquisition Specialist Training Program (DEASTP)
+
|Oct 15-26, FLETC, Glynco, GA
+
|http://www.fletc.gov/training/programs/computer-financial-investigations/technology-investigation/
+
|Limited to Law Enforcement
+
|-
+
|BlackBag Introductory MacIntosh Forensics
+
|Oct 15-19, Tacoma, WA
+
|http://www.blackbagtech.com/products/training.htm
+
|-
+
|Paraben E-Discovery: E-mail & Mobile E-mail Devices
+
|Oct 15-19, Mississauga, Ontario, Canada
+
|http://www.paraben-training.com/
+
|-
+
|EnCase v6 Network Intrusion Investigations - Phase II
+
|Oct 15-18, Los Angeles, CA
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|Paraben Advanced Cell Phone Forensics
+
|Oct 15-17, Mississauga, Ontario, Canada
+
|http://www.paraben-training.com/
+
|-
+
|EnCase v6 Computer Forensics I - Private Sector
+
|Oct 16-19, Washington DC
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Network Intrusion Investigations - Phase I
+
|Oct 16-19, United Kingdom
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Computer Forensics II
+
|Oct 16-19, Washington DC and Houston, TX
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Advanced Computer Forensics
+
|Oct 16-19, Chicago, IL
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase Enterprise v6 - Phase II
+
|Oct 16-19, Chicago, IL
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Network Intrusion Investigations - Phase II
+
|Oct 16-19, The Netherlands
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Advanced Internet Examinations
+
|Oct 16-19, Austin, TX
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 EnScript Programming - Phase I
+
|Oct 16-19, Toronto, Canada
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|Paraben Advanced SIM Card Forensics
+
|Oct 18-19, Mississauga, Ontario, Canada
+
|http://www.paraben-training.com/
+
|-
+
|X-Ways Forensics
+
|Oct 22-24, Hong Kong
+
|http://www.x-ways.net/training/hong_kong.html
+
|-
+
|EnCase v6 Computer Forensics II
+
|Oct 23-26, Toronto, Canada
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Advanced Internet Examinations
+
|Oct 23-26, Canberra, Australia
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Computer Forensics I - Private Sector
+
|Oct 23-26, Houston, TX
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Computer Forensics I
+
|Oct 23-26, Los Angeles, CA and Singapore
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase eDiscovery with v6
+
|Oct 23-26, Washington DC
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Computer Forensics II – Private Sector
+
|Oct 23-26, Washington DC
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|File Systems Revealed
+
|Oct 25-26, Hong Kong
+
|http://www.x-ways.net/training/hong_kong.html
+
|-
+
|SARC Steganography Examiner Training
+
|Oct 26 - 27, Gaithersburg, MD (Techno Forensics Conference 2007)
+
|http://www.sarc-wv.com/training.aspx
+
|-
+
|Seized Computer Evidence Recovery Specialist (SCERS)
+
|Oct 29-Nov 9, FLETC, Glynco, GA
+
|http://www.fletc.gov/training/programs/computer-financial-investigations/technology-investigation/
+
|Limited to Law Enforcement
+
|-
+
|Search and Seizure of Computers and Electronic Evidence
+
|Oct 29-30, Oxford, MS
+
|http://www.security.cse.msstate.edu/ftc/schedule.php
+
|Limited to Law Enforcement
+
|-
+
|EnCase v6 Computer Forensics II
+
|Oct 30-Nov 02, Los Angeles, CA and The Netherlands
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Advanced Computer Forensics
+
|Oct 30-Nov 02, Washington DC and Toronto, Ontario, Canada
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Computer Forensics I - Private Sector
+
|Oct 30-Nov 02, United Kingdom
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Network Intrusion Investigations - Phase I
+
|Oct 30-Nov 02, Chicago, IL
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase Enterprise v6 - Phase I
+
|Oct 30-Nov 02, Washington DC
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Computer Forensics II – Private Sector
+
|Oct 30-Nov 02, Houston, TX
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|Paraben Handheld Forensic Course
+
|Nov 05-08, Mississauga, Ontario, Canada
+
|http://www.paraben-training.com/
+
|-
+
|SMART for Linux
+
|Nov 05-08, Austin, TX
+
|http://asrdata.com/training/training2.html
+
|-
+
|EnCase v6 Network Intrusion Investigations - Phase II
+
|Nov 05-08, Chicago, IL
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase Enterprise v6 - Phase II
+
|Nov 05-08, Washington DC
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|Introduction to Cyber Crime
+
|Nov 05-07, Jackson, Mississippi
+
|http://www.security.cse.msstate.edu/ftc/schedule.php
+
|Limited to Law Enforcement
+
|-
+
|EnCase v6 Computer Forensics II – Private Sector
+
|Nov 06-09, United Kingdom
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Advanced Computer Forensics
+
|Nov 06-09, Los Angeles, CA
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 NTFS
+
|Nov 06-09, Washington DC
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Computer Forensics II
+
|Nov 06-09, Houston, TX
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|AccessData BootCamp
+
|Nov 06-08, Austin, TX
+
|http://www.accessdata.com/training
+
|-
+
|AccessData Windows Forensics
+
|Nov 06-08, Solna, Sweden
+
|http://www.accessdata.com/training
+
|-
+
|Forensics Tools and Techniques
+
|Nov 07-09, Jackson, Mississippi
+
|http://www.security.cse.msstate.edu/ftc/schedule.php
+
|Limited to Law Enforcement
+
|-
+
|SMART Linux Data Forensics
+
|Nov 12-14, Austin, TX
+
|http://asrdata.com/training/training2.html
+
|-
+
|EnCase v6 Computer Forensics I - Private Sector
+
|Nov 13-16, Houston, TX
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Advanced Computer Forensics
+
|Nov 13-16, The Netherlands and United Kingdom
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Computer Forensics I
+
|Nov 13-16, Sydney, Australia and Singapore
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Advanced Internet Examinations
+
|Nov 13-16, Chicago, IL and Los Angeles, CA
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Computer Forensics II
+
|Nov 13-16, Washington DC
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|AccessData BootCamp
+
|Nov 13-15, Mississippi State University
+
|http://www.security.cse.msstate.edu/ftc/schedule.php
+
|Limited to Law Enforcement
+
|-
+
|EnCase v6 Computer Forensics II
+
|Nov 20-23, Toronto, Canada
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 FIM/Mobile Use of EE Live Forensics
+
|Nov 20-23, Vancouver, BC
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 NTFS
+
|Nov 27-30, Vancouver, BC
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Network Intrusion Investigations - Phase I
+
|Nov 27-30, Sydney, Australia
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Computer Forensics II – Private Sector
+
|Nov 27-30, Houston, TX
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Computer Forensics I
+
|Nov 27-30, Washington DC
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase eDiscovery with v6
+
|Nov 27-30, Los Angeles, CA
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|Internet Investigations Training Program (IITP)
+
|Dec 03-07, Glynco, GA
+
|http://www.fletc.gov/training/programs/computer-financial-investigations/technology-investigation
+
|Limited to Law Enforcement
+
|-
+
|SMART for Linux
+
|Dec 03-06, Austin, TX
+
|http://asrdata.com/training/training2.html
+
|-
+
|Introduction to Cyber Crime
+
|Dec 03-05, Mississippi State University
+
|http://www.security.cse.msstate.edu/ftc/schedule.php
+
|Limited to Law Enforcement
+
|-
+
|EnCase v6 Computer Forensics I
+
|Dec 04-07, Chicago, IL; Los Angeles, CA; Houston, TX; and United Kingdom
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Network Intrusion Investigations - Phase I
+
|Dec 04-07, Washington DC
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Computer Forensics II
+
|Dec 04-07, Washington DC, Leipzig, Germany and Toronto, Ontario, Canada
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Advanced Internet Examinations
+
|Dec 04-07, Vancouver, BC
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|AccessData Internet Forensics
+
|Dec 04-06 , Solna, Sweden
+
|http://www.accessdata.com/training
+
|-
+
|Forensics Tools and Techniques
+
|Dec 05-07, Mississippi State University
+
|http://www.security.cse.msstate.edu/ftc/schedule.php
+
|Limited to Law Enforcement
+
|-
+
|EnCase v6 Network Intrusion Investigations - Phase II
+
|Dec 10-13, Washington DC
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|Enterprise Data Forensics
+
|Dec 10-12, Austin, TX
+
|http://asrdata.com/training/training2.html
+
|-
+
|EnCase v6 Computer Forensics II
+
|Dec 11-14, Chicago, IL; Houston, TX; Los Angeles, CA; United Kingdom; and Melbourne, Australia
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Advanced Computer Forensics
+
|Dec 11-14, Washington DC
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Advanced Computer Forensics
+
|Dec 17-20, Chicago, IL and Los Angeles, CA
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 FIM/Mobile Use of EE Live Forensics
+
|Dec 17-20, Washington DC
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 Advanced Internet Examinations
+
|Dec 17-20, Washington, DC and United Kingdom
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|EnCase v6 NTFS
+
|Dec 17-20, Los Angeles, CA
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|Paraben Advanced Cell Phone Forensics
+
|Dec 17-19, Mississauga, Ontario, Canada
+
|http://www.paraben-training.com/
+
|-
+
|EnCase v6 Computer Forensics II – Private Sector
+
|Dec 18-21, Houston, TX
+
|http://www.guidancesoftware.com/training/course_schedule.aspx
+
|-
+
|Paraben Advanced SIM Card Forensics
+
|Dec 20-21, Mississauga, Ontario, Canada
+
|http://www.paraben-training.com/
+
|-
+
|}
+

Revision as of 02:56, 2 November 2008

This page is for planning Carver 2.0.

Please, do not delete text (ideas) here. Use something like this:

<s>bad idea</s>
:: good idea

This will look like:

bad idea

good idea

License

BSD-3.

Joachim library based validators could require other licenses

OS

Linux/FreeBSD/MacOS

Shouldn't this just match what the underlying afflib & sleuthkit cover? RB
Yes, but you need to test and validate on each. Question: Do we want to support windows? Simsong 21:09, 30 October 2008 (UTC)
Joachim I think we would do wise to design with windows support from the start this will improve the platform independence from the start
Agreed; I would even settle at first for being able to run against Cygwin. Note that I don't even own or use a copy of Windows, but the vast majority of forensic investigators do. RB 14:01, 31 October 2008 (UTC)
Rob J Meijer Leaning heavily on the autotools might be the way to go. I do however feel that support requirements for windows would not be essential. Being able to run from a virtual machine with the main storage mounted over cifs should however be tested and if possible tuned extensively.
Joachim You'll need more than autotools to do native Windows support i.e. file access, UTF-16 support, wrap some basic system functions or have them available otherwise

Name tooling

  • Joachim A name for the tooling I propose coldcut
How about 'butcher'?  ;) RB 14:20, 31 October 2008 (UTC)
Joachim cleaver ( scalpel on steroids ;-) )

Requirements

Joachim Could we do a MoSCoW evaluation of these.

  • AFF and EWF file images supported from scratch. (Joachim I would like to have raw/split raw and device access as well)
If we base our image i/o on afflib, we get all three with one interface. RB Instead of letting the tools use afflib, better to write an afflib module for carvfs, and update the libewf module. The tool could than be oblivious of the file format. Rob J Meijer
Joachim this layer should support multi threaded decompression of compressed image types, this speeds up IO
  • Joachim volume/partition aware layer (what about carving unpartioned space)
  • File system aware layer. This could be or make use of tsk-cp.
    • By default, files are not carved. (clarify: only identified? RB; I guess that it operates like Selective file dumper .FUF 07:00, 29 October 2008 (UTC)). Alternatively, the tool could use libcarvpath and output carvpaths or create a directory with symlinks to carvpaths that point into a carvfs mountpoint Rob J Meijer.
  • Plug-in architecture for identification/validation.
    • Joachim support for multiple types of validators
      • dedicated validator
      • validator based on file library (i.e. we could specify/implement a file structure API for these)
      • configuration based validator (Can handle config files,like Revit07, to enter different file formats used by the carver.)
  • Ship with validators for:

Joachim I think we should distinguish between file format validators and content validators

    • JPEG
    • PNG
    • GIF
    • MSOLE
    • ZIP
    • TAR (gz/bz2)

Joachim For a production carver we need at least the following formats

    • Grapical Images
      • JPEG (the 3 different types with JFIF/EXIF support)
      • PNG
      • GIF
      • BMP
      • TIFF
    • Office documents
      • OLE2 (Word/Excell content support)
      • PDF
      • Open Office/Office 2007 (ZIP+XML)
Extension validation? AFAIK, MS Office 2007 DOCX format uses plain ZIP (or not?), and carved files will (or not?) have .zip extension instead of DOCX. Is there any way to fix this (may be using the file list in zip)? .FUF 20:25, 31 October 2008 (UTC)
Joachim Addition: Office 2007 also has a binary file format which is also a ZIP-ed data
    • Archive files
      • ZIP
      • 7z
      • gzip
      • bzip2
      • tar
      • RAR
    • E-mail files
      • PFF (PST/OST)
      • MBOX (text based format, base64 content support)
    • Audio/Video files
      • MPEG
      • MP2/MP3
      • AVI
      • ASF/WMV
      • QuickTime
      • MKV
    • Printer spool files
      • EMF (if I remember correctly)
    • Internet history files
      • index.dat
      • firefox (sqllite 3)
    • Other files
      • thumbs.db
      • pagefile?
  • Simple fragment recovery carving using gap carving.
    • Joachim have hook in for more advanced fragment recovery?
  • Recovering of individual ZIP sections and JPEG icons that are not sector aligned.
    • Joachim I would propose a generic fragment detection and recovery
  • Autonomous operation (some mode of operation should be completely non-interactive, requiring no human intervention to complete RB)
    • Joachim as much as possible, but allow to be overwritten by user
  • Tested on 500GB-sized images. Should be able to carve a 500GB image in roughly 50% longer than it takes to read the image.
    • Perhaps allocate a percentage budget per-validator (i.e. each validator adds N% to the carving time) RB
    • Joachim have multiple carving phases for precision/speed trade off?
  • Parallelizable
    • Joachim tunable for different architectures
  • Configuration:
    • Capability to parse some existing carvers' configuration files, either on-the-fly or as a one-way converter.
    • Disengage internal configuration structure from configuration files, create parsers that present the expected structure
    • Joachim The validator should deal with the file structure the carving algorithm should not know anything about the file structure (as in revit07 design)
    • Either extend Scalpel/Foremost syntaxes for extended features or use a tertiary syntax (Joachim I would prefer a derivative of the revit07 configuration syntax which already has encountered some problems of dealing with defining file structure in a configuration file)
  • Can output audit.txt file.
  • Joachim Can output database with offset analysis values i.e. for visualization tooling
  • Joachim Can output debug log for debugging the algorithm/validation
  • Easy integration into ascription software.
    • Joachim I'm no native speaker what do you mean with "ascription software"?
I think this was another non-native requesting easy scriptability. RB 14:20, 31 October 2008 (UTC)
Joachim that makes sense ;-)
  • Joachim When the tool output files the filenames should contain the offset in the input data (in hexadecimal?)
  • Joachim Should the tool allow to export embedded files?
  • Joachim Should the tool allow to export fragments separately?
  • Mark I really like the fact carved files are named after the physical or logical sector in which the file is found (photorec)
  • Mark I personally use photorec often for carving files in the whole volume (not only unallocated clusters), so I can store information about all potential interesting files in MySQL
  • Mark It would also be nice if the files can be hashed immediately (MD5) so looking for them in other tools (for example Encase) is a snap

Ideas

  • Use as much TSK if possible. Don't carry your own FS implementation the way photorec does.
Joachim using TSK as much as possible would not allow to add your own file system support (i.e. mobile phones, memory structures, cap files) I would propose wrapping TSK and using it as much as possible but allow to integrate own FS implementations.
  • Extracting/carving data from Thumbs.db? I've used foremost for it with some success. Vinetto has some critical bugs :( .FUF 19:18, 28 October 2008 (UTC)
Joachim this poses an interesting addition to the carver do we want to support (let's call it) 'recursive in file carving' (for now) this is different from embedded files because there is a file system structure in the file and not just another file structure

Rob J Meijer :

  • Use libcarvpath whenever possible and by default to avoid high storage requirements.
Joachim For easy deployment I would not opt for making an integral part of the tool solely dependant on a single external library or the library must be integrated in the package
Rob J Meijer Integrating libraries (libtsk,libaff.libewf,libcarvpath etc) is bad practice, autotools are your friend IMO.
Joachim I'm not talking about integrating (shared) libraries. I'm talking about that an integral part of a tool should be part of it's package. Why can't the tool package contain shared or static libraries for local use? A far worse thing to do is to have a large set of dependencies. The tool package should contain the most necessary code. afflib/libewf support could be detected by the autotools a neat separation of functionality.
  • Dont stop with filesystem detection after the first match. Often if a partition is reused with a new FS and is not all that full yet, much of the old FS can still be valid. I have seen this with ext2/fat. The fact that you have identified a valid FS on a partition doesn't mean there isn't an(almost) valid second FS that would yield additional files. Identifying doubly allocated space might in some cases also be relevant.
Joachim What your saying is that dealing with file system fragments should be part of the carving algorithm
  • Allow use where filesystem based carving is done by other tool, and the tool is used as second stage on (sets of) unallocated block (pseudo) files and/or non FS partition (pseudo) files.
Joachim I would not opt for this. The tool would be dependent on other tools and their data format, which makes the tool difficult to maintain. I would opt to integrate the functionality of having multiple recovery phases (stages) and allow the tooling to run the phases after one and other or separately.
Rob J Meijer More generically, I feel a way should exist to communicate the 'left overs' a previous (non open, for example LE-only) tool left.
Joachim I guess if the tool is designed to handle multiple phases it should store its data somewhere. So it should be possible to convert results of such non open tooling to the format required. However I would opt to design the recovery functionality of these non-open tools into open tools. And not to limit ourselves making translators due to the design of these non-open tools.
  • Ability to be used as a library instead of a tool. Ability to access metadata true library, and thus the ability to set metadata from the carving modules. This would be extremely usefull for integrating the project into a framework like ocfa.
Joachim I guess most of the code could be integrated into libraries, but I would not opt integrating tool functionality into a library
  • A wild idea that I hope at least one person will have a liking for: It might be very interesting to look at the possibilities of using a multi process style of module support and combine it with a least authority design. On platforms that support AppArmor (or similar) and uid based firewall rules, this could make for the first true POLA (principle of least authority) based forensic tool ever. POLA based forensics tools should make for a strong integrity guard against many anti forensics. Alternatively we could look at integrating a capability secure language (E?) for implementation of at least validation modules. I don't expect this idea to make it, but mentioning it I hope might spark off less strong alternatives that at least partially address the integrity + anti-forensics problem. If we can in some way introduce POLA to a wider forensics public, other tools might also pick up on it what would be great.
  • Mark I think it would be very handy to have a CSV, TSV, XML or other delimited output (log)file with information about carved files. This output file can then be stored in a database or Excel sheet (report function)

Format syntax specification

  • Carving data structures. For example, extract all TCP headers from image by defining TCP header structure and some fields (e.g. source port > 1024, dest port = 80). This will extract all data matching the pattern and write a file with other fields. Another example is carving INFO2 structures and URL activity records from index.dat .FUF 20:51, 28 October 2008 (UTC)
    • This has the opportunity to be extended to the concept of "point at blob FOO and interpret it as BAR"

.FUF added: The main idea is to allow users to define structures, for example (in pascal-like form):

Field1: Byte = 123;
SomeTextLength: DWORD;
SomeText: string[SomeTextLength];
Field4: Char = 'r';
...

This will produce something like this:

Field1 = 123
SomeTextLength = 5
SomeText = 'abcd1'
Field4 = 'r'

(In text or raw forms.)

Opinions?

Opinion: Simple pattern identification like that may not suffice, I think Simson's original intent was not only to identify but to allow for validation routines (plugins, as the original wording was). As such, the format syntax would need to implement a large chunk of some programming language in order to be sufficiently flexible. RB

Joachim In my option your example is too limited. Making the revit configuration I learned you'll need a near programming language to specify some file formats. A simple descriptive language is too limiting. I would also go for 2 bytes with endianess instead of using terminology like WORD and small integer, it's much more clear. The configuration also needs to deal with aspects like cardinality, required and optional structures.

This is simply data structures carving, see ideas above. Somebody (I cannot track so many changes per day) separated the original text. There is no need to count and join different structures. .FUF 19:53, 31 October 2008 (UTC)
Joachim This was probably me is the text back in it's original form?
I started it by moving your Revit07 comment to the validator/plugin section in this edit, since I was still at that point thinking operational configuration for that section, not parser configurations. RB
Joachim I renamed the title to format syntax, clarity is important ;-)

Please take a look at the revit07 configuration. It's not there yet but goes a far way. Some things currently missing:

  • bitwise alignment
  • handling encapsulated streams (MPEG/capture files)
  • handling content based formats (MBOX)

Caving algorithm

Joachim

  • should we allow for multiple carving phases (runs/stages)?
I opt yes (separation of concern)
  • should we allow for multiple carving algorithms?
I opt yes, this allows testing of different approaches
  • Should the algorithm try to do as much in 1 run over the input data? To reduce IO?
I opt that the tool should allow for multiple and single run over the input data to minimize the IO or the CPU as bottleneck
  • Interaction between algorithm and validators
    • does the algorithm passes data blocks to the validators?
    • does a validator need to maintain a state?
    • does a validator need to revert a state?
    • How do we deal with embedded files and content validation? Do the validators call another validator?
  • do we use the assumption that a data block can be used by a single file (with the exception of embedded/encapsulated files)?
  • Revit07 allows for multiple concurrent result files states to deal with fragmentation. One has the attribute of being active (the preferred) and the other passive. Do we want/need something similar? The algorithm adds block of input data (offsets) to these result files states.
    • if so what info would these result files states require (type, list of input data blocks)
  • how do we deal with file system remainders?
    • Can we abstract them and compare them against available file system information?
  • Do we carve file systems in files?
I opt that at least the validator uses this information

Caving scenarios

Joachim

  • normal file (file structure, loose text based structure (more a content structure?))
  • fragmented file (the file entirely exist)
  • a file fragment (the file does not entirely exist)
  • intertwined file
  • encapsulated file (MPEG/network capture)
  • embedded file (JPEG thumbnail)
  • obfuscation ('encrypted' PFF) this also entails encryption and/or compression
  • file system in file

File System Awareness

Background: Why be File System Aware?

Advantages of being FS aware:

  • You can pick up sector allocation sizes
Joachim do you mean file system block sizes?
  • Some file systems may store things off sector boundaries. (ReiserFS with tail packing)
  • Increasingly file systems have compression (NTFS compression)
  • Carve just the sectors that are not in allocated files.

Tasks that would be required

Discussion

As noted above, TSK should be utilized as much as possible, particularly the filesystem-aware portion. If we want to identify filesystems outside of its supported set, it would be more worth our time to work on implementing them there than in the carver itself. RB
I guess this tool operates like Selective file dumper and can recover files in both ways (or not?). Recovering files by using carving can recover files in situations where sleuthkit does nothing (e.g. file on NTFS was deleted using ntfs-3g, or filesystem was destroyed or just unknown). And we should build the list of filesystems supported by carver, not by TSK. .FUF 07:08, 29 October 2008 (UTC)
This tool is still in the early planning stages (requirements discovery), hence few operational details (like precise modes of operation) have been fleshed out - those will and should come later. The justification for strictly using TSK for the filesystem-sensitive approach is simple: TSK has good filesystem APIs, and it would be foolish to create yet another standalone, incompatible implementation of filesystem(foo) when time would be better spent improving those in TSK, aiding other methods of analysis as well. This is the same reason individuals that have implemented several other carvers are participating: de-duplication of effort. RB

Joachim I would like to have the carver (recovery tool) also do recovery using file allocation data or remainders of file allocation data.

Joachim I would go as far to ask you all to look beyond the carver as a tool and look from the perspective of the carver as part of the forensic investigation process. In my eyes certain information needed/acquired by the carver could be also very useful investigative information i.e. what part of a hard disk contains empty sectors.

Supportive tooling

Joachim

  • validator (definitions) tester (detest in revit07)
  • tool to make configuration based definitions
  • post carving validation
  • the carver needs to provide support for fuse mount of carved files (carvfs)

Testing

Joachim

  • automated testing
  • test data

Validator Construction

Options:

  • Write validators in C/C++
    • Joachim you mean dedicated validators
  • Have a scripting language for writing them (python? Perl?) our own?
    • Joachim use easy to embed programming languages i.e. Phyton or Lua
  • Use existing programs (libjpeg?) as plug-in validators?
    • Joachim define a file structure api for this

Existing Code that we have

Joachim Please add any missing links

Documentation/Articles

  • DFRWS2006/2007 carving challenge results
  • DFRWS2008 paper on carving

Carvers

Possible file structure validator libraries

Input support

Volume/Partition support

File system support

  • TSK
  • photorec FS code
  • implementations of FS in Linux/BSD

Content support

Zero storage support

  • libcarvpath
  • carvfs

POLA

  • joe-e (java)
  • Emily (ocaml)
  • the E language
  • AppArmor
  • iptables/ipfw
  • minorfs
  • plash

Implementation Timeline

  1. gather the available resources/ideas/wishes/needs etc. (I guess we're in this phase)
  2. start discussing a high level design (in terms of algorithm, facilities, information needed)
    1. input formats facility
    2. partition/volume facility
    3. file system facility
    4. file format facility
    5. content facility
    6. how to deal with fragment detection (do the validators allow for fragment detection?)
    7. how to deal with recombination of fragments
    8. do we want multiple carving phases in light of speed/precision tradeoffs
  3. start detailing parts of the design
    1. Discuss options for a grammar driven validator?
    2. Hard-coded plug-ins?
    3. Which existing code can we use?
  4. start building/assembling parts of the tooling for a prototype
    1. Implement simple file carving with validation.
    2. Implement gap carving
  5. Initial Release
  6. Implement the threaded carving that .FUF is describing above.

Joachim Shouldn't multi threaded carving (MTC) not be part of the 1st version? The MT approach makes for different design decisions