Forensics Wiki - User contributions [en]

Talk:Fileobject

2011-08-23T10:21:18Z

Capibara:

===sugestion: Use carvpath anotations===
I see the use of colons in byte_runs content. Earlier versions of libcarvpath used such a notation, but cross platform concerns resulted in this being changed into the plus sign.
I would suggest that the [http://sourceforge.net/apps/mediawiki/carvpath/index.php?title=CarvPath_annotations CarvPath annotation format] may be a suitable alternative for annotating
byte_runs content. This is a straight forward and simple format, for example:

4096+53248_258048+28672_S1047552

Would be the way to annotate a byte_runs consisting of:

1) A 53248 byte fragment at offset 4096

2) A 28672 byte fragment at offset 258048

3) 1047552 bytes of sparse data.

I don't suggest adding long token notations. Just the '+', '_' and 'S' stuff, and possibly the '/' for nesting purposes.

Talk:Fileobject

2011-08-23T10:18:15Z

Capibara:

Talk:Fileobject

2011-08-23T09:25:31Z

Capibara: /* sugestion: Use carvpath anotations */

===sugestion: Use carvpath anotations===
I see the use of colons in byte_runs content. Earlier versions of libcarvpath used such a notation, but cross platform concerns resulted in this being changed into the plus sign.
I would suggest that the [http://sourceforge.net/apps/mediawiki/carvpath/index.php?title=CarvPath_annotations CarvPath annotation format] may be a suitable alternative for annotating
byte_runs content. This is a straight forward and simple format, for example:

4096+53248_258048+28672_S1047552

Would be the way to annotate a byte_runs consisting of:

1) A 53248 byte fragment at offset 4096
2) A 28672 byte fragment at offset 258048
3) 1047552 bytes of sparse data.

I don't suggest adding long token notations. Just the '+', '_' and 'S' stuff.

Talk:Fileobject

2011-08-23T09:24:42Z

Capibara:

===sugestion: Use carvpath anotations===
I see the use of colons in byte_runs content. Earlier versions of libcarvpath used such a notation, but cross platform concerns resulted in this being changed into the plus sign.
I would suggest that the [http://sourceforge.net/apps/mediawiki/carvpath/index.php?title=CarvPath_annotations | CarvPath annotation format] may be a suitable alternative for annotating
byte_runs content. This is a straight forward and simple format, for example:

4096+53248_258048+28672_S1047552

Would be the way to annotate a byte_runs consisting of:

1) A 53248 byte fragment at offset 4096
2) A 28672 byte fragment at offset 258048
3) 1047552 bytes of sparse data.

I don't suggest adding long token notations. Just the '+', '_' and 'S' stuff.

Talk:Fileobject

2011-08-23T09:15:14Z

Capibara: Created page with "===sugestion: Use carvpath anotations=== I see the use of colons in byte_runs content. Earlier versions of libcarvpath used such a notation, but cross platform concerns resulted ..."

LibCarvPath

2010-03-12T14:31:41Z

Capibara:

LibCarvPath is a library designed to be used by carving and file system analysis tools.
LibCarvPath allows fragments represented by offset and size to be combined in a CarvPath
annotation that take the form of file system paths. LibCarvPath addresses the limits of
file system paths by mapping extremely fragmented files to a uniquely identifying key in
a long-path database.

The following tools use LibCarvPath and/or CarvPath Annotations:

* [[CarvFs]]
* [[tsk-cp]]
* [[scalpelcp]]
* Modules for the [[Open Computer Forensics Architecture]] that use the [[OCFA treegraph API]].

Next to these, in [[Photorec]] work has started to include LibCarvPath support.

[http://sourceforge.net/apps/mediawiki/carvpath/ CarvPath wiki]

CarvFs

2010-03-12T14:30:58Z

Capibara:

CarvFs is a modular [[Fuse]] based user space file system on top op [[LibCarvPath]].
CarvFS makes CarvPath style annotations as used by LibCarvPath available as files.
Using CarvFs makes it possible to process carved entities as files without the need for copy-out.

CarvFs is modular with respect to access to image files.
The CarvFs distribution comes with a default module for access to (split) raw files.

A separate [[LibEwf]] module is available for access to ewf images.

[http://sourceforge.net/apps/mediawiki/carvpath/ CarvPath wiki]

CarvFs

2010-03-12T14:30:15Z

Capibara:

CarvFs is a modular [[Fuse]] based user space file system on top op [[LibCarvPath]].
CarvFS makes CarvPath style annotations as used by LibCarvPath available as files.
Using CarvFs makes it possible to process carved entities as files without the need for copy-out.

CarvFs is modular with respect to access to image files.
The CarvFs distribution comes with a default module for access to (split) raw files.

A separate [[LibEwf]] module is available for access to ewf images.

[https://sourceforge.net/apps/mediawiki/carvpath/index.php?title=Main_Page CarvPath wiki]

LibCarvPath

2009-10-28T09:06:46Z

Capibara:

Carver 2.0 Planning Page

2009-10-09T05:33:07Z

Capibara: /* Existing Code that we have */

This page is for planning Carver 2.0.

Please, do not delete text (ideas) here. Use something like this:

<pre>
<s>bad idea</s>
:: good idea
</pre>

This will look like:

<s>bad idea</s>
:: good idea

= License =

BSD-3.
:: [[User:Joachim Metz|Joachim]] library based validators could require other licenses
::: Make the other libraries plug-able. If you them, you use them. [[User:Simsong|Simsong]] 06:34, 3 November 2008 (UTC)

= OS =

Linux/FreeBSD/MacOS
: Shouldn't this just match what the underlying afflib & sleuthkit cover? [[User:RB|RB]]
:: Yes, but you need to test and validate on each. Question: Do we want to support windows? [[User:Simsong|Simsong]] 21:09, 30 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] I think we would do wise to design with windows support from the start this will improve the platform independence from the start
:::: Agreed; I would even settle at first for being able to run against Cygwin. Note that I don't even own or use a copy of Windows, but the vast majority of forensic investigators do. [[User:RB|RB]] 14:01, 31 October 2008 (UTC)
:: [[User:Capibara|Rob J Meijer]] Leaning heavily on the autotools might be the way to go. I do however feel that support requirements for windows would not be essential. Being able to run from a virtual machine with the main storage mounted over cifs should however be tested and if possible tuned extensively.
:::: [[User:Joachim Metz|Joachim]] You'll need more than autotools to do native Windows support i.e. file access, UTF-16 support, wrap some basic system functions or have them available otherwise
::::::[[User:Capibara|Rob J Meijer]] That´s exactly my point, windows support as in being able to build and run on windows natively is much more trouble than its worth. Better make for a lean and mean autotools based build with little dependencies and no or little recursion, and better spent effort on a lean POLA design on POSIX based systems than on supporting building and running on non POSIX systems.

= Name tooling =

* [[User:Joachim Metz|Joachim]] A name for the tooling I propose coldcut
:: How about 'butcher'? ;) [[User:RB|RB]] 14:20, 31 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] cleaver ( scalpel on steroids ;-) )
* I would like to propose Gouge or Chisel :-) [[User:Capibara|Rob J Meijer]]

= Requirements =

[[User:Joachim Metz|Joachim]] Could we do a MoSCoW evaluation of these.
* AFF and EWF file images supported from scratch. ([[User:Joachim Metz|Joachim]] I would like to have raw/split raw and device access as well)
:: If we base our image i/o on afflib, we get all three with one interface. [[User:RB|RB]] Instead of letting the tools use afflib, better to write an afflib module for carvfs, and update the libewf module. The tool could than be oblivious of the file format. [[User:Capibara|Rob J Meijer]]
:::: [[User:Simsong|Simsong]] 06:29, 3 November 2008 (UTC) The problem with using carvfs is that this adds another dependency. Do you really want to require that people install carvfs in order to run the carver? What about having the thing ported to Windows?
:::::: [[User:Capibara|Rob J Meijer]] I would support adding one build dependency (libcarvpath) and removing two (libewf/libaff) by moving them to a layer more suited for them (carvfs) that would possibly allow some form of file handle (as cap) based POLA design. I am a proponent of making small things that do one and do one thing right, and to stack those to do what you need. In my view that would lead ideally to the following (simplified) chain:
::::::* recursive [[computer forensics framework]] (ocfa/pyflag)
::::::** The-(pola-based)-carving-tool
::::::*** The-carving-lib working on open fd's.
::::::**** libcarvpath
::::::***** carvfs (Over cifs/nfs-v4 on platforms that don't support Fuse).
::::::****** libewf
::::::****** libaff
::::::*** AppArmor (on supporting platforms)
::::::*** suid (on supporting platforms)
::::::*** iptables/ipfw (on supporting platforms)
:::::: As fow windows support, I would imagine making carvfs run over smb would come a long way, that is for as far as windows support is all that relevant.
:::::: There are two advantages to using libcarvpath and carvfs instead of libaff/libewf t this layer:
::::::* storage requirements for doing carving. Beyond what sleuthkit or alternatives provide I have seen many situations where carving was not done due to storage limitations.
::::::* File handles are like object capabilities. You can often do pretty simple POLA based implementations using file handles and something like AppArmor. POLA could IMHO be a strong weapon against the more nasty forms of anti forensics.
::::::Next to this, I would consider making different tools for different stages instead of one semi recursive one, and looking at how to integrate these tools into existing frameworks (ocfa/pyflag).
::::::Keep things simple but rigid and try to easily integrate things into existing frameworks as effectively as possible I would suggest.
::::::Please note, I am not ptoposing the lib/tool should be useless without libcarvpath, only that usage without carvfs should limit the
::::::supported image formats to raw images, and that libewf/libaff should be abstracted at the Fuse level or below and not at the tool level.
:::::::[[User:Joachim Metz|Joachim]] do you have an idea what the performance impact of this approach would be? It might be wise to do a proof of concept for this approach first.
::::::::[[User:Capibara|Rob J Meijer]] It would I think depend greatly on behavior of the carving lib/tool. Small 512 byte reads are relatively very expensive, 128kb reads have negligible impact. Here are some numbers from ntfs-3g: http://article.gmane.org/gmane.comp.file-systems.fuse.devel/6397/match=ntfs+3g+performance+ext3 that might be relevant. More relevant than performance might be library footprint. For example, using OCFA, we often would want to keep e few hundred images that total to tens of TB of projected storage size fuse mounted. If libewf/libaff have a big combined memory footprint in such cases, this can be a major issue for this approach.
:::: [[User:Joachim Metz|Joachim]] this layer should support multi threaded decompression of compressed image types, this speeds up IO
* [[User:Joachim Metz|Joachim]] volume/partition aware layer (what about carving unpartioned space)
* File system aware layer. This could be or make use of tsk-cp.
** By default, files are not carved. (clarify: only identified? [[User:RB|RB]]; I guess that it operates like [[Selective file dumper]] [[User:.FUF|.FUF]] 07:00, 29 October 2008 (UTC)). Alternatively, the tool could use libcarvpath and output carvpaths or create a directory with symlinks to carvpaths that point into a carvfs mountpoint [[User:Capibara|Rob J Meijer]].
* Plug-in architecture for identification/validation.
** [[User:Joachim Metz|Joachim]] support for multiple types of validators
*** dedicated validator
*** validator based on file library (i.e. we could specify/implement a file structure API for these)
*** configuration based validator (Can handle config files,like Revit07, to enter different file formats used by the carver.)

[[User:Joachim Metz|Joachim]]
Moderator: Could we limit the requirements for prototype version 1 of the tool to get a working version up and running ASAP?
And keep discussing future options?

I think the following set will be large enough to handle:
Input facilities
* IO support (AFF, device, EWF, RAW and split RAW)
:: Abstraction of input format and multi threaded decompression (spin-off code out of afflib?)
* Volume/Partitions support
:: at least for DOS based layout and GPT (spin-off code out of TSK/Photorec?)
* File system support
:: VFAT/NTFS (spin-off code out of TSK/Photorec?)

Carving facilities
* File format support using plug-able validator model (use dedicated validators Photorec/Scarve and/or wrap revit07 file format as validator?)
* Content support using plug-able validator model (to handle text/mbox base64)
* File system carving support (to handle file system fragments, could be linked to file system support layer?)
* Basic fragment handling

Output facilities
* audit/analysis/debug log
* extraction of result files

==Supported File Formats==
Ship with validators for
:: [[User:Joachim Metz|Joachim]] I think we should distinguish between file format validators and content validators

* Grapical Images
** JPEG (the 3 different types with JFIF/EXIF support)
:: [[User:Joachim Metz|Joachim]] How different is JPEG 2000?
** PNG
** GIF
** BMP
** TIFF
* Office documents
** Microsoft Office 97 - 2003 [[OLE Compound File]] format based with [[Word Document (DOC)]] and [[Excel Spreadsheet (XLS)]] file format support
** [[PDF]]
** Open Office and Microsoft Office 2007 [[ZIP archive]] based file formats
:: Extension validation? AFAIK, MS Office 2007 [[Word Document (DOCX)]] format uses plain ZIP (or not?), and carved files will (or not?) have .zip extension instead of DOCX. Is there any way to fix this (may be using the file list in zip)? [[User:.FUF|.FUF]] 20:25, 31 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] Addition: Office 2007 also has a binary file format which is also a ZIP-ed data [[Excel Spreadsheet (XLSB)]]
* Archive files
* [[ZIP archive]] file format
** 7z
** tar, gzip, bzip2
** RAR
* E-mail files
** [[Personal Folder File (PAB, PST, OST)]]
** MBOX (text based format, base64 content support)
* Audio/Video files
** MPEG
** MP2/MP3
** AVI
** ASF/WMV
** QuickTime
** MKV
* Printer spool files
** EMF (if I remember correctly)
* Internet history files
** index.dat
** firefox (sqllite 3)
* Other files
** thumbs.db which also is an [[OLE Compound File]] based format
** pagefile?

==Carving Strategies==
[[User:Joachim Metz|Joachim]] Note to moderator could this section be merged with the carving algorithm section?

* Simple fragment recovery carving using gap carving.
** [[User:Joachim Metz|Joachim]] have hook in for more advanced fragment recovery?
* Recovering of individual ZIP sections and JPEG icons that are not sector aligned.
** [[User:Joachim Metz|Joachim]] I would propose a generic fragment detection and recovery
* Autonomous operation (some mode of operation should be completely non-interactive, requiring no human intervention to complete [[User:RB|RB]])
** [[User:Joachim Metz|Joachim]] as much as possible, but allow to be overwritten by user
* [[User:Joachim Metz|Joachim]] When the tool output files the filenames should contain the offset in the input data (in hexadecimal?)
:: [[User:Mark Stam|Mark]] I really like the fact carved files are named after the physical or logical sector in which the file is found (photorec)
:::: [[User:Joachim Metz|Joachim]] This naming schema might cause duplicate name problem for extracting embedded files and extracting files from non sector aligned file systems.
* [[User:Joachim Metz|Joachim]] Should the tool allow to export embedded files?
* [[User:Joachim Metz|Joachim]] Should the tool allow to export fragments separately?
* [[User:Mark Stam|Mark]] I personally use photorec often for carving files in the whole volume (not only unallocated clusters), so I can store information about all potential interesting files in MySQL
:: [[User:Joachim Metz|Joachim]] interesting, Bas Kloet and me have been discussing to use information about allocated files in the recovery process, i.e. recovered fragments could be part of allocated files. Do we want to be able to extract them? Or could we rebuild the file from the fragments and the allocated files.
* [[User:Mark Stam|Mark]] It would also be nice if the files can be hashed immediately (MD5) so looking for them in other tools (for example Encase) is a snap

==Performance Requirements==
* Tested on 500GB-sized images. Should be able to carve a 500GB image in roughly 50% longer than it takes to read the image.
** Perhaps allocate a percentage budget per-validator (i.e. each validator adds N% to the carving time) [[User:RB|RB]]
** [[User:Joachim Metz|Joachim]] have multiple carving phases for precision/speed trade off?
* Parallelizable
** [[User:Joachim Metz|Joachim]] tunable for different architectures
* Configuration:
** Capability to parse some existing carvers' configuration files, either on-the-fly or as a one-way converter.
** Disengage internal configuration structure from configuration files, create parsers that present the expected structure
** [[User:Joachim Metz|Joachim]] The validator should deal with the file structure the carving algorithm should not know anything about the file structure (as in revit07 design)
** Either extend Scalpel/Foremost syntaxes for extended features or use a tertiary syntax ([[User:Joachim Metz|Joachim]] I would prefer a derivative of the revit07 configuration syntax which already has encountered some problems of dealing with defining file structure in a configuration file)

==Output==
* Can output audit.txt file.
* [[User:Joachim Metz|Joachim]] Can output database with offset analysis values i.e. for visualization tooling
* [[User:Joachim Metz|Joachim]] Can output debug log for debugging the algorithm/validation
* Easy integration into ascription software.
:: [[User:Joachim Metz|Joachim]] I'm no native speaker what do you mean with "ascription software"?
::: I think this was another non-native requesting easy scriptability. [[User:RB|RB]] 14:20, 31 October 2008 (UTC)
:::: [[User:Joachim Metz|Joachim]] that makes sense ;-)
::::: Incorrect. Ascription software is software that determines who the owner of a file is. [[User:Simsong|Simsong]] 06:36, 3 November 2008 (UTC)

= Ideas =
* Use as much TSK if possible. Don't carry your own FS implementation the way photorec does.
:: [[User:Joachim Metz|Joachim]] using TSK as much as possible would not allow to add your own file system support (i.e. mobile phones, memory structures, cap files) I would propose wrapping TSK and using it as much as possible but allow to integrate own FS implementations.
::::[[User:Capibara|Rob J Meijer]] I'm currently working on wrapping TSK filesystem as several loadable modules for OCFA. In OCFA a loadable (tree-graph) module implements the OCFA tree-graph API that basically states 'everything is a tree-graph node'. Possibly you could look at the OCFA treegraph API and module loading interface as an example, or we could work together on changing the API and module loading interface in such a way that it doesn't break OCFA and is usefull for the stand alone carver, but allowing both to use exactly the same tree-graph loadable modules.
* Extracting/carving data from [[Thumbs.db]]? I've used [[foremost]] for it with some success. [[Vinetto]] has some critical bugs :( [[User:.FUF|.FUF]] 19:18, 28 October 2008 (UTC)
==Recursive Carving==
[[User:Joachim Metz|Joachim]] do we want to support (let's call it) 'recursive in file carving' (for now) this is different from embedded files because there is a file system structure in the file and not just another file structure
* Is it just me, or do a lot of the above (and below) ideas somewhat skirt around the fact that many of us want recursive carving? Can we bend back to that instead of discussing object particulars? I think this can be distilled down to three requirements:
** Simple recursion: once an object is identified, have the ability to re-carve it for internal structures
** Directed recursion: the carver should be able to be directed at arbitrary blobs and told to carve it as a specified type. This allows programmatically more simple methods of dealing with unidentifiably compressed or encrypted data. Or filesystem fragments.
** Export: the ability to export an object (recognized or not) for later or external "recursion". Should go without saying for a carver, but...
:--[[User:RB|RB]] 18:45, 2 November 2008 (UTC)
:: [[User:Simsong|Simsong]] 06:30, 3 November 2008 (UTC) pyflag already does recursive carving. Are we just going to reimplement pyflag as a single executable?
::: [[User:Joachim Metz|Joachim]] that could be useful ;-)

==Library Dependencies==
[[User:Capibara|Rob J Meijer]] :
* Use libcarvpath whenever possible and by default to avoid high storage requirements.
:: [[User:Joachim Metz|Joachim]] For easy deployment I would not opt for making an integral part of the tool solely dependant on a single external library or the library must be integrated in the package
::[[User:Capibara|Rob J Meijer]] Integrating libraries (libtsk,libaff.libewf,libcarvpath etc) is bad practice, autotools are your friend IMO.
:: [[User:Joachim Metz|Joachim]] I'm not talking about integrating (shared) libraries. I'm talking about that an integral part of a tool should be part of it's package. Why can't the tool package contain shared or static libraries for local use? A far worse thing to do is to have a large set of dependencies and making the tool difficult to install for most users. The tool package should contain the most necessary code. afflib/libewf support could be detected by the autotools a neat separation of functionality.
::: From a packager's standpoint, [[User:Joachim Metz|Joachim]]'s other libraries do a really good job of this, carrying around what they need but using a system-global version if available. [[User:RB|RB]]
* libtsk
* libaff ? : possibly the discussion in the requirements section should move to his section.
* libewf ? : possibly the discussion in the requirements section should move to his section.
* posix ? : Can we depend especially on the availability of UNIX domain sockets and the possibility to use msg_accrights for passing opn file handles as ocaps?

==Filesystem Detection==
* Dont stop with filesystem detection after the first match. Often if a partition is reused with a new FS and is not all that full yet, much of the old FS can still be valid. I have seen this with ext2/fat. The fact that you have identified a valid FS on a partition doesn't mean there isn't an(almost) valid second FS that would yield additional files. Identifying doubly allocated space might in some cases also be relevant.
:: [[User:Joachim Metz|Joachim]] What your saying is that dealing with file system fragments should be part of the carving algorithm
* Allow use where filesystem based carving is done by other tool, and the tool is used as second stage on (sets of) unallocated block (pseudo) files and/or non FS partition (pseudo) files.
:: [[User:Joachim Metz|Joachim]] I would not opt for this. The tool would be dependent on other tools and their data format, which makes the tool difficult to maintain. I would opt to integrate the functionality of having multiple recovery phases (stages) and allow the tooling to run the phases after one and other or separately.
::[[User:Capibara|Rob J Meijer]] More generically, I feel a way should exist to communicate the 'left overs' a previous (non open, for example LE-only) tool left.
:: [[User:Joachim Metz|Joachim]] I guess if the tool is designed to handle multiple phases it should store its data somewhere. So it should be possible to convert results of such non open tooling to the format required. However I would opt to design the recovery functionality of these non-open tools into open tools. And not to limit ourselves making translators due to the design of these non-open tools.
* Ability to be used as a library instead of a tool. Ability to access metadata true library, and thus the ability to set metadata from the carving modules. This would be extremely usefull for integrating the project into a [[computer forensics framework]] .
:: [[User:Joachim Metz|Joachim]] I guess most of the code could be integrated into libraries, but I would not opt integrating tool functionality into a library
* [[User:Mark Stam|Mark]] I think it would be very handy to have a CSV, TSV, XML or other delimited output (log)file with information about carved files. This output file can then be stored in a database or Excel sheet (report function)

==Anti forensics and system integrity concerns==
* It might be very interesting to look at the possibilities of using a multi process style of module support and combine it with a least authority design. On platforms that support AppArmor (or similar) and uid based firewall rules, this could make for the first true POLA (principle of least authority) based forensic tool ever. POLA based forensics tools should make for a strong integrity guard against many anti forensics. Alternatively we could look at integrating a capability secure language (E?) for implementation of at least validation modules. I don't expect this idea to make it, but mentioning it I hope might spark off less strong alternatives that at least partially address the integrity + anti-forensics problem. If we can in some way introduce POLA to a wider forensics public, other tools might also pick up on it what would be great.
:: [[User:Joachim Metz|Joachim]] Could you give an example of how you see this in action?
::::[[User:Capibara|Rob J Meijer]] I see two layers where using POLA could be applied. The best one would require one of the folowing as prerequisites:
::::* The libaff/libewf layer is moved to a fuse implementation (for example carvfs).
::::* Libewf/Libaff are updated to accept opened filhandles instead of demanding to open their own files.
::::If one of these is fulfilled, than the tool running as some user can just have the simple task of opening the image files, starting up the 'real' tool and handing over the appropriate file handles. If the real tool runs with a restrictive AppArmor profile, and is started suid to a tool specific user that also has its own iptables uid based filter, than the real tool will run with least authority.
:::: A second alternative, if neither of the first prerequisite could not be bet, would be to run the modules as confined processes and have a non confined process run as proxy for the first.
:::: A third probably far fetched alternative would be to embed an object capability language in the tool and make the module interface thus that modules are to be written in this ocap language.
::::A 4th alternative might include minorfs or plash, but I havn't geven those sufficient thinking hours yet.

== Format syntax specification ==
* Carving data structures. For example, extract all TCP headers from image by defining TCP header structure and some fields (e.g. source port > 1024, dest port = 80). This will extract all data matching the pattern and write a file with other fields. Another example is carving INFO2 structures and URL activity records from index.dat [[User:.FUF|.FUF]] 20:51, 28 October 2008 (UTC)
** This has the opportunity to be extended to the concept of "point at blob FOO and interpret it as BAR"
.FUF added:
The main idea is to allow users to define structures, for example (in pascal-like form):

<pre>
Field1: Byte = 123;
SomeTextLength: DWORD;
SomeText: string[SomeTextLength];
Field4: Char = 'r';
...
</pre>

This will produce something like this:
<pre>
Field1 = 123
SomeTextLength = 5
SomeText = 'abcd1'
Field4 = 'r'
</pre>

(In text or raw forms.)

Opinions?

Opinion: Simple pattern identification like that may not suffice, I think Simson's original intent was not only to identify but to allow for validation routines (plugins, as the original wording was). As such, the format syntax would need to implement a large chunk of some programming language in order to be sufficiently flexible. [[User:RB|RB]]

[[User:Joachim Metz|Joachim]]
In my option your example is too limited. Making the revit configuration I learned you'll need a near programming language to specify some file formats.
A simple descriptive language is too limiting. I would also go for 2 bytes with endianess instead of using terminology like WORD and small integer, it's much more clear. The configuration also needs to deal with aspects like cardinality, required and optional structures.
:: This is simply data structures carving, see ideas above. Somebody (I cannot track so many changes per day) separated the original text. There is no need to count and join different structures. [[User:.FUF|.FUF]] 19:53, 31 October 2008 (UTC)
:::: [[User:Joachim Metz|Joachim]] This was probably me is the text back in it's original form?
:::: I started it by moving your Revit07 comment to the validator/plugin section in [http://www.forensicswiki.org/index.php?title=Carver_2.0_Planning_Page&diff=prev&oldid=7583 this edit], since I was still at that point thinking operational configuration for that section, not parser configurations. [[User:RB|RB]]
:::: [[User:Joachim Metz|Joachim]] I renamed the title to format syntax, clarity is important ;-)

Please take a look at the revit07 format syntax specification (configuration). It's not there yet but goes a far way. Some things currently missing:
* bitwise alignment
* handling encapsulated streams (MPEG/capture files)
* handling content based formats (MBOX)

=Caving algorithm =
[[User:Joachim Metz|Joachim]]
* should we allow for multiple carving phases (runs/stages)?
:: I opt yes (separation of concern)
* should we allow for multiple carving algorithms?
:: I opt yes, this allows testing of different approaches
* Should the algorithm try to do as much in 1 run over the input data? To reduce IO?
:: I opt that the tool should allow for multiple and single run over the input data to minimize the IO or the CPU as bottleneck
* Interaction between algorithm and validators
** does the algorithm passes data blocks to the validators?
** does a validator need to maintain a state?
** does a validator need to revert a state?
** How do we deal with embedded files and content validation? Do the validators call another validator?
* do we use the assumption that a data block can be used by a single file (with the exception of embedded/encapsulated files)?
* Revit07 allows for multiple concurrent result files states to deal with fragmentation. One has the attribute of being active (the preferred) and the other passive. Do we want/need something similar? The algorithm adds block of input data (offsets) to these result files states.
** if so what info would these result files states require (type, list of input data blocks)
* how do we deal with file system remainders?
** Can we abstract them and compare them against available file system information?
* Do we carve file systems in files?
:: I opt that at least the validator uses this information

==Caving scenarios ==
[[User:Joachim Metz|Joachim]]
* normal file (file structure, loose text based structure (more a content structure?))
* fragmented file (the file entirely exist)
* a file fragment (the file does not entirely exist)
* intertwined file
* encapsulated file (MPEG/network capture)
* embedded file (JPEG thumbnail)
* obfuscation ('encrypted' PFF) this also entails encryption and/or compression
* file system in file

=File System Awareness =
==Background: Why be File System Aware?==
Advantages of being FS aware:
* You can pick up sector allocation sizes
:: [[User:Joachim Metz|Joachim]] do you mean file system block sizes?
* Some file systems may store things off sector boundaries. (ReiserFS with tail packing)
* Increasingly file systems have compression (NTFS compression)
:: [[User:Joachim Metz|Joachim]] Carving NTFS-compressed (lznt1) files (http://sourceforge.net/projects/revit/files/Documentation/Carving%20NTFS-compressed%20data/Carving%20for%20NTFS%20compressed%20files.pdf/download)
* Carve just the sectors that are not in allocated files.
:: [[User:Joachim Metz|Joachim]] sparse (file system) blocks e.g. NTFS cluster blocks

==Tasks that would be required==

==Discussion==
:: As noted above, TSK should be utilized as much as possible, particularly the filesystem-aware portion. If we want to identify filesystems outside of its supported set, it would be more worth our time to work on implementing them there than in the carver itself. [[User:RB|RB]]

:::: I guess this tool operates like [[Selective file dumper]] and can recover files in both ways (or not?). Recovering files by using carving can recover files in situations where sleuthkit does nothing (e.g. file on NTFS was deleted using ntfs-3g, or filesystem was destroyed or just unknown). And we should build the list of filesystems supported by carver, not by TSK. [[User:.FUF|.FUF]] 07:08, 29 October 2008 (UTC)

:: This tool is still in the early planning stages (requirements discovery), hence few operational details (like precise modes of operation) have been fleshed out - those will and should come later. The justification for strictly using TSK for the filesystem-sensitive approach is simple: TSK has good filesystem APIs, and it would be foolish to create yet another standalone, incompatible implementation of filesystem(foo) when time would be better spent improving those in TSK, aiding other methods of analysis as well. This is the same reason individuals that have implemented several other carvers are participating: de-duplication of effort. [[User:RB|RB]]

:: [[User:Joachim Metz|Joachim]] A design problem might be that TSK currently is a single library operating on multiple layers (storage media IO, volume/partition analysis and file system analysis). I'm not aware how easily the parts can be used separately. But I estimate that for the carver we want to use these 3 layers differently than TSK currently does.

[[User:Joachim Metz|Joachim]] I would like to have the carver (recovery tool) also do recovery using file allocation data or remainders of file allocation data.

[[User:Joachim Metz|Joachim]]
I would go as far to ask you all to look beyond the carver as a tool and look from the perspective of the carver as part of the forensic investigation process. In my eyes certain information needed/acquired by the carver could be also very useful investigative information i.e. what part of a hard disk contains empty sectors.

=Supportive tooling=
[[User:Joachim Metz|Joachim]]
* validator (definitions) tester (detest in revit07)
* tool to make configuration based definitions
* post carving validation
* the carver needs to provide support for fuse mount of carved files (carvfs)

=Testing =
[[User:Joachim Metz|Joachim]]
* automated testing
* test data

=Validator Construction=
Options:
* Write validators in C/C++
** [[User:Joachim Metz|Joachim]] you mean dedicated validators
* Have a scripting language for writing them (python? Perl?) our own?
** [[User:Joachim Metz|Joachim]] use easy to embed programming languages i.e. Phyton or Lua
* Use existing programs (libjpeg?) as plug-in validators?
** [[User:Joachim Metz|Joachim]] define a file structure api for this

=Existing Code that we have=
[[User:Joachim Metz|Joachim]]
Please add any missing links

Documentation/Articles
* DFRWS2006/2007 carving challenge results
* DFRWS2008 paper on carving

Carvers
* DFRWS2006/2007 carving challenge results
* photorec (http://www.cgsecurity.org/wiki/PhotoRec)
* revit06 and revit07 (http://sourceforge.net/projects/revit/)
* s3/scarve

Possible file structure validator libraries
* divers existing file support libraries
* libole2 (inhouse experimental code of OLE2 support)
* [[libnk2]]
* [[libpff]]

Input support
* AFF (http://www.afflib.org/)
* [[libewf]]
* TSK device & raw & split raw (http://www.sleuthkit.org/)

Volume/Partition support
* disktype (http://disktype.sourceforge.net/)
* testdisk (http://www.cgsecurity.org/wiki/TestDisk)
* TSK

File system support
* TSK
* photorec FS code
* implementations of FS in Linux/BSD
* The tree-graph loadable module support, module loader and loadable modules of the Open Computer Forensics Architecture.

Content support

Zero storage support
* libcarvpath ( http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=210704 )
* carvfs ( http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=210954 )
* tsk-cp ( http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=267227 )
* carvfsmodewf (http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=268256 )
POLA
* joe-e (java) ( http://code.google.com/p/joe-e/ )
* Emily (ocaml) ( http://erights.org/download/emily/ )
* the E language ( http://www.erights.org/ )
* AppArmor
* iptables/ipfw
* minorfs ( http://polacanthus.net/minorfs.html )
* plash ( http://plash.beasts.org/wiki/ )

=Implementation Timeline=
# gather the available resources/ideas/wishes/needs etc. (I guess we're in this phase)
# start discussing a high level design (in terms of algorithm, facilities, information needed)
## input formats facility
## partition/volume facility
## file system facility
## file format facility
## content facility
## how to deal with fragment detection (do the validators allow for fragment detection?)
## how to deal with recombination of fragments
## do we want multiple carving phases in light of speed/precision tradeoffs
# start detailing parts of the design
## Discuss options for a grammar driven validator?
## Hard-coded plug-ins?
## Which existing code can we use?
# start building/assembling parts of the tooling for a prototype
## Implement simple file carving with validation.
## Implement gap carving
# Initial Release
# Implement the ''threaded carving'' that [[User:.FUF|.FUF]] is describing above.

[[User:Joachim Metz|Joachim]] Shouldn't multi threaded carving (MTC) not be part of the 1st version?
The MT approach makes for different design decisions
: It is virtually impossible to turn a non-MT application into an MT application .[[User:Simsong|Simsong]] 06:37, 3 November 2008 (UTC)

Carver 2.0 Planning Page

2009-10-09T05:31:25Z

Capibara: /* Ideas */

This page is for planning Carver 2.0.

Please, do not delete text (ideas) here. Use something like this:

<pre>
<s>bad idea</s>
:: good idea
</pre>

This will look like:

<s>bad idea</s>
:: good idea

= License =

BSD-3.
:: [[User:Joachim Metz|Joachim]] library based validators could require other licenses
::: Make the other libraries plug-able. If you them, you use them. [[User:Simsong|Simsong]] 06:34, 3 November 2008 (UTC)

= OS =

Linux/FreeBSD/MacOS
: Shouldn't this just match what the underlying afflib & sleuthkit cover? [[User:RB|RB]]
:: Yes, but you need to test and validate on each. Question: Do we want to support windows? [[User:Simsong|Simsong]] 21:09, 30 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] I think we would do wise to design with windows support from the start this will improve the platform independence from the start
:::: Agreed; I would even settle at first for being able to run against Cygwin. Note that I don't even own or use a copy of Windows, but the vast majority of forensic investigators do. [[User:RB|RB]] 14:01, 31 October 2008 (UTC)
:: [[User:Capibara|Rob J Meijer]] Leaning heavily on the autotools might be the way to go. I do however feel that support requirements for windows would not be essential. Being able to run from a virtual machine with the main storage mounted over cifs should however be tested and if possible tuned extensively.
:::: [[User:Joachim Metz|Joachim]] You'll need more than autotools to do native Windows support i.e. file access, UTF-16 support, wrap some basic system functions or have them available otherwise
::::::[[User:Capibara|Rob J Meijer]] That´s exactly my point, windows support as in being able to build and run on windows natively is much more trouble than its worth. Better make for a lean and mean autotools based build with little dependencies and no or little recursion, and better spent effort on a lean POLA design on POSIX based systems than on supporting building and running on non POSIX systems.

= Name tooling =

* [[User:Joachim Metz|Joachim]] A name for the tooling I propose coldcut
:: How about 'butcher'? ;) [[User:RB|RB]] 14:20, 31 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] cleaver ( scalpel on steroids ;-) )
* I would like to propose Gouge or Chisel :-) [[User:Capibara|Rob J Meijer]]

= Requirements =

[[User:Joachim Metz|Joachim]] Could we do a MoSCoW evaluation of these.
* AFF and EWF file images supported from scratch. ([[User:Joachim Metz|Joachim]] I would like to have raw/split raw and device access as well)
:: If we base our image i/o on afflib, we get all three with one interface. [[User:RB|RB]] Instead of letting the tools use afflib, better to write an afflib module for carvfs, and update the libewf module. The tool could than be oblivious of the file format. [[User:Capibara|Rob J Meijer]]
:::: [[User:Simsong|Simsong]] 06:29, 3 November 2008 (UTC) The problem with using carvfs is that this adds another dependency. Do you really want to require that people install carvfs in order to run the carver? What about having the thing ported to Windows?
:::::: [[User:Capibara|Rob J Meijer]] I would support adding one build dependency (libcarvpath) and removing two (libewf/libaff) by moving them to a layer more suited for them (carvfs) that would possibly allow some form of file handle (as cap) based POLA design. I am a proponent of making small things that do one and do one thing right, and to stack those to do what you need. In my view that would lead ideally to the following (simplified) chain:
::::::* recursive [[computer forensics framework]] (ocfa/pyflag)
::::::** The-(pola-based)-carving-tool
::::::*** The-carving-lib working on open fd's.
::::::**** libcarvpath
::::::***** carvfs (Over cifs/nfs-v4 on platforms that don't support Fuse).
::::::****** libewf
::::::****** libaff
::::::*** AppArmor (on supporting platforms)
::::::*** suid (on supporting platforms)
::::::*** iptables/ipfw (on supporting platforms)
:::::: As fow windows support, I would imagine making carvfs run over smb would come a long way, that is for as far as windows support is all that relevant.
:::::: There are two advantages to using libcarvpath and carvfs instead of libaff/libewf t this layer:
::::::* storage requirements for doing carving. Beyond what sleuthkit or alternatives provide I have seen many situations where carving was not done due to storage limitations.
::::::* File handles are like object capabilities. You can often do pretty simple POLA based implementations using file handles and something like AppArmor. POLA could IMHO be a strong weapon against the more nasty forms of anti forensics.
::::::Next to this, I would consider making different tools for different stages instead of one semi recursive one, and looking at how to integrate these tools into existing frameworks (ocfa/pyflag).
::::::Keep things simple but rigid and try to easily integrate things into existing frameworks as effectively as possible I would suggest.
::::::Please note, I am not ptoposing the lib/tool should be useless without libcarvpath, only that usage without carvfs should limit the
::::::supported image formats to raw images, and that libewf/libaff should be abstracted at the Fuse level or below and not at the tool level.
:::::::[[User:Joachim Metz|Joachim]] do you have an idea what the performance impact of this approach would be? It might be wise to do a proof of concept for this approach first.
::::::::[[User:Capibara|Rob J Meijer]] It would I think depend greatly on behavior of the carving lib/tool. Small 512 byte reads are relatively very expensive, 128kb reads have negligible impact. Here are some numbers from ntfs-3g: http://article.gmane.org/gmane.comp.file-systems.fuse.devel/6397/match=ntfs+3g+performance+ext3 that might be relevant. More relevant than performance might be library footprint. For example, using OCFA, we often would want to keep e few hundred images that total to tens of TB of projected storage size fuse mounted. If libewf/libaff have a big combined memory footprint in such cases, this can be a major issue for this approach.
:::: [[User:Joachim Metz|Joachim]] this layer should support multi threaded decompression of compressed image types, this speeds up IO
* [[User:Joachim Metz|Joachim]] volume/partition aware layer (what about carving unpartioned space)
* File system aware layer. This could be or make use of tsk-cp.
** By default, files are not carved. (clarify: only identified? [[User:RB|RB]]; I guess that it operates like [[Selective file dumper]] [[User:.FUF|.FUF]] 07:00, 29 October 2008 (UTC)). Alternatively, the tool could use libcarvpath and output carvpaths or create a directory with symlinks to carvpaths that point into a carvfs mountpoint [[User:Capibara|Rob J Meijer]].
* Plug-in architecture for identification/validation.
** [[User:Joachim Metz|Joachim]] support for multiple types of validators
*** dedicated validator
*** validator based on file library (i.e. we could specify/implement a file structure API for these)
*** configuration based validator (Can handle config files,like Revit07, to enter different file formats used by the carver.)

[[User:Joachim Metz|Joachim]]
Moderator: Could we limit the requirements for prototype version 1 of the tool to get a working version up and running ASAP?
And keep discussing future options?

I think the following set will be large enough to handle:
Input facilities
* IO support (AFF, device, EWF, RAW and split RAW)
:: Abstraction of input format and multi threaded decompression (spin-off code out of afflib?)
* Volume/Partitions support
:: at least for DOS based layout and GPT (spin-off code out of TSK/Photorec?)
* File system support
:: VFAT/NTFS (spin-off code out of TSK/Photorec?)

Carving facilities
* File format support using plug-able validator model (use dedicated validators Photorec/Scarve and/or wrap revit07 file format as validator?)
* Content support using plug-able validator model (to handle text/mbox base64)
* File system carving support (to handle file system fragments, could be linked to file system support layer?)
* Basic fragment handling

Output facilities
* audit/analysis/debug log
* extraction of result files

==Supported File Formats==
Ship with validators for
:: [[User:Joachim Metz|Joachim]] I think we should distinguish between file format validators and content validators

* Grapical Images
** JPEG (the 3 different types with JFIF/EXIF support)
:: [[User:Joachim Metz|Joachim]] How different is JPEG 2000?
** PNG
** GIF
** BMP
** TIFF
* Office documents
** Microsoft Office 97 - 2003 [[OLE Compound File]] format based with [[Word Document (DOC)]] and [[Excel Spreadsheet (XLS)]] file format support
** [[PDF]]
** Open Office and Microsoft Office 2007 [[ZIP archive]] based file formats
:: Extension validation? AFAIK, MS Office 2007 [[Word Document (DOCX)]] format uses plain ZIP (or not?), and carved files will (or not?) have .zip extension instead of DOCX. Is there any way to fix this (may be using the file list in zip)? [[User:.FUF|.FUF]] 20:25, 31 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] Addition: Office 2007 also has a binary file format which is also a ZIP-ed data [[Excel Spreadsheet (XLSB)]]
* Archive files
* [[ZIP archive]] file format
** 7z
** tar, gzip, bzip2
** RAR
* E-mail files
** [[Personal Folder File (PAB, PST, OST)]]
** MBOX (text based format, base64 content support)
* Audio/Video files
** MPEG
** MP2/MP3
** AVI
** ASF/WMV
** QuickTime
** MKV
* Printer spool files
** EMF (if I remember correctly)
* Internet history files
** index.dat
** firefox (sqllite 3)
* Other files
** thumbs.db which also is an [[OLE Compound File]] based format
** pagefile?

==Carving Strategies==
[[User:Joachim Metz|Joachim]] Note to moderator could this section be merged with the carving algorithm section?

* Simple fragment recovery carving using gap carving.
** [[User:Joachim Metz|Joachim]] have hook in for more advanced fragment recovery?
* Recovering of individual ZIP sections and JPEG icons that are not sector aligned.
** [[User:Joachim Metz|Joachim]] I would propose a generic fragment detection and recovery
* Autonomous operation (some mode of operation should be completely non-interactive, requiring no human intervention to complete [[User:RB|RB]])
** [[User:Joachim Metz|Joachim]] as much as possible, but allow to be overwritten by user
* [[User:Joachim Metz|Joachim]] When the tool output files the filenames should contain the offset in the input data (in hexadecimal?)
:: [[User:Mark Stam|Mark]] I really like the fact carved files are named after the physical or logical sector in which the file is found (photorec)
:::: [[User:Joachim Metz|Joachim]] This naming schema might cause duplicate name problem for extracting embedded files and extracting files from non sector aligned file systems.
* [[User:Joachim Metz|Joachim]] Should the tool allow to export embedded files?
* [[User:Joachim Metz|Joachim]] Should the tool allow to export fragments separately?
* [[User:Mark Stam|Mark]] I personally use photorec often for carving files in the whole volume (not only unallocated clusters), so I can store information about all potential interesting files in MySQL
:: [[User:Joachim Metz|Joachim]] interesting, Bas Kloet and me have been discussing to use information about allocated files in the recovery process, i.e. recovered fragments could be part of allocated files. Do we want to be able to extract them? Or could we rebuild the file from the fragments and the allocated files.
* [[User:Mark Stam|Mark]] It would also be nice if the files can be hashed immediately (MD5) so looking for them in other tools (for example Encase) is a snap

==Performance Requirements==
* Tested on 500GB-sized images. Should be able to carve a 500GB image in roughly 50% longer than it takes to read the image.
** Perhaps allocate a percentage budget per-validator (i.e. each validator adds N% to the carving time) [[User:RB|RB]]
** [[User:Joachim Metz|Joachim]] have multiple carving phases for precision/speed trade off?
* Parallelizable
** [[User:Joachim Metz|Joachim]] tunable for different architectures
* Configuration:
** Capability to parse some existing carvers' configuration files, either on-the-fly or as a one-way converter.
** Disengage internal configuration structure from configuration files, create parsers that present the expected structure
** [[User:Joachim Metz|Joachim]] The validator should deal with the file structure the carving algorithm should not know anything about the file structure (as in revit07 design)
** Either extend Scalpel/Foremost syntaxes for extended features or use a tertiary syntax ([[User:Joachim Metz|Joachim]] I would prefer a derivative of the revit07 configuration syntax which already has encountered some problems of dealing with defining file structure in a configuration file)

==Output==
* Can output audit.txt file.
* [[User:Joachim Metz|Joachim]] Can output database with offset analysis values i.e. for visualization tooling
* [[User:Joachim Metz|Joachim]] Can output debug log for debugging the algorithm/validation
* Easy integration into ascription software.
:: [[User:Joachim Metz|Joachim]] I'm no native speaker what do you mean with "ascription software"?
::: I think this was another non-native requesting easy scriptability. [[User:RB|RB]] 14:20, 31 October 2008 (UTC)
:::: [[User:Joachim Metz|Joachim]] that makes sense ;-)
::::: Incorrect. Ascription software is software that determines who the owner of a file is. [[User:Simsong|Simsong]] 06:36, 3 November 2008 (UTC)

= Ideas =
* Use as much TSK if possible. Don't carry your own FS implementation the way photorec does.
:: [[User:Joachim Metz|Joachim]] using TSK as much as possible would not allow to add your own file system support (i.e. mobile phones, memory structures, cap files) I would propose wrapping TSK and using it as much as possible but allow to integrate own FS implementations.
::::[[User:Capibara|Rob J Meijer]] I'm currently working on wrapping TSK filesystem as several loadable modules for OCFA. In OCFA a loadable (tree-graph) module implements the OCFA tree-graph API that basically states 'everything is a tree-graph node'. Possibly you could look at the OCFA treegraph API and module loading interface as an example, or we could work together on changing the API and module loading interface in such a way that it doesn't break OCFA and is usefull for the stand alone carver, but allowing both to use exactly the same tree-graph loadable modules.
* Extracting/carving data from [[Thumbs.db]]? I've used [[foremost]] for it with some success. [[Vinetto]] has some critical bugs :( [[User:.FUF|.FUF]] 19:18, 28 October 2008 (UTC)
==Recursive Carving==
[[User:Joachim Metz|Joachim]] do we want to support (let's call it) 'recursive in file carving' (for now) this is different from embedded files because there is a file system structure in the file and not just another file structure
* Is it just me, or do a lot of the above (and below) ideas somewhat skirt around the fact that many of us want recursive carving? Can we bend back to that instead of discussing object particulars? I think this can be distilled down to three requirements:
** Simple recursion: once an object is identified, have the ability to re-carve it for internal structures
** Directed recursion: the carver should be able to be directed at arbitrary blobs and told to carve it as a specified type. This allows programmatically more simple methods of dealing with unidentifiably compressed or encrypted data. Or filesystem fragments.
** Export: the ability to export an object (recognized or not) for later or external "recursion". Should go without saying for a carver, but...
:--[[User:RB|RB]] 18:45, 2 November 2008 (UTC)
:: [[User:Simsong|Simsong]] 06:30, 3 November 2008 (UTC) pyflag already does recursive carving. Are we just going to reimplement pyflag as a single executable?
::: [[User:Joachim Metz|Joachim]] that could be useful ;-)

==Library Dependencies==
[[User:Capibara|Rob J Meijer]] :
* Use libcarvpath whenever possible and by default to avoid high storage requirements.
:: [[User:Joachim Metz|Joachim]] For easy deployment I would not opt for making an integral part of the tool solely dependant on a single external library or the library must be integrated in the package
::[[User:Capibara|Rob J Meijer]] Integrating libraries (libtsk,libaff.libewf,libcarvpath etc) is bad practice, autotools are your friend IMO.
:: [[User:Joachim Metz|Joachim]] I'm not talking about integrating (shared) libraries. I'm talking about that an integral part of a tool should be part of it's package. Why can't the tool package contain shared or static libraries for local use? A far worse thing to do is to have a large set of dependencies and making the tool difficult to install for most users. The tool package should contain the most necessary code. afflib/libewf support could be detected by the autotools a neat separation of functionality.
::: From a packager's standpoint, [[User:Joachim Metz|Joachim]]'s other libraries do a really good job of this, carrying around what they need but using a system-global version if available. [[User:RB|RB]]
* libtsk
* libaff ? : possibly the discussion in the requirements section should move to his section.
* libewf ? : possibly the discussion in the requirements section should move to his section.
* posix ? : Can we depend especially on the availability of UNIX domain sockets and the possibility to use msg_accrights for passing opn file handles as ocaps?

==Filesystem Detection==
* Dont stop with filesystem detection after the first match. Often if a partition is reused with a new FS and is not all that full yet, much of the old FS can still be valid. I have seen this with ext2/fat. The fact that you have identified a valid FS on a partition doesn't mean there isn't an(almost) valid second FS that would yield additional files. Identifying doubly allocated space might in some cases also be relevant.
:: [[User:Joachim Metz|Joachim]] What your saying is that dealing with file system fragments should be part of the carving algorithm
* Allow use where filesystem based carving is done by other tool, and the tool is used as second stage on (sets of) unallocated block (pseudo) files and/or non FS partition (pseudo) files.
:: [[User:Joachim Metz|Joachim]] I would not opt for this. The tool would be dependent on other tools and their data format, which makes the tool difficult to maintain. I would opt to integrate the functionality of having multiple recovery phases (stages) and allow the tooling to run the phases after one and other or separately.
::[[User:Capibara|Rob J Meijer]] More generically, I feel a way should exist to communicate the 'left overs' a previous (non open, for example LE-only) tool left.
:: [[User:Joachim Metz|Joachim]] I guess if the tool is designed to handle multiple phases it should store its data somewhere. So it should be possible to convert results of such non open tooling to the format required. However I would opt to design the recovery functionality of these non-open tools into open tools. And not to limit ourselves making translators due to the design of these non-open tools.
* Ability to be used as a library instead of a tool. Ability to access metadata true library, and thus the ability to set metadata from the carving modules. This would be extremely usefull for integrating the project into a [[computer forensics framework]] .
:: [[User:Joachim Metz|Joachim]] I guess most of the code could be integrated into libraries, but I would not opt integrating tool functionality into a library
* [[User:Mark Stam|Mark]] I think it would be very handy to have a CSV, TSV, XML or other delimited output (log)file with information about carved files. This output file can then be stored in a database or Excel sheet (report function)

==Anti forensics and system integrity concerns==
* It might be very interesting to look at the possibilities of using a multi process style of module support and combine it with a least authority design. On platforms that support AppArmor (or similar) and uid based firewall rules, this could make for the first true POLA (principle of least authority) based forensic tool ever. POLA based forensics tools should make for a strong integrity guard against many anti forensics. Alternatively we could look at integrating a capability secure language (E?) for implementation of at least validation modules. I don't expect this idea to make it, but mentioning it I hope might spark off less strong alternatives that at least partially address the integrity + anti-forensics problem. If we can in some way introduce POLA to a wider forensics public, other tools might also pick up on it what would be great.
:: [[User:Joachim Metz|Joachim]] Could you give an example of how you see this in action?
::::[[User:Capibara|Rob J Meijer]] I see two layers where using POLA could be applied. The best one would require one of the folowing as prerequisites:
::::* The libaff/libewf layer is moved to a fuse implementation (for example carvfs).
::::* Libewf/Libaff are updated to accept opened filhandles instead of demanding to open their own files.
::::If one of these is fulfilled, than the tool running as some user can just have the simple task of opening the image files, starting up the 'real' tool and handing over the appropriate file handles. If the real tool runs with a restrictive AppArmor profile, and is started suid to a tool specific user that also has its own iptables uid based filter, than the real tool will run with least authority.
:::: A second alternative, if neither of the first prerequisite could not be bet, would be to run the modules as confined processes and have a non confined process run as proxy for the first.
:::: A third probably far fetched alternative would be to embed an object capability language in the tool and make the module interface thus that modules are to be written in this ocap language.
::::A 4th alternative might include minorfs or plash, but I havn't geven those sufficient thinking hours yet.

== Format syntax specification ==
* Carving data structures. For example, extract all TCP headers from image by defining TCP header structure and some fields (e.g. source port > 1024, dest port = 80). This will extract all data matching the pattern and write a file with other fields. Another example is carving INFO2 structures and URL activity records from index.dat [[User:.FUF|.FUF]] 20:51, 28 October 2008 (UTC)
** This has the opportunity to be extended to the concept of "point at blob FOO and interpret it as BAR"
.FUF added:
The main idea is to allow users to define structures, for example (in pascal-like form):

<pre>
Field1: Byte = 123;
SomeTextLength: DWORD;
SomeText: string[SomeTextLength];
Field4: Char = 'r';
...
</pre>

This will produce something like this:
<pre>
Field1 = 123
SomeTextLength = 5
SomeText = 'abcd1'
Field4 = 'r'
</pre>

(In text or raw forms.)

Opinions?

Opinion: Simple pattern identification like that may not suffice, I think Simson's original intent was not only to identify but to allow for validation routines (plugins, as the original wording was). As such, the format syntax would need to implement a large chunk of some programming language in order to be sufficiently flexible. [[User:RB|RB]]

[[User:Joachim Metz|Joachim]]
In my option your example is too limited. Making the revit configuration I learned you'll need a near programming language to specify some file formats.
A simple descriptive language is too limiting. I would also go for 2 bytes with endianess instead of using terminology like WORD and small integer, it's much more clear. The configuration also needs to deal with aspects like cardinality, required and optional structures.
:: This is simply data structures carving, see ideas above. Somebody (I cannot track so many changes per day) separated the original text. There is no need to count and join different structures. [[User:.FUF|.FUF]] 19:53, 31 October 2008 (UTC)
:::: [[User:Joachim Metz|Joachim]] This was probably me is the text back in it's original form?
:::: I started it by moving your Revit07 comment to the validator/plugin section in [http://www.forensicswiki.org/index.php?title=Carver_2.0_Planning_Page&diff=prev&oldid=7583 this edit], since I was still at that point thinking operational configuration for that section, not parser configurations. [[User:RB|RB]]
:::: [[User:Joachim Metz|Joachim]] I renamed the title to format syntax, clarity is important ;-)

Please take a look at the revit07 format syntax specification (configuration). It's not there yet but goes a far way. Some things currently missing:
* bitwise alignment
* handling encapsulated streams (MPEG/capture files)
* handling content based formats (MBOX)

=Caving algorithm =
[[User:Joachim Metz|Joachim]]
* should we allow for multiple carving phases (runs/stages)?
:: I opt yes (separation of concern)
* should we allow for multiple carving algorithms?
:: I opt yes, this allows testing of different approaches
* Should the algorithm try to do as much in 1 run over the input data? To reduce IO?
:: I opt that the tool should allow for multiple and single run over the input data to minimize the IO or the CPU as bottleneck
* Interaction between algorithm and validators
** does the algorithm passes data blocks to the validators?
** does a validator need to maintain a state?
** does a validator need to revert a state?
** How do we deal with embedded files and content validation? Do the validators call another validator?
* do we use the assumption that a data block can be used by a single file (with the exception of embedded/encapsulated files)?
* Revit07 allows for multiple concurrent result files states to deal with fragmentation. One has the attribute of being active (the preferred) and the other passive. Do we want/need something similar? The algorithm adds block of input data (offsets) to these result files states.
** if so what info would these result files states require (type, list of input data blocks)
* how do we deal with file system remainders?
** Can we abstract them and compare them against available file system information?
* Do we carve file systems in files?
:: I opt that at least the validator uses this information

==Caving scenarios ==
[[User:Joachim Metz|Joachim]]
* normal file (file structure, loose text based structure (more a content structure?))
* fragmented file (the file entirely exist)
* a file fragment (the file does not entirely exist)
* intertwined file
* encapsulated file (MPEG/network capture)
* embedded file (JPEG thumbnail)
* obfuscation ('encrypted' PFF) this also entails encryption and/or compression
* file system in file

=File System Awareness =
==Background: Why be File System Aware?==
Advantages of being FS aware:
* You can pick up sector allocation sizes
:: [[User:Joachim Metz|Joachim]] do you mean file system block sizes?
* Some file systems may store things off sector boundaries. (ReiserFS with tail packing)
* Increasingly file systems have compression (NTFS compression)
:: [[User:Joachim Metz|Joachim]] Carving NTFS-compressed (lznt1) files (http://sourceforge.net/projects/revit/files/Documentation/Carving%20NTFS-compressed%20data/Carving%20for%20NTFS%20compressed%20files.pdf/download)
* Carve just the sectors that are not in allocated files.
:: [[User:Joachim Metz|Joachim]] sparse (file system) blocks e.g. NTFS cluster blocks

==Tasks that would be required==

==Discussion==
:: As noted above, TSK should be utilized as much as possible, particularly the filesystem-aware portion. If we want to identify filesystems outside of its supported set, it would be more worth our time to work on implementing them there than in the carver itself. [[User:RB|RB]]

:::: I guess this tool operates like [[Selective file dumper]] and can recover files in both ways (or not?). Recovering files by using carving can recover files in situations where sleuthkit does nothing (e.g. file on NTFS was deleted using ntfs-3g, or filesystem was destroyed or just unknown). And we should build the list of filesystems supported by carver, not by TSK. [[User:.FUF|.FUF]] 07:08, 29 October 2008 (UTC)

:: This tool is still in the early planning stages (requirements discovery), hence few operational details (like precise modes of operation) have been fleshed out - those will and should come later. The justification for strictly using TSK for the filesystem-sensitive approach is simple: TSK has good filesystem APIs, and it would be foolish to create yet another standalone, incompatible implementation of filesystem(foo) when time would be better spent improving those in TSK, aiding other methods of analysis as well. This is the same reason individuals that have implemented several other carvers are participating: de-duplication of effort. [[User:RB|RB]]

:: [[User:Joachim Metz|Joachim]] A design problem might be that TSK currently is a single library operating on multiple layers (storage media IO, volume/partition analysis and file system analysis). I'm not aware how easily the parts can be used separately. But I estimate that for the carver we want to use these 3 layers differently than TSK currently does.

[[User:Joachim Metz|Joachim]] I would like to have the carver (recovery tool) also do recovery using file allocation data or remainders of file allocation data.

[[User:Joachim Metz|Joachim]]
I would go as far to ask you all to look beyond the carver as a tool and look from the perspective of the carver as part of the forensic investigation process. In my eyes certain information needed/acquired by the carver could be also very useful investigative information i.e. what part of a hard disk contains empty sectors.

=Supportive tooling=
[[User:Joachim Metz|Joachim]]
* validator (definitions) tester (detest in revit07)
* tool to make configuration based definitions
* post carving validation
* the carver needs to provide support for fuse mount of carved files (carvfs)

=Testing =
[[User:Joachim Metz|Joachim]]
* automated testing
* test data

=Validator Construction=
Options:
* Write validators in C/C++
** [[User:Joachim Metz|Joachim]] you mean dedicated validators
* Have a scripting language for writing them (python? Perl?) our own?
** [[User:Joachim Metz|Joachim]] use easy to embed programming languages i.e. Phyton or Lua
* Use existing programs (libjpeg?) as plug-in validators?
** [[User:Joachim Metz|Joachim]] define a file structure api for this

=Existing Code that we have=
[[User:Joachim Metz|Joachim]]
Please add any missing links

Documentation/Articles
* DFRWS2006/2007 carving challenge results
* DFRWS2008 paper on carving

Carvers
* DFRWS2006/2007 carving challenge results
* photorec (http://www.cgsecurity.org/wiki/PhotoRec)
* revit06 and revit07 (http://sourceforge.net/projects/revit/)
* s3/scarve

Possible file structure validator libraries
* divers existing file support libraries
* libole2 (inhouse experimental code of OLE2 support)
* [[libnk2]]
* [[libpff]]

Input support
* AFF (http://www.afflib.org/)
* [[libewf]]
* TSK device & raw & split raw (http://www.sleuthkit.org/)

Volume/Partition support
* disktype (http://disktype.sourceforge.net/)
* testdisk (http://www.cgsecurity.org/wiki/TestDisk)
* TSK

File system support
* TSK
* photorec FS code
* implementations of FS in Linux/BSD

Content support

Zero storage support
* libcarvpath ( http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=210704 )
* carvfs ( http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=210954 )
* tsk-cp ( http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=267227 )
* carvfsmodewf (http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=268256 )
POLA
* joe-e (java) ( http://code.google.com/p/joe-e/ )
* Emily (ocaml) ( http://erights.org/download/emily/ )
* the E language ( http://www.erights.org/ )
* AppArmor
* iptables/ipfw
* minorfs ( http://polacanthus.net/minorfs.html )
* plash ( http://plash.beasts.org/wiki/ )

=Implementation Timeline=
# gather the available resources/ideas/wishes/needs etc. (I guess we're in this phase)
# start discussing a high level design (in terms of algorithm, facilities, information needed)
## input formats facility
## partition/volume facility
## file system facility
## file format facility
## content facility
## how to deal with fragment detection (do the validators allow for fragment detection?)
## how to deal with recombination of fragments
## do we want multiple carving phases in light of speed/precision tradeoffs
# start detailing parts of the design
## Discuss options for a grammar driven validator?
## Hard-coded plug-ins?
## Which existing code can we use?
# start building/assembling parts of the tooling for a prototype
## Implement simple file carving with validation.
## Implement gap carving
# Initial Release
# Implement the ''threaded carving'' that [[User:.FUF|.FUF]] is describing above.

[[User:Joachim Metz|Joachim]] Shouldn't multi threaded carving (MTC) not be part of the 1st version?
The MT approach makes for different design decisions
: It is virtually impossible to turn a non-MT application into an MT application .[[User:Simsong|Simsong]] 06:37, 3 November 2008 (UTC)

The Sleuth Kit

2009-08-28T10:20:13Z

Capibara: /* See Also */

{{Infobox_Software |
name = The Sleuth Kit |
maintainer = [[Brian Carrier]] |
os = {{Linux}}, {{FreeBSD}}, {{OpenBSD}}, {{Mac OS X}}, {{SunOS}} |
genre = {{Analysis}} |
license = {{IBM Open Source License}}, {{Common Public License}}, {{GPL}} |
website = [http://www.sleuthkit.org/ sleuthkit.org] |
}}

'''The Sleuth Kit''' ('''TSK''') is a collection of [[UNIX]]-based command line tools that allow you to investigate a computer. The current focus of the tools is the file and volume systems and TSK supports [[FAT]] (12/16/32), [[Ext2]]/[[Ext3|3]], [[NTFS]], [[Ufs|UFS]] (1 & 2), and ISO 9660 [[file system]]s.

[[Autopsy]] is a frontend for TSK which allows browser-based access to the TSK tools.

=Features=

The Sleuth Kit is arranged in layers. There is a ''data layer'' which is concerned with how information is stored on a disk and a ''metadata layer'' which is considered with information such as [[inode]]s and [[directory|directories]]. The commands that deal with the data layer are prefixed with the letter ''d'', which the commands that deal with the metadata layer are prefixed with the letter ''i''.

Some of the commands in Sleuth Kit are:

; dcat
: Views the contents of a [[block]].

; dls
: Lists [[unallocated block]]s. Makes keyword searches more efficient. Gets a list of unallocated blocks.

; dcalc
: Tells you where an unallocated blocks are.

; dstat
: Details about a given block.

; icat
: View contents of a file given its inode value or [[cluster number]]. Doesn't list directories, lists the contents.

; ils
: Lists the files extents on a disk.

; istat
: Information about an inode number.

==File Systems Understood==

* [[NTFS]]
* [[FAT]]
* [[Ext2]], [[Ext3]]
* [[Ufs|UFS]] (1 & 2)
* ISO 9660

==File Search Facilities==

* Lists allocated and unallocated files.
* Lists and sorts by file type.
* Shows a time of creation and change.

==Historical Reconstruction==

==Searching Abilities==

* Searches for keywords.
* Builds an index.

==Hash Databases==

* Uses [[MD5]] or [[SHA-1]].
* Interfaces with NIST [[NSRL]], [[Hashkeeper]] and customer databases.

==Evidence Collection Features==

* Tracks forensic activity.

=History=

==License Notes==

"The file system tools (in the src/fstools directory) are released
under the IBM open source license and Common Public License, both
are located in the license directory. The modifications to 'mactime'
from the original 'mactime' in TCT and 'mac-daddy' are released
under the Common Public License. Other tools in the src directory
are either Common Public License or the GNU Public License."

= See Also =
* [[The Sleuth Kit How-To]]
* [[tsk-cp]]
* The mmls [[OCFA treegraph API]] example module.

= External Links =

* [http://www.sleuthkit.org/autopsy/desc.php Autopsy website]

==External Reviews==

Zero storage carving

2009-08-28T10:17:55Z

Capibara:

Zero storage carving is the concept of using techniques to enable doing [[carving]] of meaningfull and processable chunks or files of uncompressed unencoded data on disks,
disk-images or container files without the need for additional storage to be allocated for copies of the relevant data chunks or files.
Zero storage carving is sometimes also referred to as in-line carving.

Tools with support or facilities for zero storage carving include:

* [[tsk-cp]]
* [[scalpel]]
* [[PhotoRec]]
* [[CarvFs]]
* [[LibCarvPath]]
* [[OCFA treegraph API]]

Zero storage carving

2009-08-28T10:17:29Z

Capibara:

Zero storage carving

2009-08-28T10:16:16Z

Capibara:

Zero storage carving

2009-08-28T10:15:42Z

Capibara:

Zero storage carving is the concept of using techniques to enable doing [[carving]] of meaningfull and processable chunks or files of uncompressed unencoded data on disks,
disk-images or container files without the need for additional storage to be allocated for copies of the relevant data chunks or files.
Zero storage carving is sometimes also referred to as in-line carving.

Tools with support or facilities for zero storage carving include:

* [[tsk-cp]]
* [[scalpel]]
* [[Photorec]]
* [[CarvFs]]
* [[LibCarvPath]]
* [[Ocfa Treegraph API]]

Zero storage carving

2009-08-28T10:13:02Z

Capibara: Created page with 'Zero storage carving is the concept of using techniques to enable doing carving of meaningfull and processable chunks or files of uncompressed unencoded data on disks, disk-…'

Open Computer Forensics Architecture

2009-08-28T10:03:30Z

Capibara:

The '''Open Computer Forensics Architecture''' ('''OCFA''') is a modular [[computer forensics framework]] built by the [[Dutch National Police Agency]]. The main goal is to automate the digital forensic process to speed up the investigation and give tactical [[investigator]]s direct access to the seized data through an easy to use search and browse interface.

The architecture forms an environment where existing forensic [[tools]] and libraries can be easily plugged into the architecture and can thus be made part of the recursive extraction of data and [[metadata]] from digital evidence.

The Open Computer Forensics Architecture aims to be highly modular, robust, fault tolerant, recursive and scalable in order to be usable in large investigations that spawn numerous terabytes of evidence data and covers hundreds of evidence items.

Modules in OCFA for reasons of fault tolerance are processes. The basic [[OcfaLib API]] makes it possible and relatively easy to build an OCFA module out of any data processing library or tool. OCFA comes with numerous such modules that are mostly wrappers around libraries like [[libmagic]] or tools such as those found in the [[Sleuthkit]].

The 2.2 version of OCFA (released April 2009) makes the previously internal [[OCFA treegraph API]] available for OCFA module development. The OCFA treegraph API allows more advanced dissectors that produce data and meta-data for a treegraph representation of an input file. The OCFA treegraph API also allows dissectors that are programed to be [[CarvFs]] aware to use [[zero storage carving]].

Communication between modules within OCFA is governed by a two layered communication infrastructure as provided by OCFA. At the lowest layer is a messaging system with at is center the OCFA Anycast Relay. The Anycast Relay provides the facilities of module crash resistance, distributed processing load balancing and flow control.
At a higher level of communication, the OCFA XML Router provides for the routing of individual pieces of evidence through the most appropriate tool chain for its particular type of content.

Although OCFA contains a rudimentary user interface, most of its power is in the backend architecture.
The last and final module in the tool chain of any evidence will be the OCFA Data Store Module. This module
processes the evidence XML (that contains all of the evidence data its meta data) and stores relevant parts into a postgesql database. Extending the apache based user interface with interfaces for your own case bound queries
is something that should proof very useful in most investigations.

For more information consult [http://sourceforge.net/projects/ocfa/ sourceforge.net/projects/ocfa/ ] .

OCFA treegraph API

2009-08-28T10:01:54Z

Capibara:

The OCFA treegraph API is a more advanced API for the [[Open Computer Forensics Architecture]]. The basic [[OcfaLib API]] allows for the fast and simple creation of simple dissector and extractor modules for OCFA, but has some limitations.
To overcome these limitations, the 2.2 version of OCFA re-vectored and promoted an API that was previously used internally by th OCFA library to an API available to module builders.
The OCFA treegraph API defines an interface that a loadable library must implement in order to be usable as an advanced dissector module bu the Open Computer Forensics Architecture.
Basically it defines an interface 'TreeGraphNode' that a treegraph module will need to derive one or many classes from. A TreeGraphNode can contain data, meta-data and sub node's that are also TreeGraphNode implementations.
The data interface of the TreeGraphNode also allows treegraph modules that are [[CarvFs]] aware, to return a carvpath as so called soft linkable path'. Doing so allows OCFA to use substantially less storage resources.

An example of a treegraph module for OCFA is included in the 2.2 release of OCFA. This example is the OCFA mmls module.
The ocfa mmls module reproduces the functionality of the [[sleuthkit]] mmls tool. It does this using the OCFA treegraph library,
the [[LibCarvPath]] library, and the [[sleuthkit]] library.

OCFA treegraph API

2009-08-28T10:00:59Z

Capibara: Created page with 'The OCFA treegraph API is a more advanced API for the Open Computer Forensics Architecture. The basic OCFA API allows for the fast and simple creation of simple dissector…'

The OCFA treegraph API is a more advanced API for the [[Open Computer Forensics Architecture]]. The basic [[OCFA API]] allows for the fast and simple creation of simple dissector and extractor modules for OCFA, but has some limitations.
To overcome these limitations, the 2.2 version of OCFA re-vectored and promoted an API that was previously used internally by th OCFA library to an API available to module builders.
The OCFA treegraph API defines an interface that a loadable library must implement in order to be usable as an advanced dissector module bu the Open Computer Forensics Architecture.
Basically it defines an interface 'TreeGraphNode' that a treegraph module will need to derive one or many classes from. A TreeGraphNode can contain data, meta-data and sub node's that are also TreeGraphNode implementations.
The data interface of the TreeGraphNode also allows treegraph modules that are [[CarvFs]] aware, to return a carvpath as so called soft linkable path'. Doing so allows OCFA to use substantially less storage resources.

An example of a treegraph module for OCFA is included in the 2.2 release of OCFA. This example is the OCFA mmls module.
The ocfa mmls module reproduces the functionality of the [[sleuthkit]] mmls tool. It does this using the OCFA treegraph library,
the [[LibCarvPath]] library, and the [[sleuthkit]] library.

LibCarvPath

2009-08-28T09:31:30Z

Capibara:

LibCarvPath is a library designed to be used by carving and file system analysis tools.
LibCarvPath allows fragments represented by offset and size to be combined in a CarvPath
annotation that take the form of file system paths. LibCarvPath addresses the limits of
file system paths by mapping extremely fragmented files to a uniquely identifying key in
a long-path database.

The following tools use LibCarvPath and/or CarvPath Annotations:

* [[CarvFs]]
* [[tsk-cp]]
* [[scalpelcp]]
* Modules for the [[Open Computer Forensics Architecture]] that use the [[OCFA Treegraph API]].

Next to these, in [[Photorec]] work has started to include LibCarvPath support.

[http://sourceforge.net/projects/ocfa/ http://sourceforge.net/projects/ocfa/]

Open Computer Forensics Architecture

2009-08-28T09:28:12Z

Capibara:

The '''Open Computer Forensics Architecture''' ('''OCFA''') is a modular [[computer forensics framework]] built by the [[Dutch National Police Agency]]. The main goal is to automate the digital forensic process to speed up the investigation and give tactical [[investigator]]s direct access to the seized data through an easy to use search and browse interface.

The architecture forms an environment where existing forensic [[tools]] and libraries can be easily plugged into the architecture and can thus be made part of the recursive extraction of data and [[metadata]] from digital evidence.

The Open Computer Forensics Architecture aims to be highly modular, robust, fault tolerant, recursive and scalable in order to be usable in large investigations that spawn numerous terabytes of evidence data and covers hundreds of evidence items.

Modules in OCFA for reasons of fault tolerance are processes. The basic [[OcfaLib API]] makes it possible and relatively easy to build an OCFA module out of any data processing library or tool. OCFA comes with numerous such modules that are mostly wrappers around libraries like [[libmagic]] or tools such as those found in the [[Sleuthkit]].

The 2.2 version of OCFA (released April 2009) makes the previously internal [[OCFA treegraph API]] available for OCFA module development. The OCFA treegraph API allows more advanced dissectors that produce data and meta-data for a treegraph representation of an input file. The OCFA treegraph API also allows dissectors that are programed to be [[CarvFs]] aware to use [[Zero Storage Carving]].

Communication between modules within OCFA is governed by a two layered communication infrastructure as provided by OCFA. At the lowest layer is a messaging system with at is center the OCFA Anycast Relay. The Anycast Relay provides the facilities of module crash resistance, distributed processing load balancing and flow control.
At a higher level of communication, the OCFA XML Router provides for the routing of individual pieces of evidence through the most appropriate tool chain for its particular type of content.

Although OCFA contains a rudimentary user interface, most of its power is in the backend architecture.
The last and final module in the tool chain of any evidence will be the OCFA Data Store Module. This module
processes the evidence XML (that contains all of the evidence data its meta data) and stores relevant parts into a postgesql database. Extending the apache based user interface with interfaces for your own case bound queries
is something that should proof very useful in most investigations.

For more information consult [http://sourceforge.net/projects/ocfa/ sourceforge.net/projects/ocfa/ ] .

Computer forensics framework

2009-01-15T06:17:41Z

Capibara:

The following more or less generic computer forensics frameworks exist:

* [[Open Computer Forensics Architecture]]
* [[Pyflag]]
* [[DELV]]
* [[XIRAF]]

The Sleuth Kit

2008-12-13T10:47:06Z

Capibara: /* See Also */

{{Infobox_Software |
name = The Sleuth Kit |
maintainer = [[Brian Carrier]] |
os = {{Linux}}, {{FreeBSD}}, {{OpenBSD}}, {{Mac OS X}}, {{SunOS}} |
genre = {{Analysis}} |
license = {{IBM Open Source License}}, {{Common Public License}}, {{GPL}} |
website = [http://www.sleuthkit.org/ sleuthkit.org] |
}}

'''The Sleuth Kit''' ('''TSK''') is a collection of [[UNIX]]-based command line tools that allow you to investigate a computer. The current focus of the tools is the file and volume systems and TSK supports [[FAT]] (12/16/32), [[Ext2]]/[[Ext3|3]], [[NTFS]], [[Ufs|UFS]] (1 & 2), and ISO 9660 [[file system]]s.

[[Autopsy]] is a frontend for TSK which allows browser-based access to the TSK tools.

=Features=

The Sleuth Kit is arranged in layers. There is a ''data layer'' which is concerned with how information is stored on a disk and a ''metadata layer'' which is considered with information such as [[inode]]s and [[directory|directories]]. The commands that deal with the data layer are prefixed with the letter ''d'', which the commands that deal with the metadata layer are prefixed with the letter ''i''.

Some of the commands in Sleuth Kit are:

; dcat
: Views the contents of a [[block]].

; dls
: Lists [[unallocated block]]s. Makes keyword searches more efficient. Gets a list of unallocated blocks.

; dcalc
: Tells you where an unallocated blocks are.

; dstat
: Details about a given block.

; icat
: View contents of a file given its inode value or [[cluster number]]. Doesn't list directories, lists the contents.

; ils
: Lists the files extents on a disk.

; istat
: Information about an inode number.

==File Systems Understood==

* [[NTFS]]
* [[FAT]]
* [[Ext2]], [[Ext3]]
* [[Ufs|UFS]] (1 & 2)
* ISO 9660

==File Search Facilities==

* Lists allocated and unallocated files.
* Lists and sorts by file type.
* Shows a time of creation and change.

==Historical Reconstruction==

==Searching Abilities==

* Searches for keywords.
* Builds an index.

==Hash Databases==

* Uses [[MD5]] or [[SHA-1]].
* Interfaces with NIST [[NSRL]], [[Hashkeeper]] and customer databases.

==Evidence Collection Features==

* Tracks forensic activity.

=History=

==License Notes==

"The file system tools (in the src/fstools directory) are released
under the IBM open source license and Common Public License, both
are located in the license directory. The modifications to 'mactime'
from the original 'mactime' in TCT and 'mac-daddy' are released
under the Common Public License. Other tools in the src directory
are either Common Public License or the GNU Public License."

= See Also =
* [[The Sleuth Kit How-To]]
* [[tsk-cp]]

= External Links =

* [http://www.sleuthkit.org/autopsy/desc.php Autopsy website]

==External Reviews==

Tsk-cp

2008-12-12T19:01:21Z

Capibara:

Tsk-cp is a set of [[LibCarvPath]] aware versions of [[Sleuthkit]] tools, that are for use together with the
normal versions of the other sleuthkit tools in the process of doing [[zero storage carving]].

The tools are:

* mmls-cp : A CarvPath based version of mmls for listing a partitioned carvpath disk images as a list of partition carvpaths.
* dls-cp : A CarvPath based version of dls for listing all continuous unallocated fragments of a carvpath partition holding a filesystem as a list of unallocated block carvpaths.
* icat-cp : A CarvPath based version of icat that instead of copying out the data of an inode within a carvpath partition holding a filesystem as the carvpath of the file and the carvpath of the [[file slack]].

The carvpaths output by dls-cp can be used as the input of a CarvPath aware carving tool.

[http://sourceforge.net/projects/ocfa/ http://sourceforge.net/projects/ocfa/]

LibCarvPath

2008-12-12T19:00:49Z

Capibara:

LibCarvPath is a library designed to be used by carving and file system analysis tools.
LibCarvPath allows fragments represented by offset and size to be combined in a CarvPath
annotation that take the form of file system paths. LibCarvPath addresses the limits of
file system paths by mapping extremely fragmented files to a uniquely identifying key in
a long-path database.

The following tools use LibCarvPath and/or CarvPath Annotations:

* [[CarvFs]]
* [[tsk-cp]]
* [[scalpelcp]]

Next to these, in [[Photorec]] work has started to include LibCarvPath support.

[http://sourceforge.net/projects/ocfa/ http://sourceforge.net/projects/ocfa/]

CarvFs

2008-12-12T18:59:01Z

Capibara:

Scalpelcp

2008-12-12T18:52:29Z

Capibara: New page: ScalpelCp is a simple perl script that processes the logfile of scalpel run in preview mode on a CarvPath file and populates the output directory with symbolic links to CarvFs Carv...

ScalpelCp is a simple perl script that processes the logfile of [[scalpel]] run in preview mode on a CarvPath file and populates
the output directory with symbolic links to [[CarvFs]] CarvPaths.

Tsk-cp

2008-12-12T18:49:23Z

Capibara: New page: Tsk-cp is a set of LibCarvPath aware versions of Sleuthkit tools, that are for use together with the normal versions of the other sleuthkit tools in the process of doing [[zero sto...

LibCarvPath

2008-12-12T18:37:01Z

Capibara: New page: LibCarvPath is a library designed to be used by carving and file system analysis tools. LibCarvPath allows fragments represented by offset and size to be combined in a CarvPath annotation ...

LibCarvPath is a library designed to be used by carving and file system analysis tools.
LibCarvPath allows fragments represented by offset and size to be combined in a CarvPath
annotation that take the form of file system paths. LibCarvPath addresses the limits of
file system paths by mapping extremely fragmented files to a uniquely identifying key in
a long-path database.

The following tools use LibCarvPath and/or CarvPath Annotations:

* [[CarvFs]]
* [[tsk-cp]]
* [[scalpelcp]]

Next to these, in [[Photorec]] work has started to include LibCarvPath support.

CarvFs

2008-12-12T18:25:02Z

Capibara: New page: CarvFs is a modular Fuse based user space file system on top op LibCarvPath. CarvFS makes CarvPath style annotations as used by LibCarvPath available as files. Using CarvFs makes ...

Tools:Data Recovery

2008-12-12T18:14:32Z

Capibara: /* Carving */

= Partition Recovery =

*[http://www.ptdd.com/index.htm Partition Table Doctor]
: Recover deleted or lost partitions (FAT16/FAT32/NTFS/NTFS5/EXT2/EXT3/SWAP).

*[http://www.diskinternals.com/ntfs-recovery/ NTFS Recovery]
: DiskInternals NTFS Recovery is a fully automatic utility that recovers data from damaged or formatted disks.

*[http://www.stud.uni-hannover.de/user/76201/gpart/ gpart]
: Gpart is a tool which tries to guess the primary partition table of a PC-type hard disk in case the primary partition table in sector 0 is damaged, incorrect or deleted.

*[http://www.cgsecurity.org/wiki/TestDisk Testdisk]
: TestDisk is OpenSource software and is licensed under the GNU Public License (GPL).

== See Also ==

* [http://support.microsoft.com/?kbid=166997 Using Norton Disk Edit to Backup Your Master Boot Record]

== Notes ==

* "fdisk /mbr" restores the boot code in the [[Master Boot Record]], but not the partition itself. On newer versions of Windows you should use fixmbr, bootrec, mbrfix, or mbrwizard. You can also extract a copy of the specific standard MBR code from tools like bootrec.exe and diskpart.exe in Windows (from various offsets) and copy it to disk with dd (Use bs=446 count=1). For Windows XP SP2 c:\%WINDIR%\System32\diskpart.exe the MBR code is found between offset 1b818h and 1ba17h.

= Data Recovery =
The term "Data Recovery" is frequently used to mean forensic recovery, but the term really should be used for recovering data from damaged media.

*[http://www.toolsthatwork.com/bringback.htm BringBack]
: BringBack offers easy to use, inexpensive, and highly successful data recovery for Windows and Linux (ext2) operating systems and digital images stored on memory cards, etc.

*[http://www.runtime.org/raid.htm RAID Reconstructor]
: Runtime Software's RAID Reconstructor will reconstruct RAID Level 0 (Striping) and RAID Level 5 drives.

*[http://www.salvationdata.com Salvation Data]
: Claims to have a program that can read the "bad blocks" of Maxtor drives with proprietary commands.

* [http://www.e-rol.com/en/ e-ROL]
: Erol allows you to recover through the internet files erased by mistake. Recover your files online for free.

* [http://www.recuva.com/ Recuva]
: Recuva is a freeware Windows tool that will recover accidentally deleted files.

* [http://www.snapfiles.com/get/restoration.html Restoration]
: Restoration is a freeware Windows software that will allow you to recover deleted files

* [http://www.undelete-plus.com/ Undelete Plus]
: Undelete Plus is a free deleted file recovery tool that works for all versions of Windows (95-Vista), FAT12/16/32, NTFS and NTFS5 filesystems and can perform recovery on various solid state devices.

* [http://www.data-recovery-software.net/ R-Studio]
: R-Studio is a data recovery software suite that can recover files from FAT(12-32), NTFS, NTFS 5, HFS/HFS+, FFS, UFS/UFS2 (*BSD, Solaris), Ext2/Ext3 (Linux) and so on.

See also [[Data Recovery Stories]]

=Carving=
*[http://www.datalifter.com/products.htm DataLifter® - File Extractor Pro]
: Data carving runs on multiple threads to make use of modern processors

*[http://foremost.sourceforge.net/ Foremost]
: Foremost is a console program to recover files based on their headers, footers, and internal data structures.

*[http://www.digitalforensicssolutions.com/Scalpel/ Scalpel]
: Scalpel is a fast file carver that reads a database of header and footer definitions and extracts matching files from a set of image files or raw device files. Scalpel is filesystem-independent and will carve files from FATx, NTFS, ext2/3, or raw partitions.

*[[EnCase]]
: EnCase comes with some eScripts that will do carving.

*[[CarvFs]]
: A virtual file system (fuse) implementation that can provide carving tools with the possibility to do recursive multi tool zero-storage carving (also called in-place carving). Patches and scripts for scalpel and foremost are provided. Works on raw and encase images.

*[[LibCarvPath]]
: A shared library that allows carving tools to use zero-storage carving on carvfs virtual files.

*[http://www.cgsecurity.org/wiki/PhotoRec PhotoRec]
: PhotoRec is file data recovery software designed to recover lost files including video, documents and archives from Hard Disks and CDRom and lost pictures (thus, its 'Photo Recovery' name) from digital camera memory.

*[http://www.datarescue.com/photorescue/ PhotoRescue]
: Datarescue PhotoRescue Advanced is picture and photo data recovery solution made by the creators of IDA Pro. PhotoRescue will undelete, unerase and recover pictures and files lost on corrupted, erased or damaged compact flash (CF) cards, SD Cards, Memory Sticks, SmartMedia and XD cards.

* [https://www.uitwisselplatform.nl/projects/revit RevIt]
: RevIt (Revive It) is an experimental carving tool, initially developed for the DFRWS 2006 carving challenge. It uses 'file structure based carving'. Note that RevIt currently is a work in progress.

* [http://jbj.rapanden.dk/magicrescue/ Magic Rescue]
: Magic Rescue is a file carving tool that uses "magic bytes" in a file contents to recover data.

Carver 2.0 Planning Page

2008-12-03T22:01:29Z

Capibara: /* Filesystem Detection */

This page is for planning Carver 2.0.

Please, do not delete text (ideas) here. Use something like this:

<pre>
<s>bad idea</s>
:: good idea
</pre>

This will look like:

<s>bad idea</s>
:: good idea

= License =

BSD-3.
:: [[User:Joachim Metz|Joachim]] library based validators could require other licenses
::: Make the other libraries plug-able. If you them, you use them. [[User:Simsong|Simsong]] 06:34, 3 November 2008 (UTC)

= OS =

Linux/FreeBSD/MacOS
: Shouldn't this just match what the underlying afflib & sleuthkit cover? [[User:RB|RB]]
:: Yes, but you need to test and validate on each. Question: Do we want to support windows? [[User:Simsong|Simsong]] 21:09, 30 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] I think we would do wise to design with windows support from the start this will improve the platform independence from the start
:::: Agreed; I would even settle at first for being able to run against Cygwin. Note that I don't even own or use a copy of Windows, but the vast majority of forensic investigators do. [[User:RB|RB]] 14:01, 31 October 2008 (UTC)
:: [[User:Capibara|Rob J Meijer]] Leaning heavily on the autotools might be the way to go. I do however feel that support requirements for windows would not be essential. Being able to run from a virtual machine with the main storage mounted over cifs should however be tested and if possible tuned extensively.
:::: [[User:Joachim Metz|Joachim]] You'll need more than autotools to do native Windows support i.e. file access, UTF-16 support, wrap some basic system functions or have them available otherwise
::::::[[User:Capibara|Rob J Meijer]] That´s exactly my point, windows support as in being able to build and run on windows natively is much more trouble than its worth. Better make for a lean and mean autotools based build with little dependencies and no or little recursion, and better spent effort on a lean POLA design on POSIX based systems than on supporting building and running on non POSIX systems.

= Name tooling =

* [[User:Joachim Metz|Joachim]] A name for the tooling I propose coldcut
:: How about 'butcher'? ;) [[User:RB|RB]] 14:20, 31 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] cleaver ( scalpel on steroids ;-) )
* I would like to propose Gouge or Chisel :-) [[User:Capibara|Rob J Meijer]]

= Requirements =

[[User:Joachim Metz|Joachim]] Could we do a MoSCoW evaluation of these.
* AFF and EWF file images supported from scratch. ([[User:Joachim Metz|Joachim]] I would like to have raw/split raw and device access as well)
:: If we base our image i/o on afflib, we get all three with one interface. [[User:RB|RB]] Instead of letting the tools use afflib, better to write an afflib module for carvfs, and update the libewf module. The tool could than be oblivious of the file format. [[User:Capibara|Rob J Meijer]]
:::: [[User:Simsong|Simsong]] 06:29, 3 November 2008 (UTC) The problem with using carvfs is that this adds another dependency. Do you really want to require that people install carvfs in order to run the carver? What about having the thing ported to Windows?
:::::: [[User:Capibara|Rob J Meijer]] I would support adding one build dependency (libcarvpath) and removing two (libewf/libaff) by moving them to a layer more suited for them (carvfs) that would possibly allow some form of file handle (as cap) based POLA design. I am a proponent of making small things that do one and do one thing right, and to stack those to do what you need. In my view that would lead ideally to the following (simplified) chain:
::::::* recursive [[computer forensics framework]] (ocfa/pyflag)
::::::** The-(pola-based)-carving-tool
::::::*** The-carving-lib working on open fd's.
::::::**** libcarvpath
::::::***** carvfs (Over cifs/nfs-v4 on platforms that don't support Fuse).
::::::****** libewf
::::::****** libaff
::::::*** AppArmor (on supporting platforms)
::::::*** suid (on supporting platforms)
::::::*** iptables/ipfw (on supporting platforms)
:::::: As fow windows support, I would imagine making carvfs run over smb would come a long way, that is for as far as windows support is all that relevant.
:::::: There are two advantages to using libcarvpath and carvfs instead of libaff/libewf t this layer:
::::::* storage requirements for doing carving. Beyond what sleuthkit or alternatives provide I have seen many situations where carving was not done due to storage limitations.
::::::* File handles are like object capabilities. You can often do pretty simple POLA based implementations using file handles and something like AppArmor. POLA could IMHO be a strong weapon against the more nasty forms of anti forensics.
::::::Next to this, I would consider making different tools for different stages instead of one semi recursive one, and looking at how to integrate these tools into existing frameworks (ocfa/pyflag).
::::::Keep things simple but rigid and try to easily integrate things into existing frameworks as effectively as possible I would suggest.
::::::Please note, I am not ptoposing the lib/tool should be useless without libcarvpath, only that usage without carvfs should limit the
::::::supported image formats to raw images, and that libewf/libaff should be abstracted at the Fuse level or below and not at the tool level.
:::::::[[User:Joachim Metz|Joachim]] do you have an idea what the performance impact of this approach would be? It might be wise to do a proof of concept for this approach first.
::::::::[[User:Capibara|Rob J Meijer]] It would I think depend greatly on behavior of the carving lib/tool. Small 512 byte reads are relatively very expensive, 128kb reads have negligible impact. Here are some numbers from ntfs-3g: http://article.gmane.org/gmane.comp.file-systems.fuse.devel/6397/match=ntfs+3g+performance+ext3 that might be relevant. More relevant than performance might be library footprint. For example, using OCFA, we often would want to keep e few hundred images that total to tens of TB of projected storage size fuse mounted. If libewf/libaff have a big combined memory footprint in such cases, this can be a major issue for this approach.
:::: [[User:Joachim Metz|Joachim]] this layer should support multi threaded decompression of compressed image types, this speeds up IO
* [[User:Joachim Metz|Joachim]] volume/partition aware layer (what about carving unpartioned space)
* File system aware layer. This could be or make use of tsk-cp.
** By default, files are not carved. (clarify: only identified? [[User:RB|RB]]; I guess that it operates like [[Selective file dumper]] [[User:.FUF|.FUF]] 07:00, 29 October 2008 (UTC)). Alternatively, the tool could use libcarvpath and output carvpaths or create a directory with symlinks to carvpaths that point into a carvfs mountpoint [[User:Capibara|Rob J Meijer]].
* Plug-in architecture for identification/validation.
** [[User:Joachim Metz|Joachim]] support for multiple types of validators
*** dedicated validator
*** validator based on file library (i.e. we could specify/implement a file structure API for these)
*** configuration based validator (Can handle config files,like Revit07, to enter different file formats used by the carver.)

[[User:Joachim Metz|Joachim]]
Moderator: Could we limit the requirements for prototype version 1 of the tool to get a working version up and running ASAP?
And keep discussing future options?

I think the following set will be large enough to handle:
Input facilities
* IO support (AFF, device, EWF, RAW and split RAW)
:: Abstraction of input format and multi threaded decompression (spin-off code out of afflib?)
* Volume/Partitions support
:: at least for DOS based layout and GPT (spin-off code out of TSK/Photorec?)
* File system support
:: VFAT/NTFS (spin-off code out of TSK/Photorec?)

Carving facilities
* File format support using plug-able validator model (use dedicated validators Photorec/Scarve and/or wrap revit07 file format as validator?)
* Content support using plug-able validator model (to handle text/mbox base64)
* File system carving support (to handle file system fragments, could be linked to file system support layer?)
* Basic fragment handling

Output facilities
* audit/analysis/debug log
* extraction of result files

==Supported File Formats==
* Ship with validators for:
[[User:Joachim Metz|Joachim]] I think we should distinguish between file format validators and content validators
** JPEG
** PNG
** GIF
** MSOLE
** ZIP
** TAR (gz/bz2)

[[User:Joachim Metz|Joachim]] For a production carver we need at least the following formats
** Grapical Images
*** JPEG (the 3 different types with JFIF/EXIF support)
*** PNG
*** GIF
*** BMP
*** TIFF
** Office documents
*** OLE2 (Word/Excell content support)
*** PDF
*** Open Office/Office 2007 (ZIP+XML)
:: Extension validation? AFAIK, MS Office 2007 [[DOCX]] format uses plain ZIP (or not?), and carved files will (or not?) have .zip extension instead of DOCX. Is there any way to fix this (may be using the file list in zip)? [[User:.FUF|.FUF]] 20:25, 31 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] Addition: Office 2007 also has a binary file format which is also a ZIP-ed data

==Archive Files==
** Archive files
*** ZIP
*** 7z
*** gzip
*** bzip2
*** tar
*** RAR
** E-mail files
*** PFF (PST/OST)
*** MBOX (text based format, base64 content support)
** Audio/Video files
*** MPEG
*** MP2/MP3
*** AVI
*** ASF/WMV
*** QuickTime
*** MKV
** Printer spool files
*** EMF (if I remember correctly)
** Internet history files
*** index.dat
*** firefox (sqllite 3)
** Other files
*** thumbs.db
*** pagefile?

==Carving Strategies==
[[User:Joachim Metz|Joachim]] Note to moderator could this section be merged with the carving algorithm section?

* Simple fragment recovery carving using gap carving.
** [[User:Joachim Metz|Joachim]] have hook in for more advanced fragment recovery?
* Recovering of individual ZIP sections and JPEG icons that are not sector aligned.
** [[User:Joachim Metz|Joachim]] I would propose a generic fragment detection and recovery
* Autonomous operation (some mode of operation should be completely non-interactive, requiring no human intervention to complete [[User:RB|RB]])
** [[User:Joachim Metz|Joachim]] as much as possible, but allow to be overwritten by user
* [[User:Joachim Metz|Joachim]] When the tool output files the filenames should contain the offset in the input data (in hexadecimal?)
:: [[User:Mark Stam|Mark]] I really like the fact carved files are named after the physical or logical sector in which the file is found (photorec)
:::: [[User:Joachim Metz|Joachim]] This naming schema might cause duplicate name problem for extracting embedded files and extracting files from non sector aligned file systems.
* [[User:Joachim Metz|Joachim]] Should the tool allow to export embedded files?
* [[User:Joachim Metz|Joachim]] Should the tool allow to export fragments separately?
* [[User:Mark Stam|Mark]] I personally use photorec often for carving files in the whole volume (not only unallocated clusters), so I can store information about all potential interesting files in MySQL
:: [[User:Joachim Metz|Joachim]] interesting, Bas Kloet and me have been discussing to use information about allocated files in the recovery process, i.e. recovered fragments could be part of allocated files. Do we want to be able to extract them? Or could we rebuild the file from the fragments and the allocated files.
* [[User:Mark Stam|Mark]] It would also be nice if the files can be hashed immediately (MD5) so looking for them in other tools (for example Encase) is a snap

==Performance Requirements==
* Tested on 500GB-sized images. Should be able to carve a 500GB image in roughly 50% longer than it takes to read the image.
** Perhaps allocate a percentage budget per-validator (i.e. each validator adds N% to the carving time) [[User:RB|RB]]
** [[User:Joachim Metz|Joachim]] have multiple carving phases for precision/speed trade off?
* Parallelizable
** [[User:Joachim Metz|Joachim]] tunable for different architectures
* Configuration:
** Capability to parse some existing carvers' configuration files, either on-the-fly or as a one-way converter.
** Disengage internal configuration structure from configuration files, create parsers that present the expected structure
** [[User:Joachim Metz|Joachim]] The validator should deal with the file structure the carving algorithm should not know anything about the file structure (as in revit07 design)
** Either extend Scalpel/Foremost syntaxes for extended features or use a tertiary syntax ([[User:Joachim Metz|Joachim]] I would prefer a derivative of the revit07 configuration syntax which already has encountered some problems of dealing with defining file structure in a configuration file)

==Output==
* Can output audit.txt file.
* [[User:Joachim Metz|Joachim]] Can output database with offset analysis values i.e. for visualization tooling
* [[User:Joachim Metz|Joachim]] Can output debug log for debugging the algorithm/validation
* Easy integration into ascription software.
:: [[User:Joachim Metz|Joachim]] I'm no native speaker what do you mean with "ascription software"?
::: I think this was another non-native requesting easy scriptability. [[User:RB|RB]] 14:20, 31 October 2008 (UTC)
:::: [[User:Joachim Metz|Joachim]] that makes sense ;-)
::::: Incorrect. Ascription software is software that determines who the owner of a file is. [[User:Simsong|Simsong]] 06:36, 3 November 2008 (UTC)

= Ideas =
* Use as much TSK if possible. Don't carry your own FS implementation the way photorec does.
:: [[User:Joachim Metz|Joachim]] using TSK as much as possible would not allow to add your own file system support (i.e. mobile phones, memory structures, cap files) I would propose wrapping TSK and using it as much as possible but allow to integrate own FS implementations.
* Extracting/carving data from [[Thumbs.db]]? I've used [[foremost]] for it with some success. [[Vinetto]] has some critical bugs :( [[User:.FUF|.FUF]] 19:18, 28 October 2008 (UTC)

==Recursive Carving==
[[User:Joachim Metz|Joachim]] do we want to support (let's call it) 'recursive in file carving' (for now) this is different from embedded files because there is a file system structure in the file and not just another file structure
* Is it just me, or do a lot of the above (and below) ideas somewhat skirt around the fact that many of us want recursive carving? Can we bend back to that instead of discussing object particulars? I think this can be distilled down to three requirements:
** Simple recursion: once an object is identified, have the ability to re-carve it for internal structures
** Directed recursion: the carver should be able to be directed at arbitrary blobs and told to carve it as a specified type. This allows programmatically more simple methods of dealing with unidentifiably compressed or encrypted data. Or filesystem fragments.
** Export: the ability to export an object (recognized or not) for later or external "recursion". Should go without saying for a carver, but...
:--[[User:RB|RB]] 18:45, 2 November 2008 (UTC)
:: [[User:Simsong|Simsong]] 06:30, 3 November 2008 (UTC) pyflag already does recursive carving. Are we just going to reimplement pyflag as a single executable?

==Library Dependencies==
[[User:Capibara|Rob J Meijer]] :
* Use libcarvpath whenever possible and by default to avoid high storage requirements.
:: [[User:Joachim Metz|Joachim]] For easy deployment I would not opt for making an integral part of the tool solely dependant on a single external library or the library must be integrated in the package
::[[User:Capibara|Rob J Meijer]] Integrating libraries (libtsk,libaff.libewf,libcarvpath etc) is bad practice, autotools are your friend IMO.
:: [[User:Joachim Metz|Joachim]] I'm not talking about integrating (shared) libraries. I'm talking about that an integral part of a tool should be part of it's package. Why can't the tool package contain shared or static libraries for local use? A far worse thing to do is to have a large set of dependencies and making the tool difficult to install for most users. The tool package should contain the most necessary code. afflib/libewf support could be detected by the autotools a neat separation of functionality.
::: From a packager's standpoint, [[User:Joachim Metz|Joachim]]'s other libraries do a really good job of this, carrying around what they need but using a system-global version if available. [[User:RB|RB]]
* libtsk
* libaff ? : possibly the discussion in the requirements section should move to his section.
* libewf ? : possibly the discussion in the requirements section should move to his section.
* posix ? : Can we depend especially on the availability of UNIX domain sockets and the possibility to use msg_accrights for passing opn file handles as ocaps?

==Filesystem Detection==
* Dont stop with filesystem detection after the first match. Often if a partition is reused with a new FS and is not all that full yet, much of the old FS can still be valid. I have seen this with ext2/fat. The fact that you have identified a valid FS on a partition doesn't mean there isn't an(almost) valid second FS that would yield additional files. Identifying doubly allocated space might in some cases also be relevant.
:: [[User:Joachim Metz|Joachim]] What your saying is that dealing with file system fragments should be part of the carving algorithm
* Allow use where filesystem based carving is done by other tool, and the tool is used as second stage on (sets of) unallocated block (pseudo) files and/or non FS partition (pseudo) files.
:: [[User:Joachim Metz|Joachim]] I would not opt for this. The tool would be dependent on other tools and their data format, which makes the tool difficult to maintain. I would opt to integrate the functionality of having multiple recovery phases (stages) and allow the tooling to run the phases after one and other or separately.
::[[User:Capibara|Rob J Meijer]] More generically, I feel a way should exist to communicate the 'left overs' a previous (non open, for example LE-only) tool left.
:: [[User:Joachim Metz|Joachim]] I guess if the tool is designed to handle multiple phases it should store its data somewhere. So it should be possible to convert results of such non open tooling to the format required. However I would opt to design the recovery functionality of these non-open tools into open tools. And not to limit ourselves making translators due to the design of these non-open tools.
* Ability to be used as a library instead of a tool. Ability to access metadata true library, and thus the ability to set metadata from the carving modules. This would be extremely usefull for integrating the project into a [[computer forensics framework]] .
:: [[User:Joachim Metz|Joachim]] I guess most of the code could be integrated into libraries, but I would not opt integrating tool functionality into a library
* [[User:Mark Stam|Mark]] I think it would be very handy to have a CSV, TSV, XML or other delimited output (log)file with information about carved files. This output file can then be stored in a database or Excel sheet (report function)

==Anti forensics and system integrity concerns==
* It might be very interesting to look at the possibilities of using a multi process style of module support and combine it with a least authority design. On platforms that support AppArmor (or similar) and uid based firewall rules, this could make for the first true POLA (principle of least authority) based forensic tool ever. POLA based forensics tools should make for a strong integrity guard against many anti forensics. Alternatively we could look at integrating a capability secure language (E?) for implementation of at least validation modules. I don't expect this idea to make it, but mentioning it I hope might spark off less strong alternatives that at least partially address the integrity + anti-forensics problem. If we can in some way introduce POLA to a wider forensics public, other tools might also pick up on it what would be great.
:: [[User:Joachim Metz|Joachim]] Could you give an example of how you see this in action?
::::[[User:Capibara|Rob J Meijer]] I see two layers where using POLA could be applied. The best one would require one of the folowing as prerequisites:
::::* The libaff/libewf layer is moved to a fuse implementation (for example carvfs).
::::* Libewf/Libaff are updated to accept opened filhandles instead of demanding to open their own files.
::::If one of these is fulfilled, than the tool running as some user can just have the simple task of opening the image files, starting up the 'real' tool and handing over the appropriate file handles. If the real tool runs with a restrictive AppArmor profile, and is started suid to a tool specific user that also has its own iptables uid based filter, than the real tool will run with least authority.
:::: A second alternative, if neither of the first prerequisite could not be bet, would be to run the modules as confined processes and have a non confined process run as proxy for the first.
:::: A third probably far fetched alternative would be to embed an object capability language in the tool and make the module interface thus that modules are to be written in this ocap language.
::::A 4th alternative might include minorfs or plash, but I havn't geven those sufficient thinking hours yet.

== Format syntax specification ==
* Carving data structures. For example, extract all TCP headers from image by defining TCP header structure and some fields (e.g. source port > 1024, dest port = 80). This will extract all data matching the pattern and write a file with other fields. Another example is carving INFO2 structures and URL activity records from index.dat [[User:.FUF|.FUF]] 20:51, 28 October 2008 (UTC)
** This has the opportunity to be extended to the concept of "point at blob FOO and interpret it as BAR"
.FUF added:
The main idea is to allow users to define structures, for example (in pascal-like form):

<pre>
Field1: Byte = 123;
SomeTextLength: DWORD;
SomeText: string[SomeTextLength];
Field4: Char = 'r';
...
</pre>

This will produce something like this:
<pre>
Field1 = 123
SomeTextLength = 5
SomeText = 'abcd1'
Field4 = 'r'
</pre>

(In text or raw forms.)

Opinions?

Opinion: Simple pattern identification like that may not suffice, I think Simson's original intent was not only to identify but to allow for validation routines (plugins, as the original wording was). As such, the format syntax would need to implement a large chunk of some programming language in order to be sufficiently flexible. [[User:RB|RB]]

[[User:Joachim Metz|Joachim]]
In my option your example is too limited. Making the revit configuration I learned you'll need a near programming language to specify some file formats.
A simple descriptive language is too limiting. I would also go for 2 bytes with endianess instead of using terminology like WORD and small integer, it's much more clear. The configuration also needs to deal with aspects like cardinality, required and optional structures.
:: This is simply data structures carving, see ideas above. Somebody (I cannot track so many changes per day) separated the original text. There is no need to count and join different structures. [[User:.FUF|.FUF]] 19:53, 31 October 2008 (UTC)
:::: [[User:Joachim Metz|Joachim]] This was probably me is the text back in it's original form?
:::: I started it by moving your Revit07 comment to the validator/plugin section in [http://www.forensicswiki.org/index.php?title=Carver_2.0_Planning_Page&diff=prev&oldid=7583 this edit], since I was still at that point thinking operational configuration for that section, not parser configurations. [[User:RB|RB]]
:::: [[User:Joachim Metz|Joachim]] I renamed the title to format syntax, clarity is important ;-)

Please take a look at the revit07 format syntax specification (configuration). It's not there yet but goes a far way. Some things currently missing:
* bitwise alignment
* handling encapsulated streams (MPEG/capture files)
* handling content based formats (MBOX)

=Caving algorithm =
[[User:Joachim Metz|Joachim]]
* should we allow for multiple carving phases (runs/stages)?
:: I opt yes (separation of concern)
* should we allow for multiple carving algorithms?
:: I opt yes, this allows testing of different approaches
* Should the algorithm try to do as much in 1 run over the input data? To reduce IO?
:: I opt that the tool should allow for multiple and single run over the input data to minimize the IO or the CPU as bottleneck
* Interaction between algorithm and validators
** does the algorithm passes data blocks to the validators?
** does a validator need to maintain a state?
** does a validator need to revert a state?
** How do we deal with embedded files and content validation? Do the validators call another validator?
* do we use the assumption that a data block can be used by a single file (with the exception of embedded/encapsulated files)?
* Revit07 allows for multiple concurrent result files states to deal with fragmentation. One has the attribute of being active (the preferred) and the other passive. Do we want/need something similar? The algorithm adds block of input data (offsets) to these result files states.
** if so what info would these result files states require (type, list of input data blocks)
* how do we deal with file system remainders?
** Can we abstract them and compare them against available file system information?
* Do we carve file systems in files?
:: I opt that at least the validator uses this information

==Caving scenarios ==
[[User:Joachim Metz|Joachim]]
* normal file (file structure, loose text based structure (more a content structure?))
* fragmented file (the file entirely exist)
* a file fragment (the file does not entirely exist)
* intertwined file
* encapsulated file (MPEG/network capture)
* embedded file (JPEG thumbnail)
* obfuscation ('encrypted' PFF) this also entails encryption and/or compression
* file system in file

=File System Awareness =
==Background: Why be File System Aware?==
Advantages of being FS aware:
* You can pick up sector allocation sizes
:: [[User:Joachim Metz|Joachim]] do you mean file system block sizes?
* Some file systems may store things off sector boundaries. (ReiserFS with tail packing)
* Increasingly file systems have compression (NTFS compression)
* Carve just the sectors that are not in allocated files.

==Tasks that would be required==

==Discussion==
:: As noted above, TSK should be utilized as much as possible, particularly the filesystem-aware portion. If we want to identify filesystems outside of its supported set, it would be more worth our time to work on implementing them there than in the carver itself. [[User:RB|RB]]

:::: I guess this tool operates like [[Selective file dumper]] and can recover files in both ways (or not?). Recovering files by using carving can recover files in situations where sleuthkit does nothing (e.g. file on NTFS was deleted using ntfs-3g, or filesystem was destroyed or just unknown). And we should build the list of filesystems supported by carver, not by TSK. [[User:.FUF|.FUF]] 07:08, 29 October 2008 (UTC)

:: This tool is still in the early planning stages (requirements discovery), hence few operational details (like precise modes of operation) have been fleshed out - those will and should come later. The justification for strictly using TSK for the filesystem-sensitive approach is simple: TSK has good filesystem APIs, and it would be foolish to create yet another standalone, incompatible implementation of filesystem(foo) when time would be better spent improving those in TSK, aiding other methods of analysis as well. This is the same reason individuals that have implemented several other carvers are participating: de-duplication of effort. [[User:RB|RB]]

:: [[User:Joachim Metz|Joachim]] A design problem might be that TSK currently is a single library operating on multiple layers (storage media IO, volume/partition analysis and file system analysis). I'm not aware how easily the parts can be used separately. But I estimate that for the carver we want to use these 3 layers differently than TSK currently does.

[[User:Joachim Metz|Joachim]] I would like to have the carver (recovery tool) also do recovery using file allocation data or remainders of file allocation data.

[[User:Joachim Metz|Joachim]]
I would go as far to ask you all to look beyond the carver as a tool and look from the perspective of the carver as part of the forensic investigation process. In my eyes certain information needed/acquired by the carver could be also very useful investigative information i.e. what part of a hard disk contains empty sectors.

=Supportive tooling=
[[User:Joachim Metz|Joachim]]
* validator (definitions) tester (detest in revit07)
* tool to make configuration based definitions
* post carving validation
* the carver needs to provide support for fuse mount of carved files (carvfs)

=Testing =
[[User:Joachim Metz|Joachim]]
* automated testing
* test data

=Validator Construction=
Options:
* Write validators in C/C++
** [[User:Joachim Metz|Joachim]] you mean dedicated validators
* Have a scripting language for writing them (python? Perl?) our own?
** [[User:Joachim Metz|Joachim]] use easy to embed programming languages i.e. Phyton or Lua
* Use existing programs (libjpeg?) as plug-in validators?
** [[User:Joachim Metz|Joachim]] define a file structure api for this

=Existing Code that we have=
[[User:Joachim Metz|Joachim]]
Please add any missing links

Documentation/Articles
* DFRWS2006/2007 carving challenge results
* DFRWS2008 paper on carving

Carvers
* DFRWS2006/2007 carving challenge results
* photorec (http://www.cgsecurity.org/wiki/PhotoRec)
* revit06 and revit07 (http://sourceforge.net/projects/revit/)
* s3/scarve

Possible file structure validator libraries
* divers existing file support libraries
* libole2 (inhouse experimental code of OLE2 support)
* libpff (alpha release for PFF (PST/OST) file support) (http://sourceforge.net/projects/libpff/)

Input support
* AFF (http://www.afflib.org/)
* EWF (http://sourceforge.net/projects/libewf/)
* TSK device & raw & split raw (http://www.sleuthkit.org/)

Volume/Partition support
* disktype (http://disktype.sourceforge.net/)
* testdisk (http://www.cgsecurity.org/wiki/TestDisk)
* TSK

File system support
* TSK
* photorec FS code
* implementations of FS in Linux/BSD

Content support

Zero storage support
* libcarvpath ( http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=210704 )
* carvfs ( http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=210954 )
* tsk-cp ( http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=267227 )
* carvfsmodewf (http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=268256 )
POLA
* joe-e (java) ( http://code.google.com/p/joe-e/ )
* Emily (ocaml) ( http://erights.org/download/emily/ )
* the E language ( http://www.erights.org/ )
* AppArmor
* iptables/ipfw
* minorfs ( http://polacanthus.net/minorfs.html )
* plash ( http://plash.beasts.org/wiki/ )

=Implementation Timeline=
# gather the available resources/ideas/wishes/needs etc. (I guess we're in this phase)
# start discussing a high level design (in terms of algorithm, facilities, information needed)
## input formats facility
## partition/volume facility
## file system facility
## file format facility
## content facility
## how to deal with fragment detection (do the validators allow for fragment detection?)
## how to deal with recombination of fragments
## do we want multiple carving phases in light of speed/precision tradeoffs
# start detailing parts of the design
## Discuss options for a grammar driven validator?
## Hard-coded plug-ins?
## Which existing code can we use?
# start building/assembling parts of the tooling for a prototype
## Implement simple file carving with validation.
## Implement gap carving
# Initial Release
# Implement the ''threaded carving'' that [[User:.FUF|.FUF]] is describing above.

[[User:Joachim Metz|Joachim]] Shouldn't multi threaded carving (MTC) not be part of the 1st version?
The MT approach makes for different design decisions
: It is virtually impossible to turn a non-MT application into an MT application .[[User:Simsong|Simsong]] 06:37, 3 November 2008 (UTC)

Carver 2.0 Planning Page

2008-12-03T22:00:36Z

Capibara: /* Filesystem Detection */

This page is for planning Carver 2.0.

Please, do not delete text (ideas) here. Use something like this:

<pre>
<s>bad idea</s>
:: good idea
</pre>

This will look like:

<s>bad idea</s>
:: good idea

= License =

BSD-3.
:: [[User:Joachim Metz|Joachim]] library based validators could require other licenses
::: Make the other libraries plug-able. If you them, you use them. [[User:Simsong|Simsong]] 06:34, 3 November 2008 (UTC)

= OS =

Linux/FreeBSD/MacOS
: Shouldn't this just match what the underlying afflib & sleuthkit cover? [[User:RB|RB]]
:: Yes, but you need to test and validate on each. Question: Do we want to support windows? [[User:Simsong|Simsong]] 21:09, 30 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] I think we would do wise to design with windows support from the start this will improve the platform independence from the start
:::: Agreed; I would even settle at first for being able to run against Cygwin. Note that I don't even own or use a copy of Windows, but the vast majority of forensic investigators do. [[User:RB|RB]] 14:01, 31 October 2008 (UTC)
:: [[User:Capibara|Rob J Meijer]] Leaning heavily on the autotools might be the way to go. I do however feel that support requirements for windows would not be essential. Being able to run from a virtual machine with the main storage mounted over cifs should however be tested and if possible tuned extensively.
:::: [[User:Joachim Metz|Joachim]] You'll need more than autotools to do native Windows support i.e. file access, UTF-16 support, wrap some basic system functions or have them available otherwise
::::::[[User:Capibara|Rob J Meijer]] That´s exactly my point, windows support as in being able to build and run on windows natively is much more trouble than its worth. Better make for a lean and mean autotools based build with little dependencies and no or little recursion, and better spent effort on a lean POLA design on POSIX based systems than on supporting building and running on non POSIX systems.

= Name tooling =

* [[User:Joachim Metz|Joachim]] A name for the tooling I propose coldcut
:: How about 'butcher'? ;) [[User:RB|RB]] 14:20, 31 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] cleaver ( scalpel on steroids ;-) )
* I would like to propose Gouge or Chisel :-) [[User:Capibara|Rob J Meijer]]

= Requirements =

[[User:Joachim Metz|Joachim]] Could we do a MoSCoW evaluation of these.
* AFF and EWF file images supported from scratch. ([[User:Joachim Metz|Joachim]] I would like to have raw/split raw and device access as well)
:: If we base our image i/o on afflib, we get all three with one interface. [[User:RB|RB]] Instead of letting the tools use afflib, better to write an afflib module for carvfs, and update the libewf module. The tool could than be oblivious of the file format. [[User:Capibara|Rob J Meijer]]
:::: [[User:Simsong|Simsong]] 06:29, 3 November 2008 (UTC) The problem with using carvfs is that this adds another dependency. Do you really want to require that people install carvfs in order to run the carver? What about having the thing ported to Windows?
:::::: [[User:Capibara|Rob J Meijer]] I would support adding one build dependency (libcarvpath) and removing two (libewf/libaff) by moving them to a layer more suited for them (carvfs) that would possibly allow some form of file handle (as cap) based POLA design. I am a proponent of making small things that do one and do one thing right, and to stack those to do what you need. In my view that would lead ideally to the following (simplified) chain:
::::::* recursive [[computer forensics framework]] (ocfa/pyflag)
::::::** The-(pola-based)-carving-tool
::::::*** The-carving-lib working on open fd's.
::::::**** libcarvpath
::::::***** carvfs (Over cifs/nfs-v4 on platforms that don't support Fuse).
::::::****** libewf
::::::****** libaff
::::::*** AppArmor (on supporting platforms)
::::::*** suid (on supporting platforms)
::::::*** iptables/ipfw (on supporting platforms)
:::::: As fow windows support, I would imagine making carvfs run over smb would come a long way, that is for as far as windows support is all that relevant.
:::::: There are two advantages to using libcarvpath and carvfs instead of libaff/libewf t this layer:
::::::* storage requirements for doing carving. Beyond what sleuthkit or alternatives provide I have seen many situations where carving was not done due to storage limitations.
::::::* File handles are like object capabilities. You can often do pretty simple POLA based implementations using file handles and something like AppArmor. POLA could IMHO be a strong weapon against the more nasty forms of anti forensics.
::::::Next to this, I would consider making different tools for different stages instead of one semi recursive one, and looking at how to integrate these tools into existing frameworks (ocfa/pyflag).
::::::Keep things simple but rigid and try to easily integrate things into existing frameworks as effectively as possible I would suggest.
::::::Please note, I am not ptoposing the lib/tool should be useless without libcarvpath, only that usage without carvfs should limit the
::::::supported image formats to raw images, and that libewf/libaff should be abstracted at the Fuse level or below and not at the tool level.
:::::::[[User:Joachim Metz|Joachim]] do you have an idea what the performance impact of this approach would be? It might be wise to do a proof of concept for this approach first.
::::::::[[User:Capibara|Rob J Meijer]] It would I think depend greatly on behavior of the carving lib/tool. Small 512 byte reads are relatively very expensive, 128kb reads have negligible impact. Here are some numbers from ntfs-3g: http://article.gmane.org/gmane.comp.file-systems.fuse.devel/6397/match=ntfs+3g+performance+ext3 that might be relevant. More relevant than performance might be library footprint. For example, using OCFA, we often would want to keep e few hundred images that total to tens of TB of projected storage size fuse mounted. If libewf/libaff have a big combined memory footprint in such cases, this can be a major issue for this approach.
:::: [[User:Joachim Metz|Joachim]] this layer should support multi threaded decompression of compressed image types, this speeds up IO
* [[User:Joachim Metz|Joachim]] volume/partition aware layer (what about carving unpartioned space)
* File system aware layer. This could be or make use of tsk-cp.
** By default, files are not carved. (clarify: only identified? [[User:RB|RB]]; I guess that it operates like [[Selective file dumper]] [[User:.FUF|.FUF]] 07:00, 29 October 2008 (UTC)). Alternatively, the tool could use libcarvpath and output carvpaths or create a directory with symlinks to carvpaths that point into a carvfs mountpoint [[User:Capibara|Rob J Meijer]].
* Plug-in architecture for identification/validation.
** [[User:Joachim Metz|Joachim]] support for multiple types of validators
*** dedicated validator
*** validator based on file library (i.e. we could specify/implement a file structure API for these)
*** configuration based validator (Can handle config files,like Revit07, to enter different file formats used by the carver.)

[[User:Joachim Metz|Joachim]]
Moderator: Could we limit the requirements for prototype version 1 of the tool to get a working version up and running ASAP?
And keep discussing future options?

I think the following set will be large enough to handle:
Input facilities
* IO support (AFF, device, EWF, RAW and split RAW)
:: Abstraction of input format and multi threaded decompression (spin-off code out of afflib?)
* Volume/Partitions support
:: at least for DOS based layout and GPT (spin-off code out of TSK/Photorec?)
* File system support
:: VFAT/NTFS (spin-off code out of TSK/Photorec?)

Carving facilities
* File format support using plug-able validator model (use dedicated validators Photorec/Scarve and/or wrap revit07 file format as validator?)
* Content support using plug-able validator model (to handle text/mbox base64)
* File system carving support (to handle file system fragments, could be linked to file system support layer?)
* Basic fragment handling

Output facilities
* audit/analysis/debug log
* extraction of result files

==Supported File Formats==
* Ship with validators for:
[[User:Joachim Metz|Joachim]] I think we should distinguish between file format validators and content validators
** JPEG
** PNG
** GIF
** MSOLE
** ZIP
** TAR (gz/bz2)

[[User:Joachim Metz|Joachim]] For a production carver we need at least the following formats
** Grapical Images
*** JPEG (the 3 different types with JFIF/EXIF support)
*** PNG
*** GIF
*** BMP
*** TIFF
** Office documents
*** OLE2 (Word/Excell content support)
*** PDF
*** Open Office/Office 2007 (ZIP+XML)
:: Extension validation? AFAIK, MS Office 2007 [[DOCX]] format uses plain ZIP (or not?), and carved files will (or not?) have .zip extension instead of DOCX. Is there any way to fix this (may be using the file list in zip)? [[User:.FUF|.FUF]] 20:25, 31 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] Addition: Office 2007 also has a binary file format which is also a ZIP-ed data

==Archive Files==
** Archive files
*** ZIP
*** 7z
*** gzip
*** bzip2
*** tar
*** RAR
** E-mail files
*** PFF (PST/OST)
*** MBOX (text based format, base64 content support)
** Audio/Video files
*** MPEG
*** MP2/MP3
*** AVI
*** ASF/WMV
*** QuickTime
*** MKV
** Printer spool files
*** EMF (if I remember correctly)
** Internet history files
*** index.dat
*** firefox (sqllite 3)
** Other files
*** thumbs.db
*** pagefile?

==Carving Strategies==
[[User:Joachim Metz|Joachim]] Note to moderator could this section be merged with the carving algorithm section?

* Simple fragment recovery carving using gap carving.
** [[User:Joachim Metz|Joachim]] have hook in for more advanced fragment recovery?
* Recovering of individual ZIP sections and JPEG icons that are not sector aligned.
** [[User:Joachim Metz|Joachim]] I would propose a generic fragment detection and recovery
* Autonomous operation (some mode of operation should be completely non-interactive, requiring no human intervention to complete [[User:RB|RB]])
** [[User:Joachim Metz|Joachim]] as much as possible, but allow to be overwritten by user
* [[User:Joachim Metz|Joachim]] When the tool output files the filenames should contain the offset in the input data (in hexadecimal?)
:: [[User:Mark Stam|Mark]] I really like the fact carved files are named after the physical or logical sector in which the file is found (photorec)
:::: [[User:Joachim Metz|Joachim]] This naming schema might cause duplicate name problem for extracting embedded files and extracting files from non sector aligned file systems.
* [[User:Joachim Metz|Joachim]] Should the tool allow to export embedded files?
* [[User:Joachim Metz|Joachim]] Should the tool allow to export fragments separately?
* [[User:Mark Stam|Mark]] I personally use photorec often for carving files in the whole volume (not only unallocated clusters), so I can store information about all potential interesting files in MySQL
:: [[User:Joachim Metz|Joachim]] interesting, Bas Kloet and me have been discussing to use information about allocated files in the recovery process, i.e. recovered fragments could be part of allocated files. Do we want to be able to extract them? Or could we rebuild the file from the fragments and the allocated files.
* [[User:Mark Stam|Mark]] It would also be nice if the files can be hashed immediately (MD5) so looking for them in other tools (for example Encase) is a snap

==Performance Requirements==
* Tested on 500GB-sized images. Should be able to carve a 500GB image in roughly 50% longer than it takes to read the image.
** Perhaps allocate a percentage budget per-validator (i.e. each validator adds N% to the carving time) [[User:RB|RB]]
** [[User:Joachim Metz|Joachim]] have multiple carving phases for precision/speed trade off?
* Parallelizable
** [[User:Joachim Metz|Joachim]] tunable for different architectures
* Configuration:
** Capability to parse some existing carvers' configuration files, either on-the-fly or as a one-way converter.
** Disengage internal configuration structure from configuration files, create parsers that present the expected structure
** [[User:Joachim Metz|Joachim]] The validator should deal with the file structure the carving algorithm should not know anything about the file structure (as in revit07 design)
** Either extend Scalpel/Foremost syntaxes for extended features or use a tertiary syntax ([[User:Joachim Metz|Joachim]] I would prefer a derivative of the revit07 configuration syntax which already has encountered some problems of dealing with defining file structure in a configuration file)

==Output==
* Can output audit.txt file.
* [[User:Joachim Metz|Joachim]] Can output database with offset analysis values i.e. for visualization tooling
* [[User:Joachim Metz|Joachim]] Can output debug log for debugging the algorithm/validation
* Easy integration into ascription software.
:: [[User:Joachim Metz|Joachim]] I'm no native speaker what do you mean with "ascription software"?
::: I think this was another non-native requesting easy scriptability. [[User:RB|RB]] 14:20, 31 October 2008 (UTC)
:::: [[User:Joachim Metz|Joachim]] that makes sense ;-)
::::: Incorrect. Ascription software is software that determines who the owner of a file is. [[User:Simsong|Simsong]] 06:36, 3 November 2008 (UTC)

= Ideas =
* Use as much TSK if possible. Don't carry your own FS implementation the way photorec does.
:: [[User:Joachim Metz|Joachim]] using TSK as much as possible would not allow to add your own file system support (i.e. mobile phones, memory structures, cap files) I would propose wrapping TSK and using it as much as possible but allow to integrate own FS implementations.
* Extracting/carving data from [[Thumbs.db]]? I've used [[foremost]] for it with some success. [[Vinetto]] has some critical bugs :( [[User:.FUF|.FUF]] 19:18, 28 October 2008 (UTC)

==Recursive Carving==
[[User:Joachim Metz|Joachim]] do we want to support (let's call it) 'recursive in file carving' (for now) this is different from embedded files because there is a file system structure in the file and not just another file structure
* Is it just me, or do a lot of the above (and below) ideas somewhat skirt around the fact that many of us want recursive carving? Can we bend back to that instead of discussing object particulars? I think this can be distilled down to three requirements:
** Simple recursion: once an object is identified, have the ability to re-carve it for internal structures
** Directed recursion: the carver should be able to be directed at arbitrary blobs and told to carve it as a specified type. This allows programmatically more simple methods of dealing with unidentifiably compressed or encrypted data. Or filesystem fragments.
** Export: the ability to export an object (recognized or not) for later or external "recursion". Should go without saying for a carver, but...
:--[[User:RB|RB]] 18:45, 2 November 2008 (UTC)
:: [[User:Simsong|Simsong]] 06:30, 3 November 2008 (UTC) pyflag already does recursive carving. Are we just going to reimplement pyflag as a single executable?

==Library Dependencies==
[[User:Capibara|Rob J Meijer]] :
* Use libcarvpath whenever possible and by default to avoid high storage requirements.
:: [[User:Joachim Metz|Joachim]] For easy deployment I would not opt for making an integral part of the tool solely dependant on a single external library or the library must be integrated in the package
::[[User:Capibara|Rob J Meijer]] Integrating libraries (libtsk,libaff.libewf,libcarvpath etc) is bad practice, autotools are your friend IMO.
:: [[User:Joachim Metz|Joachim]] I'm not talking about integrating (shared) libraries. I'm talking about that an integral part of a tool should be part of it's package. Why can't the tool package contain shared or static libraries for local use? A far worse thing to do is to have a large set of dependencies and making the tool difficult to install for most users. The tool package should contain the most necessary code. afflib/libewf support could be detected by the autotools a neat separation of functionality.
::: From a packager's standpoint, [[User:Joachim Metz|Joachim]]'s other libraries do a really good job of this, carrying around what they need but using a system-global version if available. [[User:RB|RB]]
* libtsk
* libaff ? : possibly the discussion in the requirements section should move to his section.
* libewf ? : possibly the discussion in the requirements section should move to his section.
* posix ? : Can we depend especially on the availability of UNIX domain sockets and the possibility to use msg_accrights for passing opn file handles as ocaps?

==Filesystem Detection==
* Dont stop with filesystem detection after the first match. Often if a partition is reused with a new FS and is not all that full yet, much of the old FS can still be valid. I have seen this with ext2/fat. The fact that you have identified a valid FS on a partition doesn't mean there isn't an(almost) valid second FS that would yield additional files. Identifying doubly allocated space might in some cases also be relevant.
:: [[User:Joachim Metz|Joachim]] What your saying is that dealing with file system fragments should be part of the carving algorithm
* Allow use where filesystem based carving is done by other tool, and the tool is used as second stage on (sets of) unallocated block (pseudo) files and/or non FS partition (pseudo) files.
:: [[User:Joachim Metz|Joachim]] I would not opt for this. The tool would be dependent on other tools and their data format, which makes the tool difficult to maintain. I would opt to integrate the functionality of having multiple recovery phases (stages) and allow the tooling to run the phases after one and other or separately.
::[[User:Capibara|Rob J Meijer]] More generically, I feel a way should exist to communicate the 'left overs' a previous (non open, for example LE-only) tool left.
:: [[User:Joachim Metz|Joachim]] I guess if the tool is designed to handle multiple phases it should store its data somewhere. So it should be possible to convert results of such non open tooling to the format required. However I would opt to design the recovery functionality of these non-open tools into open tools. And not to limit ourselves making translators due to the design of these non-open tools.
* Ability to be used as a library instead of a tool. Ability to access metadata true library, and thus the ability to set metadata from the carving modules. This would be extremely usefull for integrating the project into a [[computerr forensics framework]] .
:: [[User:Joachim Metz|Joachim]] I guess most of the code could be integrated into libraries, but I would not opt integrating tool functionality into a library
* [[User:Mark Stam|Mark]] I think it would be very handy to have a CSV, TSV, XML or other delimited output (log)file with information about carved files. This output file can then be stored in a database or Excel sheet (report function)

==Anti forensics and system integrity concerns==
* It might be very interesting to look at the possibilities of using a multi process style of module support and combine it with a least authority design. On platforms that support AppArmor (or similar) and uid based firewall rules, this could make for the first true POLA (principle of least authority) based forensic tool ever. POLA based forensics tools should make for a strong integrity guard against many anti forensics. Alternatively we could look at integrating a capability secure language (E?) for implementation of at least validation modules. I don't expect this idea to make it, but mentioning it I hope might spark off less strong alternatives that at least partially address the integrity + anti-forensics problem. If we can in some way introduce POLA to a wider forensics public, other tools might also pick up on it what would be great.
:: [[User:Joachim Metz|Joachim]] Could you give an example of how you see this in action?
::::[[User:Capibara|Rob J Meijer]] I see two layers where using POLA could be applied. The best one would require one of the folowing as prerequisites:
::::* The libaff/libewf layer is moved to a fuse implementation (for example carvfs).
::::* Libewf/Libaff are updated to accept opened filhandles instead of demanding to open their own files.
::::If one of these is fulfilled, than the tool running as some user can just have the simple task of opening the image files, starting up the 'real' tool and handing over the appropriate file handles. If the real tool runs with a restrictive AppArmor profile, and is started suid to a tool specific user that also has its own iptables uid based filter, than the real tool will run with least authority.
:::: A second alternative, if neither of the first prerequisite could not be bet, would be to run the modules as confined processes and have a non confined process run as proxy for the first.
:::: A third probably far fetched alternative would be to embed an object capability language in the tool and make the module interface thus that modules are to be written in this ocap language.
::::A 4th alternative might include minorfs or plash, but I havn't geven those sufficient thinking hours yet.

== Format syntax specification ==
* Carving data structures. For example, extract all TCP headers from image by defining TCP header structure and some fields (e.g. source port > 1024, dest port = 80). This will extract all data matching the pattern and write a file with other fields. Another example is carving INFO2 structures and URL activity records from index.dat [[User:.FUF|.FUF]] 20:51, 28 October 2008 (UTC)
** This has the opportunity to be extended to the concept of "point at blob FOO and interpret it as BAR"
.FUF added:
The main idea is to allow users to define structures, for example (in pascal-like form):

<pre>
Field1: Byte = 123;
SomeTextLength: DWORD;
SomeText: string[SomeTextLength];
Field4: Char = 'r';
...
</pre>

This will produce something like this:
<pre>
Field1 = 123
SomeTextLength = 5
SomeText = 'abcd1'
Field4 = 'r'
</pre>

(In text or raw forms.)

Opinions?

Opinion: Simple pattern identification like that may not suffice, I think Simson's original intent was not only to identify but to allow for validation routines (plugins, as the original wording was). As such, the format syntax would need to implement a large chunk of some programming language in order to be sufficiently flexible. [[User:RB|RB]]

[[User:Joachim Metz|Joachim]]
In my option your example is too limited. Making the revit configuration I learned you'll need a near programming language to specify some file formats.
A simple descriptive language is too limiting. I would also go for 2 bytes with endianess instead of using terminology like WORD and small integer, it's much more clear. The configuration also needs to deal with aspects like cardinality, required and optional structures.
:: This is simply data structures carving, see ideas above. Somebody (I cannot track so many changes per day) separated the original text. There is no need to count and join different structures. [[User:.FUF|.FUF]] 19:53, 31 October 2008 (UTC)
:::: [[User:Joachim Metz|Joachim]] This was probably me is the text back in it's original form?
:::: I started it by moving your Revit07 comment to the validator/plugin section in [http://www.forensicswiki.org/index.php?title=Carver_2.0_Planning_Page&diff=prev&oldid=7583 this edit], since I was still at that point thinking operational configuration for that section, not parser configurations. [[User:RB|RB]]
:::: [[User:Joachim Metz|Joachim]] I renamed the title to format syntax, clarity is important ;-)

Please take a look at the revit07 format syntax specification (configuration). It's not there yet but goes a far way. Some things currently missing:
* bitwise alignment
* handling encapsulated streams (MPEG/capture files)
* handling content based formats (MBOX)

=Caving algorithm =
[[User:Joachim Metz|Joachim]]
* should we allow for multiple carving phases (runs/stages)?
:: I opt yes (separation of concern)
* should we allow for multiple carving algorithms?
:: I opt yes, this allows testing of different approaches
* Should the algorithm try to do as much in 1 run over the input data? To reduce IO?
:: I opt that the tool should allow for multiple and single run over the input data to minimize the IO or the CPU as bottleneck
* Interaction between algorithm and validators
** does the algorithm passes data blocks to the validators?
** does a validator need to maintain a state?
** does a validator need to revert a state?
** How do we deal with embedded files and content validation? Do the validators call another validator?
* do we use the assumption that a data block can be used by a single file (with the exception of embedded/encapsulated files)?
* Revit07 allows for multiple concurrent result files states to deal with fragmentation. One has the attribute of being active (the preferred) and the other passive. Do we want/need something similar? The algorithm adds block of input data (offsets) to these result files states.
** if so what info would these result files states require (type, list of input data blocks)
* how do we deal with file system remainders?
** Can we abstract them and compare them against available file system information?
* Do we carve file systems in files?
:: I opt that at least the validator uses this information

==Caving scenarios ==
[[User:Joachim Metz|Joachim]]
* normal file (file structure, loose text based structure (more a content structure?))
* fragmented file (the file entirely exist)
* a file fragment (the file does not entirely exist)
* intertwined file
* encapsulated file (MPEG/network capture)
* embedded file (JPEG thumbnail)
* obfuscation ('encrypted' PFF) this also entails encryption and/or compression
* file system in file

=File System Awareness =
==Background: Why be File System Aware?==
Advantages of being FS aware:
* You can pick up sector allocation sizes
:: [[User:Joachim Metz|Joachim]] do you mean file system block sizes?
* Some file systems may store things off sector boundaries. (ReiserFS with tail packing)
* Increasingly file systems have compression (NTFS compression)
* Carve just the sectors that are not in allocated files.

==Tasks that would be required==

==Discussion==
:: As noted above, TSK should be utilized as much as possible, particularly the filesystem-aware portion. If we want to identify filesystems outside of its supported set, it would be more worth our time to work on implementing them there than in the carver itself. [[User:RB|RB]]

:::: I guess this tool operates like [[Selective file dumper]] and can recover files in both ways (or not?). Recovering files by using carving can recover files in situations where sleuthkit does nothing (e.g. file on NTFS was deleted using ntfs-3g, or filesystem was destroyed or just unknown). And we should build the list of filesystems supported by carver, not by TSK. [[User:.FUF|.FUF]] 07:08, 29 October 2008 (UTC)

:: This tool is still in the early planning stages (requirements discovery), hence few operational details (like precise modes of operation) have been fleshed out - those will and should come later. The justification for strictly using TSK for the filesystem-sensitive approach is simple: TSK has good filesystem APIs, and it would be foolish to create yet another standalone, incompatible implementation of filesystem(foo) when time would be better spent improving those in TSK, aiding other methods of analysis as well. This is the same reason individuals that have implemented several other carvers are participating: de-duplication of effort. [[User:RB|RB]]

:: [[User:Joachim Metz|Joachim]] A design problem might be that TSK currently is a single library operating on multiple layers (storage media IO, volume/partition analysis and file system analysis). I'm not aware how easily the parts can be used separately. But I estimate that for the carver we want to use these 3 layers differently than TSK currently does.

[[User:Joachim Metz|Joachim]] I would like to have the carver (recovery tool) also do recovery using file allocation data or remainders of file allocation data.

[[User:Joachim Metz|Joachim]]
I would go as far to ask you all to look beyond the carver as a tool and look from the perspective of the carver as part of the forensic investigation process. In my eyes certain information needed/acquired by the carver could be also very useful investigative information i.e. what part of a hard disk contains empty sectors.

=Supportive tooling=
[[User:Joachim Metz|Joachim]]
* validator (definitions) tester (detest in revit07)
* tool to make configuration based definitions
* post carving validation
* the carver needs to provide support for fuse mount of carved files (carvfs)

=Testing =
[[User:Joachim Metz|Joachim]]
* automated testing
* test data

=Validator Construction=
Options:
* Write validators in C/C++
** [[User:Joachim Metz|Joachim]] you mean dedicated validators
* Have a scripting language for writing them (python? Perl?) our own?
** [[User:Joachim Metz|Joachim]] use easy to embed programming languages i.e. Phyton or Lua
* Use existing programs (libjpeg?) as plug-in validators?
** [[User:Joachim Metz|Joachim]] define a file structure api for this

=Existing Code that we have=
[[User:Joachim Metz|Joachim]]
Please add any missing links

Documentation/Articles
* DFRWS2006/2007 carving challenge results
* DFRWS2008 paper on carving

Carvers
* DFRWS2006/2007 carving challenge results
* photorec (http://www.cgsecurity.org/wiki/PhotoRec)
* revit06 and revit07 (http://sourceforge.net/projects/revit/)
* s3/scarve

Possible file structure validator libraries
* divers existing file support libraries
* libole2 (inhouse experimental code of OLE2 support)
* libpff (alpha release for PFF (PST/OST) file support) (http://sourceforge.net/projects/libpff/)

Input support
* AFF (http://www.afflib.org/)
* EWF (http://sourceforge.net/projects/libewf/)
* TSK device & raw & split raw (http://www.sleuthkit.org/)

Volume/Partition support
* disktype (http://disktype.sourceforge.net/)
* testdisk (http://www.cgsecurity.org/wiki/TestDisk)
* TSK

File system support
* TSK
* photorec FS code
* implementations of FS in Linux/BSD

Content support

Zero storage support
* libcarvpath ( http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=210704 )
* carvfs ( http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=210954 )
* tsk-cp ( http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=267227 )
* carvfsmodewf (http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=268256 )
POLA
* joe-e (java) ( http://code.google.com/p/joe-e/ )
* Emily (ocaml) ( http://erights.org/download/emily/ )
* the E language ( http://www.erights.org/ )
* AppArmor
* iptables/ipfw
* minorfs ( http://polacanthus.net/minorfs.html )
* plash ( http://plash.beasts.org/wiki/ )

=Implementation Timeline=
# gather the available resources/ideas/wishes/needs etc. (I guess we're in this phase)
# start discussing a high level design (in terms of algorithm, facilities, information needed)
## input formats facility
## partition/volume facility
## file system facility
## file format facility
## content facility
## how to deal with fragment detection (do the validators allow for fragment detection?)
## how to deal with recombination of fragments
## do we want multiple carving phases in light of speed/precision tradeoffs
# start detailing parts of the design
## Discuss options for a grammar driven validator?
## Hard-coded plug-ins?
## Which existing code can we use?
# start building/assembling parts of the tooling for a prototype
## Implement simple file carving with validation.
## Implement gap carving
# Initial Release
# Implement the ''threaded carving'' that [[User:.FUF|.FUF]] is describing above.

[[User:Joachim Metz|Joachim]] Shouldn't multi threaded carving (MTC) not be part of the 1st version?
The MT approach makes for different design decisions
: It is virtually impossible to turn a non-MT application into an MT application .[[User:Simsong|Simsong]] 06:37, 3 November 2008 (UTC)

Carver 2.0 Planning Page

2008-12-03T21:58:52Z

Capibara: /* Requirements */

This page is for planning Carver 2.0.

Please, do not delete text (ideas) here. Use something like this:

<pre>
<s>bad idea</s>
:: good idea
</pre>

This will look like:

<s>bad idea</s>
:: good idea

= License =

BSD-3.
:: [[User:Joachim Metz|Joachim]] library based validators could require other licenses
::: Make the other libraries plug-able. If you them, you use them. [[User:Simsong|Simsong]] 06:34, 3 November 2008 (UTC)

= OS =

Linux/FreeBSD/MacOS
: Shouldn't this just match what the underlying afflib & sleuthkit cover? [[User:RB|RB]]
:: Yes, but you need to test and validate on each. Question: Do we want to support windows? [[User:Simsong|Simsong]] 21:09, 30 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] I think we would do wise to design with windows support from the start this will improve the platform independence from the start
:::: Agreed; I would even settle at first for being able to run against Cygwin. Note that I don't even own or use a copy of Windows, but the vast majority of forensic investigators do. [[User:RB|RB]] 14:01, 31 October 2008 (UTC)
:: [[User:Capibara|Rob J Meijer]] Leaning heavily on the autotools might be the way to go. I do however feel that support requirements for windows would not be essential. Being able to run from a virtual machine with the main storage mounted over cifs should however be tested and if possible tuned extensively.
:::: [[User:Joachim Metz|Joachim]] You'll need more than autotools to do native Windows support i.e. file access, UTF-16 support, wrap some basic system functions or have them available otherwise
::::::[[User:Capibara|Rob J Meijer]] That´s exactly my point, windows support as in being able to build and run on windows natively is much more trouble than its worth. Better make for a lean and mean autotools based build with little dependencies and no or little recursion, and better spent effort on a lean POLA design on POSIX based systems than on supporting building and running on non POSIX systems.

= Name tooling =

* [[User:Joachim Metz|Joachim]] A name for the tooling I propose coldcut
:: How about 'butcher'? ;) [[User:RB|RB]] 14:20, 31 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] cleaver ( scalpel on steroids ;-) )
* I would like to propose Gouge or Chisel :-) [[User:Capibara|Rob J Meijer]]

= Requirements =

[[User:Joachim Metz|Joachim]] Could we do a MoSCoW evaluation of these.
* AFF and EWF file images supported from scratch. ([[User:Joachim Metz|Joachim]] I would like to have raw/split raw and device access as well)
:: If we base our image i/o on afflib, we get all three with one interface. [[User:RB|RB]] Instead of letting the tools use afflib, better to write an afflib module for carvfs, and update the libewf module. The tool could than be oblivious of the file format. [[User:Capibara|Rob J Meijer]]
:::: [[User:Simsong|Simsong]] 06:29, 3 November 2008 (UTC) The problem with using carvfs is that this adds another dependency. Do you really want to require that people install carvfs in order to run the carver? What about having the thing ported to Windows?
:::::: [[User:Capibara|Rob J Meijer]] I would support adding one build dependency (libcarvpath) and removing two (libewf/libaff) by moving them to a layer more suited for them (carvfs) that would possibly allow some form of file handle (as cap) based POLA design. I am a proponent of making small things that do one and do one thing right, and to stack those to do what you need. In my view that would lead ideally to the following (simplified) chain:
::::::* recursive [[computer forensics framework]] (ocfa/pyflag)
::::::** The-(pola-based)-carving-tool
::::::*** The-carving-lib working on open fd's.
::::::**** libcarvpath
::::::***** carvfs (Over cifs/nfs-v4 on platforms that don't support Fuse).
::::::****** libewf
::::::****** libaff
::::::*** AppArmor (on supporting platforms)
::::::*** suid (on supporting platforms)
::::::*** iptables/ipfw (on supporting platforms)
:::::: As fow windows support, I would imagine making carvfs run over smb would come a long way, that is for as far as windows support is all that relevant.
:::::: There are two advantages to using libcarvpath and carvfs instead of libaff/libewf t this layer:
::::::* storage requirements for doing carving. Beyond what sleuthkit or alternatives provide I have seen many situations where carving was not done due to storage limitations.
::::::* File handles are like object capabilities. You can often do pretty simple POLA based implementations using file handles and something like AppArmor. POLA could IMHO be a strong weapon against the more nasty forms of anti forensics.
::::::Next to this, I would consider making different tools for different stages instead of one semi recursive one, and looking at how to integrate these tools into existing frameworks (ocfa/pyflag).
::::::Keep things simple but rigid and try to easily integrate things into existing frameworks as effectively as possible I would suggest.
::::::Please note, I am not ptoposing the lib/tool should be useless without libcarvpath, only that usage without carvfs should limit the
::::::supported image formats to raw images, and that libewf/libaff should be abstracted at the Fuse level or below and not at the tool level.
:::::::[[User:Joachim Metz|Joachim]] do you have an idea what the performance impact of this approach would be? It might be wise to do a proof of concept for this approach first.
::::::::[[User:Capibara|Rob J Meijer]] It would I think depend greatly on behavior of the carving lib/tool. Small 512 byte reads are relatively very expensive, 128kb reads have negligible impact. Here are some numbers from ntfs-3g: http://article.gmane.org/gmane.comp.file-systems.fuse.devel/6397/match=ntfs+3g+performance+ext3 that might be relevant. More relevant than performance might be library footprint. For example, using OCFA, we often would want to keep e few hundred images that total to tens of TB of projected storage size fuse mounted. If libewf/libaff have a big combined memory footprint in such cases, this can be a major issue for this approach.
:::: [[User:Joachim Metz|Joachim]] this layer should support multi threaded decompression of compressed image types, this speeds up IO
* [[User:Joachim Metz|Joachim]] volume/partition aware layer (what about carving unpartioned space)
* File system aware layer. This could be or make use of tsk-cp.
** By default, files are not carved. (clarify: only identified? [[User:RB|RB]]; I guess that it operates like [[Selective file dumper]] [[User:.FUF|.FUF]] 07:00, 29 October 2008 (UTC)). Alternatively, the tool could use libcarvpath and output carvpaths or create a directory with symlinks to carvpaths that point into a carvfs mountpoint [[User:Capibara|Rob J Meijer]].
* Plug-in architecture for identification/validation.
** [[User:Joachim Metz|Joachim]] support for multiple types of validators
*** dedicated validator
*** validator based on file library (i.e. we could specify/implement a file structure API for these)
*** configuration based validator (Can handle config files,like Revit07, to enter different file formats used by the carver.)

[[User:Joachim Metz|Joachim]]
Moderator: Could we limit the requirements for prototype version 1 of the tool to get a working version up and running ASAP?
And keep discussing future options?

I think the following set will be large enough to handle:
Input facilities
* IO support (AFF, device, EWF, RAW and split RAW)
:: Abstraction of input format and multi threaded decompression (spin-off code out of afflib?)
* Volume/Partitions support
:: at least for DOS based layout and GPT (spin-off code out of TSK/Photorec?)
* File system support
:: VFAT/NTFS (spin-off code out of TSK/Photorec?)

Carving facilities
* File format support using plug-able validator model (use dedicated validators Photorec/Scarve and/or wrap revit07 file format as validator?)
* Content support using plug-able validator model (to handle text/mbox base64)
* File system carving support (to handle file system fragments, could be linked to file system support layer?)
* Basic fragment handling

Output facilities
* audit/analysis/debug log
* extraction of result files

==Supported File Formats==
* Ship with validators for:
[[User:Joachim Metz|Joachim]] I think we should distinguish between file format validators and content validators
** JPEG
** PNG
** GIF
** MSOLE
** ZIP
** TAR (gz/bz2)

[[User:Joachim Metz|Joachim]] For a production carver we need at least the following formats
** Grapical Images
*** JPEG (the 3 different types with JFIF/EXIF support)
*** PNG
*** GIF
*** BMP
*** TIFF
** Office documents
*** OLE2 (Word/Excell content support)
*** PDF
*** Open Office/Office 2007 (ZIP+XML)
:: Extension validation? AFAIK, MS Office 2007 [[DOCX]] format uses plain ZIP (or not?), and carved files will (or not?) have .zip extension instead of DOCX. Is there any way to fix this (may be using the file list in zip)? [[User:.FUF|.FUF]] 20:25, 31 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] Addition: Office 2007 also has a binary file format which is also a ZIP-ed data

==Archive Files==
** Archive files
*** ZIP
*** 7z
*** gzip
*** bzip2
*** tar
*** RAR
** E-mail files
*** PFF (PST/OST)
*** MBOX (text based format, base64 content support)
** Audio/Video files
*** MPEG
*** MP2/MP3
*** AVI
*** ASF/WMV
*** QuickTime
*** MKV
** Printer spool files
*** EMF (if I remember correctly)
** Internet history files
*** index.dat
*** firefox (sqllite 3)
** Other files
*** thumbs.db
*** pagefile?

==Carving Strategies==
[[User:Joachim Metz|Joachim]] Note to moderator could this section be merged with the carving algorithm section?

* Simple fragment recovery carving using gap carving.
** [[User:Joachim Metz|Joachim]] have hook in for more advanced fragment recovery?
* Recovering of individual ZIP sections and JPEG icons that are not sector aligned.
** [[User:Joachim Metz|Joachim]] I would propose a generic fragment detection and recovery
* Autonomous operation (some mode of operation should be completely non-interactive, requiring no human intervention to complete [[User:RB|RB]])
** [[User:Joachim Metz|Joachim]] as much as possible, but allow to be overwritten by user
* [[User:Joachim Metz|Joachim]] When the tool output files the filenames should contain the offset in the input data (in hexadecimal?)
:: [[User:Mark Stam|Mark]] I really like the fact carved files are named after the physical or logical sector in which the file is found (photorec)
:::: [[User:Joachim Metz|Joachim]] This naming schema might cause duplicate name problem for extracting embedded files and extracting files from non sector aligned file systems.
* [[User:Joachim Metz|Joachim]] Should the tool allow to export embedded files?
* [[User:Joachim Metz|Joachim]] Should the tool allow to export fragments separately?
* [[User:Mark Stam|Mark]] I personally use photorec often for carving files in the whole volume (not only unallocated clusters), so I can store information about all potential interesting files in MySQL
:: [[User:Joachim Metz|Joachim]] interesting, Bas Kloet and me have been discussing to use information about allocated files in the recovery process, i.e. recovered fragments could be part of allocated files. Do we want to be able to extract them? Or could we rebuild the file from the fragments and the allocated files.
* [[User:Mark Stam|Mark]] It would also be nice if the files can be hashed immediately (MD5) so looking for them in other tools (for example Encase) is a snap

==Performance Requirements==
* Tested on 500GB-sized images. Should be able to carve a 500GB image in roughly 50% longer than it takes to read the image.
** Perhaps allocate a percentage budget per-validator (i.e. each validator adds N% to the carving time) [[User:RB|RB]]
** [[User:Joachim Metz|Joachim]] have multiple carving phases for precision/speed trade off?
* Parallelizable
** [[User:Joachim Metz|Joachim]] tunable for different architectures
* Configuration:
** Capability to parse some existing carvers' configuration files, either on-the-fly or as a one-way converter.
** Disengage internal configuration structure from configuration files, create parsers that present the expected structure
** [[User:Joachim Metz|Joachim]] The validator should deal with the file structure the carving algorithm should not know anything about the file structure (as in revit07 design)
** Either extend Scalpel/Foremost syntaxes for extended features or use a tertiary syntax ([[User:Joachim Metz|Joachim]] I would prefer a derivative of the revit07 configuration syntax which already has encountered some problems of dealing with defining file structure in a configuration file)

==Output==
* Can output audit.txt file.
* [[User:Joachim Metz|Joachim]] Can output database with offset analysis values i.e. for visualization tooling
* [[User:Joachim Metz|Joachim]] Can output debug log for debugging the algorithm/validation
* Easy integration into ascription software.
:: [[User:Joachim Metz|Joachim]] I'm no native speaker what do you mean with "ascription software"?
::: I think this was another non-native requesting easy scriptability. [[User:RB|RB]] 14:20, 31 October 2008 (UTC)
:::: [[User:Joachim Metz|Joachim]] that makes sense ;-)
::::: Incorrect. Ascription software is software that determines who the owner of a file is. [[User:Simsong|Simsong]] 06:36, 3 November 2008 (UTC)

= Ideas =
* Use as much TSK if possible. Don't carry your own FS implementation the way photorec does.
:: [[User:Joachim Metz|Joachim]] using TSK as much as possible would not allow to add your own file system support (i.e. mobile phones, memory structures, cap files) I would propose wrapping TSK and using it as much as possible but allow to integrate own FS implementations.
* Extracting/carving data from [[Thumbs.db]]? I've used [[foremost]] for it with some success. [[Vinetto]] has some critical bugs :( [[User:.FUF|.FUF]] 19:18, 28 October 2008 (UTC)

==Recursive Carving==
[[User:Joachim Metz|Joachim]] do we want to support (let's call it) 'recursive in file carving' (for now) this is different from embedded files because there is a file system structure in the file and not just another file structure
* Is it just me, or do a lot of the above (and below) ideas somewhat skirt around the fact that many of us want recursive carving? Can we bend back to that instead of discussing object particulars? I think this can be distilled down to three requirements:
** Simple recursion: once an object is identified, have the ability to re-carve it for internal structures
** Directed recursion: the carver should be able to be directed at arbitrary blobs and told to carve it as a specified type. This allows programmatically more simple methods of dealing with unidentifiably compressed or encrypted data. Or filesystem fragments.
** Export: the ability to export an object (recognized or not) for later or external "recursion". Should go without saying for a carver, but...
:--[[User:RB|RB]] 18:45, 2 November 2008 (UTC)
:: [[User:Simsong|Simsong]] 06:30, 3 November 2008 (UTC) pyflag already does recursive carving. Are we just going to reimplement pyflag as a single executable?

==Library Dependencies==
[[User:Capibara|Rob J Meijer]] :
* Use libcarvpath whenever possible and by default to avoid high storage requirements.
:: [[User:Joachim Metz|Joachim]] For easy deployment I would not opt for making an integral part of the tool solely dependant on a single external library or the library must be integrated in the package
::[[User:Capibara|Rob J Meijer]] Integrating libraries (libtsk,libaff.libewf,libcarvpath etc) is bad practice, autotools are your friend IMO.
:: [[User:Joachim Metz|Joachim]] I'm not talking about integrating (shared) libraries. I'm talking about that an integral part of a tool should be part of it's package. Why can't the tool package contain shared or static libraries for local use? A far worse thing to do is to have a large set of dependencies and making the tool difficult to install for most users. The tool package should contain the most necessary code. afflib/libewf support could be detected by the autotools a neat separation of functionality.
::: From a packager's standpoint, [[User:Joachim Metz|Joachim]]'s other libraries do a really good job of this, carrying around what they need but using a system-global version if available. [[User:RB|RB]]
* libtsk
* libaff ? : possibly the discussion in the requirements section should move to his section.
* libewf ? : possibly the discussion in the requirements section should move to his section.
* posix ? : Can we depend especially on the availability of UNIX domain sockets and the possibility to use msg_accrights for passing opn file handles as ocaps?

==Filesystem Detection==
* Dont stop with filesystem detection after the first match. Often if a partition is reused with a new FS and is not all that full yet, much of the old FS can still be valid. I have seen this with ext2/fat. The fact that you have identified a valid FS on a partition doesn't mean there isn't an(almost) valid second FS that would yield additional files. Identifying doubly allocated space might in some cases also be relevant.
:: [[User:Joachim Metz|Joachim]] What your saying is that dealing with file system fragments should be part of the carving algorithm
* Allow use where filesystem based carving is done by other tool, and the tool is used as second stage on (sets of) unallocated block (pseudo) files and/or non FS partition (pseudo) files.
:: [[User:Joachim Metz|Joachim]] I would not opt for this. The tool would be dependent on other tools and their data format, which makes the tool difficult to maintain. I would opt to integrate the functionality of having multiple recovery phases (stages) and allow the tooling to run the phases after one and other or separately.
::[[User:Capibara|Rob J Meijer]] More generically, I feel a way should exist to communicate the 'left overs' a previous (non open, for example LE-only) tool left.
:: [[User:Joachim Metz|Joachim]] I guess if the tool is designed to handle multiple phases it should store its data somewhere. So it should be possible to convert results of such non open tooling to the format required. However I would opt to design the recovery functionality of these non-open tools into open tools. And not to limit ourselves making translators due to the design of these non-open tools.
* Ability to be used as a library instead of a tool. Ability to access metadata true library, and thus the ability to set metadata from the carving modules. This would be extremely usefull for integrating the project into a framework like ocfa.
:: [[User:Joachim Metz|Joachim]] I guess most of the code could be integrated into libraries, but I would not opt integrating tool functionality into a library
* [[User:Mark Stam|Mark]] I think it would be very handy to have a CSV, TSV, XML or other delimited output (log)file with information about carved files. This output file can then be stored in a database or Excel sheet (report function)

==Anti forensics and system integrity concerns==
* It might be very interesting to look at the possibilities of using a multi process style of module support and combine it with a least authority design. On platforms that support AppArmor (or similar) and uid based firewall rules, this could make for the first true POLA (principle of least authority) based forensic tool ever. POLA based forensics tools should make for a strong integrity guard against many anti forensics. Alternatively we could look at integrating a capability secure language (E?) for implementation of at least validation modules. I don't expect this idea to make it, but mentioning it I hope might spark off less strong alternatives that at least partially address the integrity + anti-forensics problem. If we can in some way introduce POLA to a wider forensics public, other tools might also pick up on it what would be great.
:: [[User:Joachim Metz|Joachim]] Could you give an example of how you see this in action?
::::[[User:Capibara|Rob J Meijer]] I see two layers where using POLA could be applied. The best one would require one of the folowing as prerequisites:
::::* The libaff/libewf layer is moved to a fuse implementation (for example carvfs).
::::* Libewf/Libaff are updated to accept opened filhandles instead of demanding to open their own files.
::::If one of these is fulfilled, than the tool running as some user can just have the simple task of opening the image files, starting up the 'real' tool and handing over the appropriate file handles. If the real tool runs with a restrictive AppArmor profile, and is started suid to a tool specific user that also has its own iptables uid based filter, than the real tool will run with least authority.
:::: A second alternative, if neither of the first prerequisite could not be bet, would be to run the modules as confined processes and have a non confined process run as proxy for the first.
:::: A third probably far fetched alternative would be to embed an object capability language in the tool and make the module interface thus that modules are to be written in this ocap language.
::::A 4th alternative might include minorfs or plash, but I havn't geven those sufficient thinking hours yet.

== Format syntax specification ==
* Carving data structures. For example, extract all TCP headers from image by defining TCP header structure and some fields (e.g. source port > 1024, dest port = 80). This will extract all data matching the pattern and write a file with other fields. Another example is carving INFO2 structures and URL activity records from index.dat [[User:.FUF|.FUF]] 20:51, 28 October 2008 (UTC)
** This has the opportunity to be extended to the concept of "point at blob FOO and interpret it as BAR"
.FUF added:
The main idea is to allow users to define structures, for example (in pascal-like form):

<pre>
Field1: Byte = 123;
SomeTextLength: DWORD;
SomeText: string[SomeTextLength];
Field4: Char = 'r';
...
</pre>

This will produce something like this:
<pre>
Field1 = 123
SomeTextLength = 5
SomeText = 'abcd1'
Field4 = 'r'
</pre>

(In text or raw forms.)

Opinions?

Opinion: Simple pattern identification like that may not suffice, I think Simson's original intent was not only to identify but to allow for validation routines (plugins, as the original wording was). As such, the format syntax would need to implement a large chunk of some programming language in order to be sufficiently flexible. [[User:RB|RB]]

[[User:Joachim Metz|Joachim]]
In my option your example is too limited. Making the revit configuration I learned you'll need a near programming language to specify some file formats.
A simple descriptive language is too limiting. I would also go for 2 bytes with endianess instead of using terminology like WORD and small integer, it's much more clear. The configuration also needs to deal with aspects like cardinality, required and optional structures.
:: This is simply data structures carving, see ideas above. Somebody (I cannot track so many changes per day) separated the original text. There is no need to count and join different structures. [[User:.FUF|.FUF]] 19:53, 31 October 2008 (UTC)
:::: [[User:Joachim Metz|Joachim]] This was probably me is the text back in it's original form?
:::: I started it by moving your Revit07 comment to the validator/plugin section in [http://www.forensicswiki.org/index.php?title=Carver_2.0_Planning_Page&diff=prev&oldid=7583 this edit], since I was still at that point thinking operational configuration for that section, not parser configurations. [[User:RB|RB]]
:::: [[User:Joachim Metz|Joachim]] I renamed the title to format syntax, clarity is important ;-)

Please take a look at the revit07 format syntax specification (configuration). It's not there yet but goes a far way. Some things currently missing:
* bitwise alignment
* handling encapsulated streams (MPEG/capture files)
* handling content based formats (MBOX)

=Caving algorithm =
[[User:Joachim Metz|Joachim]]
* should we allow for multiple carving phases (runs/stages)?
:: I opt yes (separation of concern)
* should we allow for multiple carving algorithms?
:: I opt yes, this allows testing of different approaches
* Should the algorithm try to do as much in 1 run over the input data? To reduce IO?
:: I opt that the tool should allow for multiple and single run over the input data to minimize the IO or the CPU as bottleneck
* Interaction between algorithm and validators
** does the algorithm passes data blocks to the validators?
** does a validator need to maintain a state?
** does a validator need to revert a state?
** How do we deal with embedded files and content validation? Do the validators call another validator?
* do we use the assumption that a data block can be used by a single file (with the exception of embedded/encapsulated files)?
* Revit07 allows for multiple concurrent result files states to deal with fragmentation. One has the attribute of being active (the preferred) and the other passive. Do we want/need something similar? The algorithm adds block of input data (offsets) to these result files states.
** if so what info would these result files states require (type, list of input data blocks)
* how do we deal with file system remainders?
** Can we abstract them and compare them against available file system information?
* Do we carve file systems in files?
:: I opt that at least the validator uses this information

==Caving scenarios ==
[[User:Joachim Metz|Joachim]]
* normal file (file structure, loose text based structure (more a content structure?))
* fragmented file (the file entirely exist)
* a file fragment (the file does not entirely exist)
* intertwined file
* encapsulated file (MPEG/network capture)
* embedded file (JPEG thumbnail)
* obfuscation ('encrypted' PFF) this also entails encryption and/or compression
* file system in file

=File System Awareness =
==Background: Why be File System Aware?==
Advantages of being FS aware:
* You can pick up sector allocation sizes
:: [[User:Joachim Metz|Joachim]] do you mean file system block sizes?
* Some file systems may store things off sector boundaries. (ReiserFS with tail packing)
* Increasingly file systems have compression (NTFS compression)
* Carve just the sectors that are not in allocated files.

==Tasks that would be required==

==Discussion==
:: As noted above, TSK should be utilized as much as possible, particularly the filesystem-aware portion. If we want to identify filesystems outside of its supported set, it would be more worth our time to work on implementing them there than in the carver itself. [[User:RB|RB]]

:::: I guess this tool operates like [[Selective file dumper]] and can recover files in both ways (or not?). Recovering files by using carving can recover files in situations where sleuthkit does nothing (e.g. file on NTFS was deleted using ntfs-3g, or filesystem was destroyed or just unknown). And we should build the list of filesystems supported by carver, not by TSK. [[User:.FUF|.FUF]] 07:08, 29 October 2008 (UTC)

:: This tool is still in the early planning stages (requirements discovery), hence few operational details (like precise modes of operation) have been fleshed out - those will and should come later. The justification for strictly using TSK for the filesystem-sensitive approach is simple: TSK has good filesystem APIs, and it would be foolish to create yet another standalone, incompatible implementation of filesystem(foo) when time would be better spent improving those in TSK, aiding other methods of analysis as well. This is the same reason individuals that have implemented several other carvers are participating: de-duplication of effort. [[User:RB|RB]]

:: [[User:Joachim Metz|Joachim]] A design problem might be that TSK currently is a single library operating on multiple layers (storage media IO, volume/partition analysis and file system analysis). I'm not aware how easily the parts can be used separately. But I estimate that for the carver we want to use these 3 layers differently than TSK currently does.

[[User:Joachim Metz|Joachim]] I would like to have the carver (recovery tool) also do recovery using file allocation data or remainders of file allocation data.

[[User:Joachim Metz|Joachim]]
I would go as far to ask you all to look beyond the carver as a tool and look from the perspective of the carver as part of the forensic investigation process. In my eyes certain information needed/acquired by the carver could be also very useful investigative information i.e. what part of a hard disk contains empty sectors.

=Supportive tooling=
[[User:Joachim Metz|Joachim]]
* validator (definitions) tester (detest in revit07)
* tool to make configuration based definitions
* post carving validation
* the carver needs to provide support for fuse mount of carved files (carvfs)

=Testing =
[[User:Joachim Metz|Joachim]]
* automated testing
* test data

=Validator Construction=
Options:
* Write validators in C/C++
** [[User:Joachim Metz|Joachim]] you mean dedicated validators
* Have a scripting language for writing them (python? Perl?) our own?
** [[User:Joachim Metz|Joachim]] use easy to embed programming languages i.e. Phyton or Lua
* Use existing programs (libjpeg?) as plug-in validators?
** [[User:Joachim Metz|Joachim]] define a file structure api for this

=Existing Code that we have=
[[User:Joachim Metz|Joachim]]
Please add any missing links

Documentation/Articles
* DFRWS2006/2007 carving challenge results
* DFRWS2008 paper on carving

Carvers
* DFRWS2006/2007 carving challenge results
* photorec (http://www.cgsecurity.org/wiki/PhotoRec)
* revit06 and revit07 (http://sourceforge.net/projects/revit/)
* s3/scarve

Possible file structure validator libraries
* divers existing file support libraries
* libole2 (inhouse experimental code of OLE2 support)
* libpff (alpha release for PFF (PST/OST) file support) (http://sourceforge.net/projects/libpff/)

Input support
* AFF (http://www.afflib.org/)
* EWF (http://sourceforge.net/projects/libewf/)
* TSK device & raw & split raw (http://www.sleuthkit.org/)

Volume/Partition support
* disktype (http://disktype.sourceforge.net/)
* testdisk (http://www.cgsecurity.org/wiki/TestDisk)
* TSK

File system support
* TSK
* photorec FS code
* implementations of FS in Linux/BSD

Content support

Zero storage support
* libcarvpath ( http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=210704 )
* carvfs ( http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=210954 )
* tsk-cp ( http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=267227 )
* carvfsmodewf (http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=268256 )
POLA
* joe-e (java) ( http://code.google.com/p/joe-e/ )
* Emily (ocaml) ( http://erights.org/download/emily/ )
* the E language ( http://www.erights.org/ )
* AppArmor
* iptables/ipfw
* minorfs ( http://polacanthus.net/minorfs.html )
* plash ( http://plash.beasts.org/wiki/ )

=Implementation Timeline=
# gather the available resources/ideas/wishes/needs etc. (I guess we're in this phase)
# start discussing a high level design (in terms of algorithm, facilities, information needed)
## input formats facility
## partition/volume facility
## file system facility
## file format facility
## content facility
## how to deal with fragment detection (do the validators allow for fragment detection?)
## how to deal with recombination of fragments
## do we want multiple carving phases in light of speed/precision tradeoffs
# start detailing parts of the design
## Discuss options for a grammar driven validator?
## Hard-coded plug-ins?
## Which existing code can we use?
# start building/assembling parts of the tooling for a prototype
## Implement simple file carving with validation.
## Implement gap carving
# Initial Release
# Implement the ''threaded carving'' that [[User:.FUF|.FUF]] is describing above.

[[User:Joachim Metz|Joachim]] Shouldn't multi threaded carving (MTC) not be part of the 1st version?
The MT approach makes for different design decisions
: It is virtually impossible to turn a non-MT application into an MT application .[[User:Simsong|Simsong]] 06:37, 3 November 2008 (UTC)

Computer forensics framework

2008-12-03T21:56:39Z

Capibara:

The following more or less generic computer forensics frameworks exist:

* [[Open Computer Forensics Architecture]]
* [[Pyflag]]

PyFlag

2008-12-03T21:49:40Z

Capibara:

{{Infobox_Software |
name = PyFlag |
maintainer = [[Michael Cohen]], [[David Collett]] |
os = {{Linux}}, {{Web-based}} |
genre = {{Analysis}} |
license = {{GPL}} |
website = [http://www.pyflag.net/ pyflag.net] |
}}

'''PyFlag''' is a web-based, database-backed ''forensic and log analysis GUI'' and [[Computer forensics framework]] written in [[Python]]. PyFlag stores disk images in the [[sgzip]] format.

=Features=

==File Systems Understood==

==File Search Facilities==

* Lists allocated and unallocated files.
* Sorts files by type.
* Searches for keywords.
* Works with compressed zip files.

==Historical Reconstruction==

Can it build timelines and search by creation date?
* Creates a "case file".

==Searching Abilities==

* Searches for keywords.
* Builds an index.

==Hash Databases==

* Hashes and compares with [[Hashkeeper]] using [[MD5]].

==Evidence Collection Features==

=History=

* Originally started by the [[Australian Department of Defence]], PyFlag is now hosted on [[SourceForge]].

==License Notes==

= External Links =
http://sourceforge.net/projects/pyflag/

==External Reviews==

Computer forensics framework

2008-12-03T21:47:12Z

Capibara: New page: * Open Computer Forensics Architecture * pyflag

* [[Open Computer Forensics Architecture]]
* [[pyflag]]

Open Computer Forensics Architecture

2008-12-03T21:45:28Z

Capibara:

The '''Open Computer Forensics Architecture''' ('''OCFA''') is a modular [[computer forensics framework]] built by the [[Dutch National Police Agency]]. The main goal is to automate the digital forensic process to speed up the investigation and give tactical [[investigator]]s direct access to the seized data through an easy to use search and browse interface.

The architecture forms an environment where existing forensic [[tools]] and libraries can be easily plugged into the architecture and can thus be made part of the recursive extraction of data and [[metadata]] from digital evidence.

The Open Computer Forensics Architecture aims to be highly modular, robust, fault tolerant, recursive and scalable in order to be usable in large investigations that spawn numerous terabytes of evidence data and covers hundreds of evidence items.

Modules in OCFA for reasons of fault tolerance are processes. The [[OcfaLib API]] makes it possible and relatively easy to build an OCFA module out of any data processing library or tool. OCFA comes with numerous such modules that are mostly wrappers around libraries like [[libmagic]] or tools such as those found in the [[Sleuthkit]].

Communication between modules within OCFA is governed by a two layered communication infrastructure as provided by OCFA. At the lowest layer is a messaging system with at is center the OCFA Anycast Relay. The Anycast Relay provides the facilities of module crash resistance, distributed processing load balancing and flow control.
At a higher level of communication, the OCFA XML Router provides for the routing of individual pieces of evidence through the most appropriate tool chain for its particular type of content.

Although OCFA contains a rudimentary user interface, most of its power is in the backend architecture.
The last and final module in the tool chain of any evidence will be the OCFA Data Store Module. This module
processes the evidence XML (that contains all of the evidence data its meta data) and stores relevant parts into a postgesql database. Extending the apache based user interface with interfaces for your own case bound queries
is something that should proof very useful in most investigations.

For more information consult [http://sourceforge.net/projects/ocfa/ sourceforge.net/projects/ocfa/ ] .

OcfaLib API

2008-12-03T20:34:27Z

Capibara:

The OcfaLib API is a C++ API that is meant for usage by modules in the [[Open Computer Forensics Architecture]].
The C++ class ocfa::module::OcfaModule defines the base class of what all OCFA modules are derived. Depending on the type of module, the ocfa::facade namespace provides different convenient subclasses.
The OcfaModule class defines a virtual function named ProcessEvidence that the final module class must implement.
A typical module implementation implements a subclass by implementing a constructor and a ProcessEvidence method.
The main function of the module program will create an instance of the module class and call the run method on the object. The module will connect and register itself to the OCFA Anycast Relay and thus be connected into the Open Computer Forensics Architecture. For each piece of evidence data the module receives, the ProcessEvidence method will get invoked. The implementation of the ProcessEvidence method can depending on the type of facade used as baseclass can:

* Gain read access to the input evidence data.
* Use its own private workdir for derived and temporary data
* Derive evidence from the input evidence.
* Access meta data created by other modules.
* Add additional metadata to the input evidence or child evidences.

For a simple example of a module that derives data from data look at the tar module in OcfaModules/dissector/tar.
For a simple example of a module that extracts metadata from data look at the pgp module in OcfaModule/extractor/pgp.

Next to simple OCFA modules, the ocfa::treegraph namespace provides an aditional sub API for building treegraph loadable modules, that can be used to create modules that map evidence data to a tree graph of data and meta data.

The only API available currently is the C++ API.
Work is currently being done to also create a Java API and plans exist to build a Perl API to OCFA in the future.

OcfaLib API

2008-12-03T20:22:44Z

Capibara: New page: The OcfaLib API is a C++ API that is meant for usage by modules in the Open Computer Forensics Architecture. The C++ class ocfa::module::OcfaModule defines the base class of what all O...

The OcfaLib API is a C++ API that is meant for usage by modules in the [[Open Computer Forensics Architecture]].
The C++ class ocfa::module::OcfaModule defines the base class of what all OCFA modules are derived. Depending on the type of module, the ocfa::facade namespace provides different convenient subclasses.
The OcfaModule class defines a virtual function named ProcessEvidence that the final module class must implement.
A typical module implementation implements a subclass by implementing a constructor and a ProcessEvidence method.
The main function of the module program will create an instance of the module class and call the run method on the object. The module will connect and register itself to the OCFA Anycast Relay and thus be connected into the Open Computer Forensics Architecture. For each piece of evidence data the module receives, the ProcessEvidence method will get invoked. The implementation of the ProcessEvidence method can depending on the type of facade used as baseclass can:

* Gain read access to the input evidence data.
* Use its own private workdir for derived and temporary data
* Derive evidence from the input evidence.
* Access meta data created by other modules.
* Add additional metadata to the input evidence or child evidences.

The only API available currently is the C++ API.
Work is currently being done to also create a Java API and plans exist to build a Perl API to OCFA in the future.

Open Computer Forensics Architecture

2008-12-03T19:03:10Z

Capibara:

The '''Open Computer Forensics Architecture''' ('''OCFA''') is a modular computer forensics framework built by the [[Dutch National Police Agency]]. The main goal is to automate the digital forensic process to speed up the investigation and give tactical [[investigator]]s direct access to the seized data through an easy to use search and browse interface.

The architecture forms an environment where existing forensic [[tools]] and libraries can be easily plugged into the architecture and can thus be made part of the recursive extraction of data and [[metadata]] from digital evidence.

The Open Computer Forensics Architecture aims to be highly modular, robust, fault tolerant, recursive and scalable in order to be usable in large investigations that spawn numerous terabytes of evidence data and covers hundreds of evidence items.

Modules in OCFA for reasons of fault tolerance are processes. The [[OcfaLib API]] makes it possible and relatively easy to build an OCFA module out of any data processing library or tool. OCFA comes with numerous such modules that are mostly wrappers around libraries like [[libmagic]] or tools such as those found in the [[Sleuthkit]].

Communication between modules within OCFA is governed by a two layered communication infrastructure as provided by OCFA. At the lowest layer is a messaging system with at is center the OCFA Anycast Relay. The Anycast Relay provides the facilities of module crash resistance, distributed processing load balancing and flow control.
At a higher level of communication, the OCFA XML Router provides for the routing of individual pieces of evidence through the most appropriate tool chain for its particular type of content.

Although OCFA contains a rudimentary user interface, most of its power is in the backend architecture.
The last and final module in the tool chain of any evidence will be the OCFA Data Store Module. This module
processes the evidence XML (that contains all of the evidence data its meta data) and stores relevant parts into a postgesql database. Extending the apache based user interface with interfaces for your own case bound queries
is something that should proof very useful in most investigations.

For more information consult [http://sourceforge.net/projects/ocfa/ sourceforge.net/projects/ocfa/ ] .

Talk:Carver 2.0 Planning Page

2008-11-24T21:40:01Z

Capibara: /* POLA */

License: have we even discussed a license yet? Who chose it? I'm not terribly opposed to a 3-clause BSD, but...? - [[User:RB|RB]] 00:39, 30 October 2008 (UTC)

[[User:Joachim Metz|Joachim]] I prefer the LPGL it's restricts the usage of the code somewhat more. When its integrated in other (closed source) tooling which is published, they must publish that the tool uses this code.

:: LGPL?
:: [[User:.FUF|.FUF]] 19:40, 31 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] GNU Library or "Lesser" General Public License (LGPL) (http://www.opensource.org/licenses/alphabetical)
:::: ''Joachim I prefer the LPGL'' :) [[User:.FUF|.FUF]] 19:51, 31 October 2008 (UTC)
:::::: [[User:Joachim Metz|Joachim]] To quote Homer Simpson "Doh!"
:: Agreed. I sit on the fence between BSD and GPL: the business half of me agrees that open licensing should place as few restrictions or qualifications as possible, whereas the idealist/OSS side wants to ensure the project's freedom. The LGPL is a more reasonable balance, encouraging widespread use but ensuring modifications' freedom. [[User:RB|RB]] 16:59, 1 November 2008 (UTC)

== Consolidation ==

We've got a '''lot''' of good ideas here, but in interest of not stepping on anyone's toes, it's getting rather disjointed and hard to read. Is anyone willing to (or allow me to) try to consolidate them into some sort of coherency? I'd like at least one of the admins ([[User:.FUF|.FUF]] or [[User:Simsong|Simsong]] to concur before anyone moves forward. I know the wiki way is to just let it grow, but even watching each addition I'm starting to have trouble visualizing where we are. [[User:RB|RB]]

:: That's good idea, but it is important to consolidate without losing the ideas and opinions. I think it's better for this page to enter some "stable" branch. And then we'll move to the next phase. What do you think, [[User:Simsong|Simsong]]? [[User:.FUF|.FUF]] 18:12, 1 November 2008 (UTC)

::: One option could be to break each major section into its own page so it can be properly discussed without clutter, then transclude each to this page. A dedicated namespace would probably be overkill, but since we're throwing out ideas should at least be mentioned. --[[User:RB|RB]] 19:03, 1 November 2008 (UTC)

:::: [[User:Joachim Metz|Joachim]] Separate the parts into topics. Have a discussion and an informational part per topic.
== POLA ==

[[User:Capibara|Capibara]] :Just noticed a presentation by Simson when looking for 'usable security' on google that included notes on designation and access (authority). From this I would gather that Simson has something of a background in [http://en.wikipedia.org/wiki/Capability-based_security capability security theory]. This makes me feel we might have sufficient ground to discuss 'open file handles as capabilities' at the library level. [[AppArmor]] allows instances of executables to be confined to least privilege. Unix domain sockets allow open file handles to be passed around as capabilities. The combination of these two results in the possibility of using [http://en.wikipedia.org/wiki/Object_capability object capability] based least authority at the process level of granularity. For such to work with libraries such as libaff, libewf and libtsk, these libraries must not insist on opening their own files. At the moment this rules out using libaff or libewf directly as such. [http://ocfa.sourceforge.net/libcarvpath/ CarvFs] can provide a way around this, but it may be desirable to not have to use CarvFs in some cases. Given that the authors of both libaff and libewf are quite active here, It may be valid to discuss the possibility of extending the API of these libs so that the libs dont demand to open their own files, but could also be used by a confined carving tool that would get open file handles passed to it over a unix domain socket.

Talk:Carver 2.0 Planning Page

2008-11-24T21:36:28Z

Capibara: /* POLA */

License: have we even discussed a license yet? Who chose it? I'm not terribly opposed to a 3-clause BSD, but...? - [[User:RB|RB]] 00:39, 30 October 2008 (UTC)

[[User:Joachim Metz|Joachim]] I prefer the LPGL it's restricts the usage of the code somewhat more. When its integrated in other (closed source) tooling which is published, they must publish that the tool uses this code.

:: LGPL?
:: [[User:.FUF|.FUF]] 19:40, 31 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] GNU Library or "Lesser" General Public License (LGPL) (http://www.opensource.org/licenses/alphabetical)
:::: ''Joachim I prefer the LPGL'' :) [[User:.FUF|.FUF]] 19:51, 31 October 2008 (UTC)
:::::: [[User:Joachim Metz|Joachim]] To quote Homer Simpson "Doh!"
:: Agreed. I sit on the fence between BSD and GPL: the business half of me agrees that open licensing should place as few restrictions or qualifications as possible, whereas the idealist/OSS side wants to ensure the project's freedom. The LGPL is a more reasonable balance, encouraging widespread use but ensuring modifications' freedom. [[User:RB|RB]] 16:59, 1 November 2008 (UTC)

== Consolidation ==

We've got a '''lot''' of good ideas here, but in interest of not stepping on anyone's toes, it's getting rather disjointed and hard to read. Is anyone willing to (or allow me to) try to consolidate them into some sort of coherency? I'd like at least one of the admins ([[User:.FUF|.FUF]] or [[User:Simsong|Simsong]] to concur before anyone moves forward. I know the wiki way is to just let it grow, but even watching each addition I'm starting to have trouble visualizing where we are. [[User:RB|RB]]

:: That's good idea, but it is important to consolidate without losing the ideas and opinions. I think it's better for this page to enter some "stable" branch. And then we'll move to the next phase. What do you think, [[User:Simsong|Simsong]]? [[User:.FUF|.FUF]] 18:12, 1 November 2008 (UTC)

::: One option could be to break each major section into its own page so it can be properly discussed without clutter, then transclude each to this page. A dedicated namespace would probably be overkill, but since we're throwing out ideas should at least be mentioned. --[[User:RB|RB]] 19:03, 1 November 2008 (UTC)

:::: [[User:Joachim Metz|Joachim]] Separate the parts into topics. Have a discussion and an informational part per topic.
== POLA ==

[[User:Capibara|Capibara]] :Just noticed a presentation by Simson when looking for 'usable security' on google that included notes on designation and access (authority). From this I would gather that Simson has something of a background in [[capability security theory]]. This makes me feel we might have sufficient ground to discuss 'open file handles as capabilities' at the library level. [[AppArmor]] allows instances of executables to be confined to least privilege. Unix domain sockets allow open file handles to be passed around as capabilities. The combination of these two results in the possibility of using [[object capability]] based least authority at the process level of granularity. For such to work with libraries such as libaff, libewf and libtsk, these libraries must not insist on opening their own files. At the moment this rules out using libaff or libewf directly as such. [http://ocfa.sourceforge.net/libcarvpath/ CarvFs] can provide a way around this, but it may be desirable to not have to use CarvFs in some cases. Given that the authors of both libaff and libewf are quite active here, It may be valid to discuss the possibility of extending the API of these libs so that the libs dont demand to open their own files, but could also be used by a confined carving tool that would get open file handles passed to it over a unix domain socket.

AppArmor

2008-11-24T21:34:07Z

Capibara: New page: AppArmor, found in Suse and Ubuntu Linux, is a system for letting applications run according to the principle of least privilege. This can be a useful for running forensic or other...

AppArmor, found in [[Suse]] and [[Ubuntu]] Linux, is a system for letting applications run according to the principle of least privilege.
This can be a useful for running forensic or other tools of questionable stability (what according to many means almost all tools) on data
that may constructed with malicious intent against the specific tool, and mitigating the potential damage to the forensic process.
While AppArmor works with static profiles, it does not intervene in communication of open files over unix sockets, that thus can be used
to dynamically communicate authority to an (untrusted) AppArmor confined tool.

Talk:Carver 2.0 Planning Page

2008-11-24T21:21:40Z

Capibara: /* POLA */

License: have we even discussed a license yet? Who chose it? I'm not terribly opposed to a 3-clause BSD, but...? - [[User:RB|RB]] 00:39, 30 October 2008 (UTC)

[[User:Joachim Metz|Joachim]] I prefer the LPGL it's restricts the usage of the code somewhat more. When its integrated in other (closed source) tooling which is published, they must publish that the tool uses this code.

:: LGPL?
:: [[User:.FUF|.FUF]] 19:40, 31 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] GNU Library or "Lesser" General Public License (LGPL) (http://www.opensource.org/licenses/alphabetical)
:::: ''Joachim I prefer the LPGL'' :) [[User:.FUF|.FUF]] 19:51, 31 October 2008 (UTC)
:::::: [[User:Joachim Metz|Joachim]] To quote Homer Simpson "Doh!"
:: Agreed. I sit on the fence between BSD and GPL: the business half of me agrees that open licensing should place as few restrictions or qualifications as possible, whereas the idealist/OSS side wants to ensure the project's freedom. The LGPL is a more reasonable balance, encouraging widespread use but ensuring modifications' freedom. [[User:RB|RB]] 16:59, 1 November 2008 (UTC)

== Consolidation ==

We've got a '''lot''' of good ideas here, but in interest of not stepping on anyone's toes, it's getting rather disjointed and hard to read. Is anyone willing to (or allow me to) try to consolidate them into some sort of coherency? I'd like at least one of the admins ([[User:.FUF|.FUF]] or [[User:Simsong|Simsong]] to concur before anyone moves forward. I know the wiki way is to just let it grow, but even watching each addition I'm starting to have trouble visualizing where we are. [[User:RB|RB]]

:: That's good idea, but it is important to consolidate without losing the ideas and opinions. I think it's better for this page to enter some "stable" branch. And then we'll move to the next phase. What do you think, [[User:Simsong|Simsong]]? [[User:.FUF|.FUF]] 18:12, 1 November 2008 (UTC)

::: One option could be to break each major section into its own page so it can be properly discussed without clutter, then transclude each to this page. A dedicated namespace would probably be overkill, but since we're throwing out ideas should at least be mentioned. --[[User:RB|RB]] 19:03, 1 November 2008 (UTC)

:::: [[User:Joachim Metz|Joachim]] Separate the parts into topics. Have a discussion and an informational part per topic.
== POLA ==

[[User:Capibara|Capibara]] :Just noticed a presentation by Simson when looking for 'usable security' on google that included notes on designation and access (authority). From this I would gather that Simson has something of a background in [[capability security theory]]. This makes me feel we might have sufficient ground to discuss 'open file handles as capabilities' at the library level. [[AppArmor]] allows instances of executables to be confined to least privilege. Unix domain sockets allow open file handles to be passed around as capabilities. The combination of these two results in the possibility of using [[object capability]] based least authority at the process level of granularity. For such to work with libraries such as libaff, libewf and libtsk, these libraries must not insist on opening their own files. At the moment this rules out using libaff or libewf directly as such. [[CarvFs]] can provide a way around this, but it may be desirable to not have to use CarvFs in some cases. Given that the authors of both libaff and libewf are quite active here, It may be valid to discuss the possibility of extending the API of these libs so that the libs dont demand to open their own files, but could also be used by a confined carving tool that would get open file handles passed to it over a unix domain socket.

Talk:Carver 2.0 Planning Page

2008-11-24T08:48:40Z

Capibara:

License: have we even discussed a license yet? Who chose it? I'm not terribly opposed to a 3-clause BSD, but...? - [[User:RB|RB]] 00:39, 30 October 2008 (UTC)

[[User:Joachim Metz|Joachim]] I prefer the LPGL it's restricts the usage of the code somewhat more. When its integrated in other (closed source) tooling which is published, they must publish that the tool uses this code.

:: LGPL?
:: [[User:.FUF|.FUF]] 19:40, 31 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] GNU Library or "Lesser" General Public License (LGPL) (http://www.opensource.org/licenses/alphabetical)
:::: ''Joachim I prefer the LPGL'' :) [[User:.FUF|.FUF]] 19:51, 31 October 2008 (UTC)
:::::: [[User:Joachim Metz|Joachim]] To quote Homer Simpson "Doh!"
:: Agreed. I sit on the fence between BSD and GPL: the business half of me agrees that open licensing should place as few restrictions or qualifications as possible, whereas the idealist/OSS side wants to ensure the project's freedom. The LGPL is a more reasonable balance, encouraging widespread use but ensuring modifications' freedom. [[User:RB|RB]] 16:59, 1 November 2008 (UTC)

== Consolidation ==

We've got a '''lot''' of good ideas here, but in interest of not stepping on anyone's toes, it's getting rather disjointed and hard to read. Is anyone willing to (or allow me to) try to consolidate them into some sort of coherency? I'd like at least one of the admins ([[User:.FUF|.FUF]] or [[User:Simsong|Simsong]] to concur before anyone moves forward. I know the wiki way is to just let it grow, but even watching each addition I'm starting to have trouble visualizing where we are. [[User:RB|RB]]

:: That's good idea, but it is important to consolidate without losing the ideas and opinions. I think it's better for this page to enter some "stable" branch. And then we'll move to the next phase. What do you think, [[User:Simsong|Simsong]]? [[User:.FUF|.FUF]] 18:12, 1 November 2008 (UTC)

::: One option could be to break each major section into its own page so it can be properly discussed without clutter, then transclude each to this page. A dedicated namespace would probably be overkill, but since we're throwing out ideas should at least be mentioned. --[[User:RB|RB]] 19:03, 1 November 2008 (UTC)

:::: [[User:Joachim Metz|Joachim]] Separate the parts into topics. Have a discussion and an informational part per topic.
== POLA ==

[[User:Capibara|Capibara]] :Just noticed a presentation by Simson when looking for 'usable security' on google that included notes on designation nd authorization. From this I would gather that Simson has something of a background in [[capability security theory]]. This makes me feel we might have sufficient ground to discuss 'open file handles as capabilities' at the library level. [[AppArmor]] allows instances of executables to be confined to least privilege. Unix domain sockets allow open file handles to be passed around as capabilities. The combination of these two results in the possibility of using [[object capability]] based least authority at the process level of granularity. For such to work with libraries such as libaff, libewf and libtsk, these libraries must not insist on opening their own files. At the moment this rules out using libaff or libewf directly as such. [[CarvFs]] can provide a way around this, but it may be desirable to not have to use CarvFs in some cases. Given that the authors of both libaff and libewf are quite active here, It may be valid to discuss the possibility of extending the API of these libs so that the libs dont demand to open their own files, but could also be used by a confined carving tool that would get open file handles passed to it over a unix domain socket.

Anti-forensic techniques

2008-11-06T05:12:51Z

Capibara:

'''Anti-forensic techniques''' try to frustrate [[forensic investigator]]s and their [[techniques]].

This can include refusing to run when [[debugging]] mode is enabled, refusing to run when running inside of a [[virtual machine]], or deliberately overwriting data. Although some anti-forensic tools have legitimate purposes, such as overwriting sensitive data that shouldn't fall into the wrong hands, like any [[Tools|tool]] they can be abused.

=Traditional anti-forensics=
==Overwriting Data and Metadata==
=== Secure Data Deletion ===

[[Secure data deletion|Securely deleting]] data, so that it cannot be restored with forensic methods.

Overwriting programs typically operate in one of three modes:
# The program can overwrite the entire media.
# The program can attempt to overwrite individual files. This task is complicated by journaling file systems: the file itself may be overwritten, but portions may be left in the journal.
# The program can attempt to overwrite files that were previously “deleted” but left on the drive. Programs typically do this by creating one or more files on the media and then writing to these files until no free space remains, taking special measures to erase small files — for example, files that exist entirely within the Windows Master File Table of an NTFS partition (Garfinkel and Malan, 2005).

Programs employ a variety of techniques to overwrite data. Apple’s Disk Utility allows data to be overwritten with a single pass of NULL bytes, with 7 passes of random data, or with 35 passes of data. Microsoft’s cipher.exe, writes a pass of zeros, a pass of FFs, and a pass of random data, in compliance with DoD standard 5220.22-M. (US DoD, 1995). In 1996 Gutmann asserted that it might be possible to recover overwritten data and proposed a 35-pass approach for assured sanitization (Gutmann 1996). However, a single overwriting pass is now viewed as sufficient for [[Sanitizing Tools|sanitizing]] data from ATA drives with capacities over 15 GB that were manufactured after 2001 (NIST 2006).

Be aware that software 'data destroyers' may not necessarily do what they state on the burb site. In particular a common mistake is the oversight of how the underlying file system actually stores files, for instance a 'wipe drive' application that will write a series of random values across unallocated space on the hard disk may not take into account the slack space at the end of allocated data blocks. Thus allowing a large portion of old data to still be recoverable. This is a very handy for a forensic analyst, but not so handy for IT Managers.

===Overwriting Metadata===
If the examiner knows when an attacker had access to a Windows, Mac or Unix system, it is frequently possible to determine which files the attacker accessed, by examining file “access” times for every file on the system. Some CFTs can prepare a “timeline” of the attacker’s actions by sorting all of the computer’s timestamps in chronological order. Although an attacker could wipe the contents of the media, this action itself might attract attention. Instead, the attacker might hide her tracks by overwriting the access times themselves so that the timeline could not be reliably constructed.

For example, [[Timestomp]] will overwrite [[NTFS]] “create,” “modify,” “access,” and “change” timestamps ([[Metasploi]]t 2006). [[The Defiler’s Toolkit]] can overwrite inode timestamps and deleted directory entries on many Unix systems; timestamps on allocated files can also be modified using the Unix touch command ([[The Grugq]] 2003).

=== Preventing Data Creation ===
Prevent the creation of certain data in the first place. Data which was never there, obviously cannot be restored with forensic methods.

For example, a partition can be mounted read-only or accessed through the raw device to prevent the file access times from being updated. The Windows registry key HKLM\SYSTEM\CurrentControlSet\Control\FileSystem\NtfsDisableLastAccessUpdate can be set to “1” to disable updating of the last-accessed timestamp; this setting is default under Windows Vista (Microsoft 2006).

==Cryptography, Steganography, and other Data Hiding Approaches==
=== Encrypted Data ===
Cryptographic file systems transparently encrypt data when it is written to the disk and decrypt data when it is read back, making the data opaque to any attacker (or CFT) that does not have the key. These file systems are now readily available for Windows, Mac OS, and Linux. The key can be protected with a passphrase or stored on an auxiliary device such as a USB token. If there is no copy of the key, intentionally destroying the key makes the data stored on the media inaccessible (Boneh and Lipton, 1996). Even if the cryptographic system lacks an intentional sanitization command or “self-destruct,” cryptography can still be a potent barrier to forensic analysis if the cryptographic key is unknown to the examiner.

Cryptography can also be used at the application level. For example, Microsoft Word can be configured to encrypt the contents of a document by specifying that the document has a “password to open.” Although older versions of Microsoft Word encrypted documents with a 40-bit key that can be cracked with commercial tools, modern versions can optionally use a 128-bit encryption that is uncrackable if a secure passphrase is used.
=== Encrypted Network Protocols===
Network traffic can likewise be encrypted to protect its content from forensic analysis. Cryptographic encapsulation protocols such as [[SSL forensics|SSL]] and SSH only protect the content of the traffic. Protecting against traffic analysis requires the use of intermediaries. Onion Routing (Goldschlag, Reed and Syverson, 1999) combines both approaches with multiple layers of encryption, so that no intermediary knows both ends of the communication and the plaintext content.

''More information: [[Tor]] and [[VPN]].''

===Program Packers===
Packers are commonly used by attackers so that attack tools will not be subject to reverse engineering or detection by scanning. Packers such as PECompact (Bitsum 2006) and Burneye (Vrba 2004) will take a second program, compress and/or encrypt it, and wrap it with a suitable extractor. Packers can also incorporate active protection against debugging or reverse engineering techniques. For example, Shiva will exit if its process is being traced; if the process is not being traced, it will create a second process, and the two processes will then trace each other, since each process on a Unix system may only be traced by one other process. (Mehta and Clowes, 2003)

Packed programs that require a password in order to be run can be as strong as their encryption and password. However, the programs are vulnerable at runtime. Burndump is a loadable kernel module (LKM) that automatically detects when a Burneye-protected file is run, waits for the program to be decrypted, and then writes the raw, unprotected binary to another location (ByteRage 2002). Packed programs are also vulnerable to static analysis if no password is required (Eagle 2003).
=== Steganography ===
Steganography can be used to embed encrypted data in a cover text to avoid detection. Steghide embeds text in JPEG, MBP, MP3, WAV and AU files (Hetzl 2002). Hydan exploits redundancy in the x86 instruction set; it can encode roughly 1 byte per 110 (El-Khalil 2004). Stegdetect (Provos 2004) can detect some forms of steganography.

StegFS hides encrypted data in the unused blocks of a Linux ext2 file system, making the data “look like a partition in which unused blocks have recently been overwritten with random bytes using some disk wiping tool” (McDonald and Kuhn, 2003).

TrueCrypt allows a second encrypted file system to be hidden within the first (TrueCrypt 2006). The goal of this filesystem-within-a-filesystem is to allow the TrueCrypt users to have a “decoy” file system with data that is interesting but not overtly sensitive. A person who is arrested or captured with a TrueCrypt-protected laptop could then give up the first file system’s password, with the hope that the decoy would be sufficient to satisfy the person’s interrogators.

=== Generic Data Hiding===
Data can also be hidden in unallocated or otherwise unreachable locations that are ignored by the current generation of forensic tools.

Metasploit’s Slacker will hide data within the slack space of FAT or NTFS file system. FragFS hides data within the NTFS Master File Table. RuneFS (Grugq 2003) stores data in bad blocks. (Thompson and Monroe, 2006). Waffen FS stores data in the ext3 journal file (Eckstein and Jahnke 2005). KY FS stores data in directories (Grugq 2003). Data Mule FS stores data in inode reserved space (Grugq 2003). It is also possible to store information in the unallocated pages of Microsoft Office files.

Information can be stored in the Host Protected Area (HPA) and the Device Configuration Overlay (DCO) areas of modern ATA hard drives. Data in the HPA and DCO is not visible to the BIOS or operating system, although it can be extracted with special tools.

== Detecting Forensic Analysis ==

There are methods to detect whether an [[investigator]] tries to perform a (live) forensic analysis on the system. A malicious user or program could react to that by destroying evidence, for example.

=Other Anti Forensics=
==Targeting forensic tool blind spots==
==Targeting forensic tool vulnerabilities==
==Targeting generic tool/lib vulnerabilities==
=References=
Garfinkel, S., Anti-Forensics: Techniques, Detection and Countermeasures, The 2nd International Conference on i-Warfare and Security (ICIW), Naval Postgraduate School, Monterey, CA, March 8-9, 2007. [http://www.simson.net/clips/academic/2007.ICIW.AntiForensics.pdf]

Henrique, G. Wendel, Anti Forensics: Making computer forensics hard, Code Breakers III, São Paulo, Brazil, Setember 2006.
[http://ws.hackaholic.org/slides/AntiForensics-CodeBreakers2006-Translation-To-English.pdf]
== See also ==

* [[:Category:Anti-forensics tools|Anti-forensics tools category]]

== Externals Links ==

* [http://www.safehack.com/Textware/forensic/Anti_Forensic_Break_Encase.pdf Breaking Encase with FILE0 and Winhex]

* [http://ws.hackaholic.org/slides/AntiForensics-CodeBreakers2006-Translation-To-English.pdf Anti Forensics: making computer forensics hard]

* [http://seclists.org/bugtraq/2008/Nov/0038.html PTK Forensic Local Command Execution Vulnerability]

[[Category:Anti-Forensics]]

Carver 2.0 Planning Page

2008-11-03T19:26:01Z

Capibara: /* Ideas */

This page is for planning Carver 2.0.

Please, do not delete text (ideas) here. Use something like this:

<pre>
<s>bad idea</s>
:: good idea
</pre>

This will look like:

<s>bad idea</s>
:: good idea

= License =

BSD-3.
:: [[User:Joachim Metz|Joachim]] library based validators could require other licenses
::: Make the other libraries plug-able. If you them, you use them. [[User:Simsong|Simsong]] 06:34, 3 November 2008 (UTC)

= OS =

Linux/FreeBSD/MacOS
: Shouldn't this just match what the underlying afflib & sleuthkit cover? [[User:RB|RB]]
:: Yes, but you need to test and validate on each. Question: Do we want to support windows? [[User:Simsong|Simsong]] 21:09, 30 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] I think we would do wise to design with windows support from the start this will improve the platform independence from the start
:::: Agreed; I would even settle at first for being able to run against Cygwin. Note that I don't even own or use a copy of Windows, but the vast majority of forensic investigators do. [[User:RB|RB]] 14:01, 31 October 2008 (UTC)
:: [[User:Capibara|Rob J Meijer]] Leaning heavily on the autotools might be the way to go. I do however feel that support requirements for windows would not be essential. Being able to run from a virtual machine with the main storage mounted over cifs should however be tested and if possible tuned extensively.
:::: [[User:Joachim Metz|Joachim]] You'll need more than autotools to do native Windows support i.e. file access, UTF-16 support, wrap some basic system functions or have them available otherwise
::::::[[User:Capibara|Rob J Meijer]] That´s exactly my point, windows support as in being able to build and run on windows natively is much more trouble than its worth. Better make for a lean and mean autotools based build with little dependencies and no or little recursion, and better spent effort on a lean POLA design on POSIX based systems than on supporting building and running on non POSIX systems.

= Name tooling =

* [[User:Joachim Metz|Joachim]] A name for the tooling I propose coldcut
:: How about 'butcher'? ;) [[User:RB|RB]] 14:20, 31 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] cleaver ( scalpel on steroids ;-) )
* I would like to propose Gouge or Chisel :-) [[User:Capibara|Rob J Meijer]]

= Requirements =

[[User:Joachim Metz|Joachim]] Could we do a MoSCoW evaluation of these.
* AFF and EWF file images supported from scratch. ([[User:Joachim Metz|Joachim]] I would like to have raw/split raw and device access as well)
:: If we base our image i/o on afflib, we get all three with one interface. [[User:RB|RB]] Instead of letting the tools use afflib, better to write an afflib module for carvfs, and update the libewf module. The tool could than be oblivious of the file format. [[User:Capibara|Rob J Meijer]]
:::: [[User:Simsong|Simsong]] 06:29, 3 November 2008 (UTC) The problem with using carvfs is that this adds another dependency. Do you really want to require that people install carvfs in order to run the carver? What about having the thing ported to Windows?
:::::: [[User:Capibara|Rob J Meijer]] I would support adding one build dependency (libcarvpath) and removing two (libewf/libaff) by moving them to a layer more suited for them (carvfs) that would possibly allow some form of file handle (as cap) based POLA design. I am a proponent of making small things that do one and do one thing right, and to stack those to do what you need. In my view that would lead ideally to the following (simplified) chain:
::::::* recursive-forensic-framework (ocfa/pyflag)
::::::** The-(pola-based)-carving-tool
::::::*** The-carving-lib working on open fd's.
::::::**** libcarvpath
::::::***** carvfs (Over cifs/nfs-v4 on platforms that don't support Fuse).
::::::****** libewf
::::::****** libaff
::::::*** AppArmor (on supporting platforms)
::::::*** suid (on supporting platforms)
::::::*** iptables/ipfw (on supporting platforms)
:::::: As fow windows support, I would imagine making carvfs run over smb would come a long way, that is for as far as windows support is all that relevant.
:::::: There are two advantages to using libcarvpath and carvfs instead of libaff/libewf t this layer:
::::::* storage requirements for doing carving. Beyond what sleuthkit or alternatives provide I have seen many situations where carving was not done due to storage limitations.
::::::* File handles are like object capabilities. You can often do pretty simple POLA based implementations using file handles and something like AppArmor. POLA could IMHO be a strong weapon against the more nasty forms of anti forensics.
::::::Next to this, I would consider making different tools for different stages instead of one semi recursive one, and looking at how to integrate these tools into existing frameworks (ocfa/pyflag).
::::::Keep things simple but rigid and try to easily integrate things into existing frameworks as effectively as possible I would suggest.
::::::Please note, I am not ptoposing the lib/tool should be useless without libcarvpath, only that usage without carvfs should limit the
::::::supported image formats to raw images, and that libewf/libaff should be abstracted at the Fuse level or below and not at the tool level.
:::::::[[User:Joachim Metz|Joachim]] do you have an idea what the performance impact of this approach would be? It might be wise to do a proof of concept for this approach first.
::::::::[[User:Capibara|Rob J Meijer]] It would I think depend greatly on behavior of the carving lib/tool. Small 512 byte reads are relatively very expensive, 128kb reads have negligible impact. Here are some numbers from ntfs-3g: http://article.gmane.org/gmane.comp.file-systems.fuse.devel/6397/match=ntfs+3g+performance+ext3 that might be relevant. More relevant than performance might be library footprint. For example, using OCFA, we often would want to keep e few hundred images that total to tens of TB of projected storage size fuse mounted. If libewf/libaff have a big combined memory footprint in such cases, this can be a major issue for this approach.
:::: [[User:Joachim Metz|Joachim]] this layer should support multi threaded decompression of compressed image types, this speeds up IO
* [[User:Joachim Metz|Joachim]] volume/partition aware layer (what about carving unpartioned space)
* File system aware layer. This could be or make use of tsk-cp.
** By default, files are not carved. (clarify: only identified? [[User:RB|RB]]; I guess that it operates like [[Selective file dumper]] [[User:.FUF|.FUF]] 07:00, 29 October 2008 (UTC)). Alternatively, the tool could use libcarvpath and output carvpaths or create a directory with symlinks to carvpaths that point into a carvfs mountpoint [[User:Capibara|Rob J Meijer]].
* Plug-in architecture for identification/validation.
** [[User:Joachim Metz|Joachim]] support for multiple types of validators
*** dedicated validator
*** validator based on file library (i.e. we could specify/implement a file structure API for these)
*** configuration based validator (Can handle config files,like Revit07, to enter different file formats used by the carver.)

[[User:Joachim Metz|Joachim]]
Moderator: Could we limit the requirements for prototype version 1 of the tool to get a working version up and running ASAP?
And keep discussing future options?

I think the following set will be large enough to handle:
Input facilities
* IO support (AFF, device, EWF, RAW and split RAW)
:: Abstraction of input format and multi threaded decompression (spin-off code out of afflib?)
* Volume/Partitions support
:: at least for DOS based layout and GPT (spin-off code out of TSK/Photorec?)
* File system support
:: VFAT/NTFS (spin-off code out of TSK/Photorec?)

Carving facilities
* File format support using plug-able validator model (use dedicated validators Photorec/Scarve and/or wrap revit07 file format as validator?)
* Content support using plug-able validator model (to handle text/mbox base64)
* File system carving support (to handle file system fragments, could be linked to file system support layer?)
* Basic fragment handling

Output facilities
* audit/analysis/debug log
* extraction of result files

==Supported File Formats==
* Ship with validators for:
[[User:Joachim Metz|Joachim]] I think we should distinguish between file format validators and content validators
** JPEG
** PNG
** GIF
** MSOLE
** ZIP
** TAR (gz/bz2)

[[User:Joachim Metz|Joachim]] For a production carver we need at least the following formats
** Grapical Images
*** JPEG (the 3 different types with JFIF/EXIF support)
*** PNG
*** GIF
*** BMP
*** TIFF
** Office documents
*** OLE2 (Word/Excell content support)
*** PDF
*** Open Office/Office 2007 (ZIP+XML)
:: Extension validation? AFAIK, MS Office 2007 [[DOCX]] format uses plain ZIP (or not?), and carved files will (or not?) have .zip extension instead of DOCX. Is there any way to fix this (may be using the file list in zip)? [[User:.FUF|.FUF]] 20:25, 31 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] Addition: Office 2007 also has a binary file format which is also a ZIP-ed data

==Archive Files==
** Archive files
*** ZIP
*** 7z
*** gzip
*** bzip2
*** tar
*** RAR
** E-mail files
*** PFF (PST/OST)
*** MBOX (text based format, base64 content support)
** Audio/Video files
*** MPEG
*** MP2/MP3
*** AVI
*** ASF/WMV
*** QuickTime
*** MKV
** Printer spool files
*** EMF (if I remember correctly)
** Internet history files
*** index.dat
*** firefox (sqllite 3)
** Other files
*** thumbs.db
*** pagefile?

==Carving Strategies==
[[User:Joachim Metz|Joachim]] Note to moderator could this section be merged with the carving algorithm section?

* Simple fragment recovery carving using gap carving.
** [[User:Joachim Metz|Joachim]] have hook in for more advanced fragment recovery?
* Recovering of individual ZIP sections and JPEG icons that are not sector aligned.
** [[User:Joachim Metz|Joachim]] I would propose a generic fragment detection and recovery
* Autonomous operation (some mode of operation should be completely non-interactive, requiring no human intervention to complete [[User:RB|RB]])
** [[User:Joachim Metz|Joachim]] as much as possible, but allow to be overwritten by user
* [[User:Joachim Metz|Joachim]] When the tool output files the filenames should contain the offset in the input data (in hexadecimal?)
:: [[User:Mark Stam|Mark]] I really like the fact carved files are named after the physical or logical sector in which the file is found (photorec)
:::: [[User:Joachim Metz|Joachim]] This naming schema might cause duplicate name problem for extracting embedded files and extracting files from non sector aligned file systems.
* [[User:Joachim Metz|Joachim]] Should the tool allow to export embedded files?
* [[User:Joachim Metz|Joachim]] Should the tool allow to export fragments separately?
* [[User:Mark Stam|Mark]] I personally use photorec often for carving files in the whole volume (not only unallocated clusters), so I can store information about all potential interesting files in MySQL
:: [[User:Joachim Metz|Joachim]] interesting, Bas Kloet and me have been discussing to use information about allocated files in the recovery process, i.e. recovered fragments could be part of allocated files. Do we want to be able to extract them? Or could we rebuild the file from the fragments and the allocated files.
* [[User:Mark Stam|Mark]] It would also be nice if the files can be hashed immediately (MD5) so looking for them in other tools (for example Encase) is a snap

==Performance Requirements==
* Tested on 500GB-sized images. Should be able to carve a 500GB image in roughly 50% longer than it takes to read the image.
** Perhaps allocate a percentage budget per-validator (i.e. each validator adds N% to the carving time) [[User:RB|RB]]
** [[User:Joachim Metz|Joachim]] have multiple carving phases for precision/speed trade off?
* Parallelizable
** [[User:Joachim Metz|Joachim]] tunable for different architectures
* Configuration:
** Capability to parse some existing carvers' configuration files, either on-the-fly or as a one-way converter.
** Disengage internal configuration structure from configuration files, create parsers that present the expected structure
** [[User:Joachim Metz|Joachim]] The validator should deal with the file structure the carving algorithm should not know anything about the file structure (as in revit07 design)
** Either extend Scalpel/Foremost syntaxes for extended features or use a tertiary syntax ([[User:Joachim Metz|Joachim]] I would prefer a derivative of the revit07 configuration syntax which already has encountered some problems of dealing with defining file structure in a configuration file)

==Output==
* Can output audit.txt file.
* [[User:Joachim Metz|Joachim]] Can output database with offset analysis values i.e. for visualization tooling
* [[User:Joachim Metz|Joachim]] Can output debug log for debugging the algorithm/validation
* Easy integration into ascription software.
:: [[User:Joachim Metz|Joachim]] I'm no native speaker what do you mean with "ascription software"?
::: I think this was another non-native requesting easy scriptability. [[User:RB|RB]] 14:20, 31 October 2008 (UTC)
:::: [[User:Joachim Metz|Joachim]] that makes sense ;-)
::::: Incorrect. Ascription software is software that determines who the owner of a file is. [[User:Simsong|Simsong]] 06:36, 3 November 2008 (UTC)

= Ideas =
* Use as much TSK if possible. Don't carry your own FS implementation the way photorec does.
:: [[User:Joachim Metz|Joachim]] using TSK as much as possible would not allow to add your own file system support (i.e. mobile phones, memory structures, cap files) I would propose wrapping TSK and using it as much as possible but allow to integrate own FS implementations.
* Extracting/carving data from [[Thumbs.db]]? I've used [[foremost]] for it with some success. [[Vinetto]] has some critical bugs :( [[User:.FUF|.FUF]] 19:18, 28 October 2008 (UTC)

==Recursive Carving==
[[User:Joachim Metz|Joachim]] do we want to support (let's call it) 'recursive in file carving' (for now) this is different from embedded files because there is a file system structure in the file and not just another file structure
* Is it just me, or do a lot of the above (and below) ideas somewhat skirt around the fact that many of us want recursive carving? Can we bend back to that instead of discussing object particulars? I think this can be distilled down to three requirements:
** Simple recursion: once an object is identified, have the ability to re-carve it for internal structures
** Directed recursion: the carver should be able to be directed at arbitrary blobs and told to carve it as a specified type. This allows programmatically more simple methods of dealing with unidentifiably compressed or encrypted data. Or filesystem fragments.
** Export: the ability to export an object (recognized or not) for later or external "recursion". Should go without saying for a carver, but...
:--[[User:RB|RB]] 18:45, 2 November 2008 (UTC)
:: [[User:Simsong|Simsong]] 06:30, 3 November 2008 (UTC) pyflag already does recursive carving. Are we just going to reimplement pyflag as a single executable?

==Library Dependencies==
[[User:Capibara|Rob J Meijer]] :
* Use libcarvpath whenever possible and by default to avoid high storage requirements.
:: [[User:Joachim Metz|Joachim]] For easy deployment I would not opt for making an integral part of the tool solely dependant on a single external library or the library must be integrated in the package
::[[User:Capibara|Rob J Meijer]] Integrating libraries (libtsk,libaff.libewf,libcarvpath etc) is bad practice, autotools are your friend IMO.
:: [[User:Joachim Metz|Joachim]] I'm not talking about integrating (shared) libraries. I'm talking about that an integral part of a tool should be part of it's package. Why can't the tool package contain shared or static libraries for local use? A far worse thing to do is to have a large set of dependencies and making the tool difficult to install for most users. The tool package should contain the most necessary code. afflib/libewf support could be detected by the autotools a neat separation of functionality.
::: From a packager's standpoint, [[User:Joachim Metz|Joachim]]'s other libraries do a really good job of this, carrying around what they need but using a system-global version if available. [[User:RB|RB]]
* libtsk
* libaff ? : possibly the discussion in the requirements section should move to his section.
* libewf ? : possibly the discussion in the requirements section should move to his section.
* posix ? : Can we depend especially on the availability of UNIX domain sockets and the possibility to use msg_accrights for passing opn file handles as ocaps?

==Filesystem Detection==
* Dont stop with filesystem detection after the first match. Often if a partition is reused with a new FS and is not all that full yet, much of the old FS can still be valid. I have seen this with ext2/fat. The fact that you have identified a valid FS on a partition doesn't mean there isn't an(almost) valid second FS that would yield additional files. Identifying doubly allocated space might in some cases also be relevant.
:: [[User:Joachim Metz|Joachim]] What your saying is that dealing with file system fragments should be part of the carving algorithm
* Allow use where filesystem based carving is done by other tool, and the tool is used as second stage on (sets of) unallocated block (pseudo) files and/or non FS partition (pseudo) files.
:: [[User:Joachim Metz|Joachim]] I would not opt for this. The tool would be dependent on other tools and their data format, which makes the tool difficult to maintain. I would opt to integrate the functionality of having multiple recovery phases (stages) and allow the tooling to run the phases after one and other or separately.
::[[User:Capibara|Rob J Meijer]] More generically, I feel a way should exist to communicate the 'left overs' a previous (non open, for example LE-only) tool left.
:: [[User:Joachim Metz|Joachim]] I guess if the tool is designed to handle multiple phases it should store its data somewhere. So it should be possible to convert results of such non open tooling to the format required. However I would opt to design the recovery functionality of these non-open tools into open tools. And not to limit ourselves making translators due to the design of these non-open tools.
* Ability to be used as a library instead of a tool. Ability to access metadata true library, and thus the ability to set metadata from the carving modules. This would be extremely usefull for integrating the project into a framework like ocfa.
:: [[User:Joachim Metz|Joachim]] I guess most of the code could be integrated into libraries, but I would not opt integrating tool functionality into a library
* [[User:Mark Stam|Mark]] I think it would be very handy to have a CSV, TSV, XML or other delimited output (log)file with information about carved files. This output file can then be stored in a database or Excel sheet (report function)

==Anti forensics and system integrity concerns==
* It might be very interesting to look at the possibilities of using a multi process style of module support and combine it with a least authority design. On platforms that support AppArmor (or similar) and uid based firewall rules, this could make for the first true POLA (principle of least authority) based forensic tool ever. POLA based forensics tools should make for a strong integrity guard against many anti forensics. Alternatively we could look at integrating a capability secure language (E?) for implementation of at least validation modules. I don't expect this idea to make it, but mentioning it I hope might spark off less strong alternatives that at least partially address the integrity + anti-forensics problem. If we can in some way introduce POLA to a wider forensics public, other tools might also pick up on it what would be great.
:: [[User:Joachim Metz|Joachim]] Could you give an example of how you see this in action?
::::[[User:Capibara|Rob J Meijer]] I see two layers where using POLA could be applied. The best one would require one of the folowing as prerequisites:
::::* The libaff/libewf layer is moved to a fuse implementation (for example carvfs).
::::* Libewf/Libaff are updated to accept opened filhandles instead of demanding to open their own files.
::::If one of these is fulfilled, than the tool running as some user can just have the simple task of opening the image files, starting up the 'real' tool and handing over the appropriate file handles. If the real tool runs with a restrictive AppArmor profile, and is started suid to a tool specific user that also has its own iptables uid based filter, than the real tool will run with least authority.
:::: A second alternative, if neither of the first prerequisite could not be bet, would be to run the modules as confined processes and have a non confined process run as proxy for the first.
:::: A third probably far fetched alternative would be to embed an object capability language in the tool and make the module interface thus that modules are to be written in this ocap language.
::::A 4th alternative might include minorfs or plash, but I havn't geven those sufficient thinking hours yet.

== Format syntax specification ==
* Carving data structures. For example, extract all TCP headers from image by defining TCP header structure and some fields (e.g. source port > 1024, dest port = 80). This will extract all data matching the pattern and write a file with other fields. Another example is carving INFO2 structures and URL activity records from index.dat [[User:.FUF|.FUF]] 20:51, 28 October 2008 (UTC)
** This has the opportunity to be extended to the concept of "point at blob FOO and interpret it as BAR"
.FUF added:
The main idea is to allow users to define structures, for example (in pascal-like form):

<pre>
Field1: Byte = 123;
SomeTextLength: DWORD;
SomeText: string[SomeTextLength];
Field4: Char = 'r';
...
</pre>

This will produce something like this:
<pre>
Field1 = 123
SomeTextLength = 5
SomeText = 'abcd1'
Field4 = 'r'
</pre>

(In text or raw forms.)

Opinions?

Opinion: Simple pattern identification like that may not suffice, I think Simson's original intent was not only to identify but to allow for validation routines (plugins, as the original wording was). As such, the format syntax would need to implement a large chunk of some programming language in order to be sufficiently flexible. [[User:RB|RB]]

[[User:Joachim Metz|Joachim]]
In my option your example is too limited. Making the revit configuration I learned you'll need a near programming language to specify some file formats.
A simple descriptive language is too limiting. I would also go for 2 bytes with endianess instead of using terminology like WORD and small integer, it's much more clear. The configuration also needs to deal with aspects like cardinality, required and optional structures.
:: This is simply data structures carving, see ideas above. Somebody (I cannot track so many changes per day) separated the original text. There is no need to count and join different structures. [[User:.FUF|.FUF]] 19:53, 31 October 2008 (UTC)
:::: [[User:Joachim Metz|Joachim]] This was probably me is the text back in it's original form?
:::: I started it by moving your Revit07 comment to the validator/plugin section in [http://www.forensicswiki.org/index.php?title=Carver_2.0_Planning_Page&diff=prev&oldid=7583 this edit], since I was still at that point thinking operational configuration for that section, not parser configurations. [[User:RB|RB]]
:::: [[User:Joachim Metz|Joachim]] I renamed the title to format syntax, clarity is important ;-)

Please take a look at the revit07 format syntax specification (configuration). It's not there yet but goes a far way. Some things currently missing:
* bitwise alignment
* handling encapsulated streams (MPEG/capture files)
* handling content based formats (MBOX)

=Caving algorithm =
[[User:Joachim Metz|Joachim]]
* should we allow for multiple carving phases (runs/stages)?
:: I opt yes (separation of concern)
* should we allow for multiple carving algorithms?
:: I opt yes, this allows testing of different approaches
* Should the algorithm try to do as much in 1 run over the input data? To reduce IO?
:: I opt that the tool should allow for multiple and single run over the input data to minimize the IO or the CPU as bottleneck
* Interaction between algorithm and validators
** does the algorithm passes data blocks to the validators?
** does a validator need to maintain a state?
** does a validator need to revert a state?
** How do we deal with embedded files and content validation? Do the validators call another validator?
* do we use the assumption that a data block can be used by a single file (with the exception of embedded/encapsulated files)?
* Revit07 allows for multiple concurrent result files states to deal with fragmentation. One has the attribute of being active (the preferred) and the other passive. Do we want/need something similar? The algorithm adds block of input data (offsets) to these result files states.
** if so what info would these result files states require (type, list of input data blocks)
* how do we deal with file system remainders?
** Can we abstract them and compare them against available file system information?
* Do we carve file systems in files?
:: I opt that at least the validator uses this information

==Caving scenarios ==
[[User:Joachim Metz|Joachim]]
* normal file (file structure, loose text based structure (more a content structure?))
* fragmented file (the file entirely exist)
* a file fragment (the file does not entirely exist)
* intertwined file
* encapsulated file (MPEG/network capture)
* embedded file (JPEG thumbnail)
* obfuscation ('encrypted' PFF) this also entails encryption and/or compression
* file system in file

=File System Awareness =
==Background: Why be File System Aware?==
Advantages of being FS aware:
* You can pick up sector allocation sizes
:: [[User:Joachim Metz|Joachim]] do you mean file system block sizes?
* Some file systems may store things off sector boundaries. (ReiserFS with tail packing)
* Increasingly file systems have compression (NTFS compression)
* Carve just the sectors that are not in allocated files.

==Tasks that would be required==

==Discussion==
:: As noted above, TSK should be utilized as much as possible, particularly the filesystem-aware portion. If we want to identify filesystems outside of its supported set, it would be more worth our time to work on implementing them there than in the carver itself. [[User:RB|RB]]

:::: I guess this tool operates like [[Selective file dumper]] and can recover files in both ways (or not?). Recovering files by using carving can recover files in situations where sleuthkit does nothing (e.g. file on NTFS was deleted using ntfs-3g, or filesystem was destroyed or just unknown). And we should build the list of filesystems supported by carver, not by TSK. [[User:.FUF|.FUF]] 07:08, 29 October 2008 (UTC)

:: This tool is still in the early planning stages (requirements discovery), hence few operational details (like precise modes of operation) have been fleshed out - those will and should come later. The justification for strictly using TSK for the filesystem-sensitive approach is simple: TSK has good filesystem APIs, and it would be foolish to create yet another standalone, incompatible implementation of filesystem(foo) when time would be better spent improving those in TSK, aiding other methods of analysis as well. This is the same reason individuals that have implemented several other carvers are participating: de-duplication of effort. [[User:RB|RB]]

:: [[User:Joachim Metz|Joachim]] A design problem might be that TSK currently is a single library operating on multiple layers (storage media IO, volume/partition analysis and file system analysis). I'm not aware how easily the parts can be used separately. But I estimate that for the carver we want to use these 3 layers differently than TSK currently does.

[[User:Joachim Metz|Joachim]] I would like to have the carver (recovery tool) also do recovery using file allocation data or remainders of file allocation data.

[[User:Joachim Metz|Joachim]]
I would go as far to ask you all to look beyond the carver as a tool and look from the perspective of the carver as part of the forensic investigation process. In my eyes certain information needed/acquired by the carver could be also very useful investigative information i.e. what part of a hard disk contains empty sectors.

=Supportive tooling=
[[User:Joachim Metz|Joachim]]
* validator (definitions) tester (detest in revit07)
* tool to make configuration based definitions
* post carving validation
* the carver needs to provide support for fuse mount of carved files (carvfs)

=Testing =
[[User:Joachim Metz|Joachim]]
* automated testing
* test data

=Validator Construction=
Options:
* Write validators in C/C++
** [[User:Joachim Metz|Joachim]] you mean dedicated validators
* Have a scripting language for writing them (python? Perl?) our own?
** [[User:Joachim Metz|Joachim]] use easy to embed programming languages i.e. Phyton or Lua
* Use existing programs (libjpeg?) as plug-in validators?
** [[User:Joachim Metz|Joachim]] define a file structure api for this

=Existing Code that we have=
[[User:Joachim Metz|Joachim]]
Please add any missing links

Documentation/Articles
* DFRWS2006/2007 carving challenge results
* DFRWS2008 paper on carving

Carvers
* DFRWS2006/2007 carving challenge results
* photorec (http://www.cgsecurity.org/wiki/PhotoRec)
* revit06 and revit07 (http://sourceforge.net/projects/revit/)
* s3/scarve

Possible file structure validator libraries
* divers existing file support libraries
* libole2 (inhouse experimental code of OLE2 support)
* libpff (alpha release for PFF (PST/OST) file support) (http://sourceforge.net/projects/libpff/)

Input support
* AFF (http://www.afflib.org/)
* EWF (http://sourceforge.net/projects/libewf/)
* TSK device & raw & split raw (http://www.sleuthkit.org/)

Volume/Partition support
* disktype (http://disktype.sourceforge.net/)
* testdisk (http://www.cgsecurity.org/wiki/TestDisk)
* TSK

File system support
* TSK
* photorec FS code
* implementations of FS in Linux/BSD

Content support

Zero storage support
* libcarvpath ( http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=210704 )
* carvfs ( http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=210954 )
* tsk-cp ( http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=267227 )
* carvfsmodewf (http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=268256 )
POLA
* joe-e (java) ( http://code.google.com/p/joe-e/ )
* Emily (ocaml) ( http://erights.org/download/emily/ )
* the E language ( http://www.erights.org/ )
* AppArmor
* iptables/ipfw
* minorfs ( http://polacanthus.net/minorfs.html )
* plash ( http://plash.beasts.org/wiki/ )

=Implementation Timeline=
# gather the available resources/ideas/wishes/needs etc. (I guess we're in this phase)
# start discussing a high level design (in terms of algorithm, facilities, information needed)
## input formats facility
## partition/volume facility
## file system facility
## file format facility
## content facility
## how to deal with fragment detection (do the validators allow for fragment detection?)
## how to deal with recombination of fragments
## do we want multiple carving phases in light of speed/precision tradeoffs
# start detailing parts of the design
## Discuss options for a grammar driven validator?
## Hard-coded plug-ins?
## Which existing code can we use?
# start building/assembling parts of the tooling for a prototype
## Implement simple file carving with validation.
## Implement gap carving
# Initial Release
# Implement the ''threaded carving'' that [[User:.FUF|.FUF]] is describing above.

[[User:Joachim Metz|Joachim]] Shouldn't multi threaded carving (MTC) not be part of the 1st version?
The MT approach makes for different design decisions
: It is virtually impossible to turn a non-MT application into an MT application .[[User:Simsong|Simsong]] 06:37, 3 November 2008 (UTC)

Carver 2.0 Planning Page

2008-11-03T19:23:46Z

Capibara: /* Library Dependencies */

This page is for planning Carver 2.0.

Please, do not delete text (ideas) here. Use something like this:

<pre>
<s>bad idea</s>
:: good idea
</pre>

This will look like:

<s>bad idea</s>
:: good idea

= License =

BSD-3.
:: [[User:Joachim Metz|Joachim]] library based validators could require other licenses
::: Make the other libraries plug-able. If you them, you use them. [[User:Simsong|Simsong]] 06:34, 3 November 2008 (UTC)

= OS =

Linux/FreeBSD/MacOS
: Shouldn't this just match what the underlying afflib & sleuthkit cover? [[User:RB|RB]]
:: Yes, but you need to test and validate on each. Question: Do we want to support windows? [[User:Simsong|Simsong]] 21:09, 30 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] I think we would do wise to design with windows support from the start this will improve the platform independence from the start
:::: Agreed; I would even settle at first for being able to run against Cygwin. Note that I don't even own or use a copy of Windows, but the vast majority of forensic investigators do. [[User:RB|RB]] 14:01, 31 October 2008 (UTC)
:: [[User:Capibara|Rob J Meijer]] Leaning heavily on the autotools might be the way to go. I do however feel that support requirements for windows would not be essential. Being able to run from a virtual machine with the main storage mounted over cifs should however be tested and if possible tuned extensively.
:::: [[User:Joachim Metz|Joachim]] You'll need more than autotools to do native Windows support i.e. file access, UTF-16 support, wrap some basic system functions or have them available otherwise
::::::[[User:Capibara|Rob J Meijer]] That´s exactly my point, windows support as in being able to build and run on windows natively is much more trouble than its worth. Better make for a lean and mean autotools based build with little dependencies and no or little recursion, and better spent effort on a lean POLA design on POSIX based systems than on supporting building and running on non POSIX systems.

= Name tooling =

* [[User:Joachim Metz|Joachim]] A name for the tooling I propose coldcut
:: How about 'butcher'? ;) [[User:RB|RB]] 14:20, 31 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] cleaver ( scalpel on steroids ;-) )
* I would like to propose Gouge or Chisel :-) [[User:Capibara|Rob J Meijer]]

= Requirements =

[[User:Joachim Metz|Joachim]] Could we do a MoSCoW evaluation of these.
* AFF and EWF file images supported from scratch. ([[User:Joachim Metz|Joachim]] I would like to have raw/split raw and device access as well)
:: If we base our image i/o on afflib, we get all three with one interface. [[User:RB|RB]] Instead of letting the tools use afflib, better to write an afflib module for carvfs, and update the libewf module. The tool could than be oblivious of the file format. [[User:Capibara|Rob J Meijer]]
:::: [[User:Simsong|Simsong]] 06:29, 3 November 2008 (UTC) The problem with using carvfs is that this adds another dependency. Do you really want to require that people install carvfs in order to run the carver? What about having the thing ported to Windows?
:::::: [[User:Capibara|Rob J Meijer]] I would support adding one build dependency (libcarvpath) and removing two (libewf/libaff) by moving them to a layer more suited for them (carvfs) that would possibly allow some form of file handle (as cap) based POLA design. I am a proponent of making small things that do one and do one thing right, and to stack those to do what you need. In my view that would lead ideally to the following (simplified) chain:
::::::* recursive-forensic-framework (ocfa/pyflag)
::::::** The-(pola-based)-carving-tool
::::::*** The-carving-lib working on open fd's.
::::::**** libcarvpath
::::::***** carvfs (Over cifs/nfs-v4 on platforms that don't support Fuse).
::::::****** libewf
::::::****** libaff
::::::*** AppArmor (on supporting platforms)
::::::*** suid (on supporting platforms)
::::::*** iptables/ipfw (on supporting platforms)
:::::: As fow windows support, I would imagine making carvfs run over smb would come a long way, that is for as far as windows support is all that relevant.
:::::: There are two advantages to using libcarvpath and carvfs instead of libaff/libewf t this layer:
::::::* storage requirements for doing carving. Beyond what sleuthkit or alternatives provide I have seen many situations where carving was not done due to storage limitations.
::::::* File handles are like object capabilities. You can often do pretty simple POLA based implementations using file handles and something like AppArmor. POLA could IMHO be a strong weapon against the more nasty forms of anti forensics.
::::::Next to this, I would consider making different tools for different stages instead of one semi recursive one, and looking at how to integrate these tools into existing frameworks (ocfa/pyflag).
::::::Keep things simple but rigid and try to easily integrate things into existing frameworks as effectively as possible I would suggest.
::::::Please note, I am not ptoposing the lib/tool should be useless without libcarvpath, only that usage without carvfs should limit the
::::::supported image formats to raw images, and that libewf/libaff should be abstracted at the Fuse level or below and not at the tool level.
:::::::[[User:Joachim Metz|Joachim]] do you have an idea what the performance impact of this approach would be? It might be wise to do a proof of concept for this approach first.
::::::::[[User:Capibara|Rob J Meijer]] It would I think depend greatly on behavior of the carving lib/tool. Small 512 byte reads are relatively very expensive, 128kb reads have negligible impact. Here are some numbers from ntfs-3g: http://article.gmane.org/gmane.comp.file-systems.fuse.devel/6397/match=ntfs+3g+performance+ext3 that might be relevant. More relevant than performance might be library footprint. For example, using OCFA, we often would want to keep e few hundred images that total to tens of TB of projected storage size fuse mounted. If libewf/libaff have a big combined memory footprint in such cases, this can be a major issue for this approach.
:::: [[User:Joachim Metz|Joachim]] this layer should support multi threaded decompression of compressed image types, this speeds up IO
* [[User:Joachim Metz|Joachim]] volume/partition aware layer (what about carving unpartioned space)
* File system aware layer. This could be or make use of tsk-cp.
** By default, files are not carved. (clarify: only identified? [[User:RB|RB]]; I guess that it operates like [[Selective file dumper]] [[User:.FUF|.FUF]] 07:00, 29 October 2008 (UTC)). Alternatively, the tool could use libcarvpath and output carvpaths or create a directory with symlinks to carvpaths that point into a carvfs mountpoint [[User:Capibara|Rob J Meijer]].
* Plug-in architecture for identification/validation.
** [[User:Joachim Metz|Joachim]] support for multiple types of validators
*** dedicated validator
*** validator based on file library (i.e. we could specify/implement a file structure API for these)
*** configuration based validator (Can handle config files,like Revit07, to enter different file formats used by the carver.)

[[User:Joachim Metz|Joachim]]
Moderator: Could we limit the requirements for prototype version 1 of the tool to get a working version up and running ASAP?
And keep discussing future options?

I think the following set will be large enough to handle:
Input facilities
* IO support (AFF, device, EWF, RAW and split RAW)
:: Abstraction of input format and multi threaded decompression (spin-off code out of afflib?)
* Volume/Partitions support
:: at least for DOS based layout and GPT (spin-off code out of TSK/Photorec?)
* File system support
:: VFAT/NTFS (spin-off code out of TSK/Photorec?)

Carving facilities
* File format support using plug-able validator model (use dedicated validators Photorec/Scarve and/or wrap revit07 file format as validator?)
* Content support using plug-able validator model (to handle text/mbox base64)
* File system carving support (to handle file system fragments, could be linked to file system support layer?)
* Basic fragment handling

Output facilities
* audit/analysis/debug log
* extraction of result files

==Supported File Formats==
* Ship with validators for:
[[User:Joachim Metz|Joachim]] I think we should distinguish between file format validators and content validators
** JPEG
** PNG
** GIF
** MSOLE
** ZIP
** TAR (gz/bz2)

[[User:Joachim Metz|Joachim]] For a production carver we need at least the following formats
** Grapical Images
*** JPEG (the 3 different types with JFIF/EXIF support)
*** PNG
*** GIF
*** BMP
*** TIFF
** Office documents
*** OLE2 (Word/Excell content support)
*** PDF
*** Open Office/Office 2007 (ZIP+XML)
:: Extension validation? AFAIK, MS Office 2007 [[DOCX]] format uses plain ZIP (or not?), and carved files will (or not?) have .zip extension instead of DOCX. Is there any way to fix this (may be using the file list in zip)? [[User:.FUF|.FUF]] 20:25, 31 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] Addition: Office 2007 also has a binary file format which is also a ZIP-ed data

==Archive Files==
** Archive files
*** ZIP
*** 7z
*** gzip
*** bzip2
*** tar
*** RAR
** E-mail files
*** PFF (PST/OST)
*** MBOX (text based format, base64 content support)
** Audio/Video files
*** MPEG
*** MP2/MP3
*** AVI
*** ASF/WMV
*** QuickTime
*** MKV
** Printer spool files
*** EMF (if I remember correctly)
** Internet history files
*** index.dat
*** firefox (sqllite 3)
** Other files
*** thumbs.db
*** pagefile?

==Carving Strategies==
[[User:Joachim Metz|Joachim]] Note to moderator could this section be merged with the carving algorithm section?

* Simple fragment recovery carving using gap carving.
** [[User:Joachim Metz|Joachim]] have hook in for more advanced fragment recovery?
* Recovering of individual ZIP sections and JPEG icons that are not sector aligned.
** [[User:Joachim Metz|Joachim]] I would propose a generic fragment detection and recovery
* Autonomous operation (some mode of operation should be completely non-interactive, requiring no human intervention to complete [[User:RB|RB]])
** [[User:Joachim Metz|Joachim]] as much as possible, but allow to be overwritten by user
* [[User:Joachim Metz|Joachim]] When the tool output files the filenames should contain the offset in the input data (in hexadecimal?)
:: [[User:Mark Stam|Mark]] I really like the fact carved files are named after the physical or logical sector in which the file is found (photorec)
:::: [[User:Joachim Metz|Joachim]] This naming schema might cause duplicate name problem for extracting embedded files and extracting files from non sector aligned file systems.
* [[User:Joachim Metz|Joachim]] Should the tool allow to export embedded files?
* [[User:Joachim Metz|Joachim]] Should the tool allow to export fragments separately?
* [[User:Mark Stam|Mark]] I personally use photorec often for carving files in the whole volume (not only unallocated clusters), so I can store information about all potential interesting files in MySQL
:: [[User:Joachim Metz|Joachim]] interesting, Bas Kloet and me have been discussing to use information about allocated files in the recovery process, i.e. recovered fragments could be part of allocated files. Do we want to be able to extract them? Or could we rebuild the file from the fragments and the allocated files.
* [[User:Mark Stam|Mark]] It would also be nice if the files can be hashed immediately (MD5) so looking for them in other tools (for example Encase) is a snap

==Performance Requirements==
* Tested on 500GB-sized images. Should be able to carve a 500GB image in roughly 50% longer than it takes to read the image.
** Perhaps allocate a percentage budget per-validator (i.e. each validator adds N% to the carving time) [[User:RB|RB]]
** [[User:Joachim Metz|Joachim]] have multiple carving phases for precision/speed trade off?
* Parallelizable
** [[User:Joachim Metz|Joachim]] tunable for different architectures
* Configuration:
** Capability to parse some existing carvers' configuration files, either on-the-fly or as a one-way converter.
** Disengage internal configuration structure from configuration files, create parsers that present the expected structure
** [[User:Joachim Metz|Joachim]] The validator should deal with the file structure the carving algorithm should not know anything about the file structure (as in revit07 design)
** Either extend Scalpel/Foremost syntaxes for extended features or use a tertiary syntax ([[User:Joachim Metz|Joachim]] I would prefer a derivative of the revit07 configuration syntax which already has encountered some problems of dealing with defining file structure in a configuration file)

==Output==
* Can output audit.txt file.
* [[User:Joachim Metz|Joachim]] Can output database with offset analysis values i.e. for visualization tooling
* [[User:Joachim Metz|Joachim]] Can output debug log for debugging the algorithm/validation
* Easy integration into ascription software.
:: [[User:Joachim Metz|Joachim]] I'm no native speaker what do you mean with "ascription software"?
::: I think this was another non-native requesting easy scriptability. [[User:RB|RB]] 14:20, 31 October 2008 (UTC)
:::: [[User:Joachim Metz|Joachim]] that makes sense ;-)
::::: Incorrect. Ascription software is software that determines who the owner of a file is. [[User:Simsong|Simsong]] 06:36, 3 November 2008 (UTC)

= Ideas =
* Use as much TSK if possible. Don't carry your own FS implementation the way photorec does.
:: [[User:Joachim Metz|Joachim]] using TSK as much as possible would not allow to add your own file system support (i.e. mobile phones, memory structures, cap files) I would propose wrapping TSK and using it as much as possible but allow to integrate own FS implementations.
* Extracting/carving data from [[Thumbs.db]]? I've used [[foremost]] for it with some success. [[Vinetto]] has some critical bugs :( [[User:.FUF|.FUF]] 19:18, 28 October 2008 (UTC)

==Recursive Carving==
[[User:Joachim Metz|Joachim]] do we want to support (let's call it) 'recursive in file carving' (for now) this is different from embedded files because there is a file system structure in the file and not just another file structure
* Is it just me, or do a lot of the above (and below) ideas somewhat skirt around the fact that many of us want recursive carving? Can we bend back to that instead of discussing object particulars? I think this can be distilled down to three requirements:
** Simple recursion: once an object is identified, have the ability to re-carve it for internal structures
** Directed recursion: the carver should be able to be directed at arbitrary blobs and told to carve it as a specified type. This allows programmatically more simple methods of dealing with unidentifiably compressed or encrypted data. Or filesystem fragments.
** Export: the ability to export an object (recognized or not) for later or external "recursion". Should go without saying for a carver, but...
:--[[User:RB|RB]] 18:45, 2 November 2008 (UTC)
:: [[User:Simsong|Simsong]] 06:30, 3 November 2008 (UTC) pyflag already does recursive carving. Are we just going to reimplement pyflag as a single executable?

==Library Dependencies==
[[User:Capibara|Rob J Meijer]] :
* Use libcarvpath whenever possible and by default to avoid high storage requirements.
:: [[User:Joachim Metz|Joachim]] For easy deployment I would not opt for making an integral part of the tool solely dependant on a single external library or the library must be integrated in the package
::[[User:Capibara|Rob J Meijer]] Integrating libraries (libtsk,libaff.libewf,libcarvpath etc) is bad practice, autotools are your friend IMO.
:: [[User:Joachim Metz|Joachim]] I'm not talking about integrating (shared) libraries. I'm talking about that an integral part of a tool should be part of it's package. Why can't the tool package contain shared or static libraries for local use? A far worse thing to do is to have a large set of dependencies and making the tool difficult to install for most users. The tool package should contain the most necessary code. afflib/libewf support could be detected by the autotools a neat separation of functionality.
::: From a packager's standpoint, [[User:Joachim Metz|Joachim]]'s other libraries do a really good job of this, carrying around what they need but using a system-global version if available. [[User:RB|RB]]
* libtsk
* libaff ? : possibly the discussion in the requirements section should move to his section.
* libewf ? : possibly the discussion in the requirements section should move to his section.
* posix ? : Can we depend especially on the availability of UNIX domain sockets and the possibility to use msg_accrights for passing opn file handles as ocaps?

==Filesystem Detection==
* Dont stop with filesystem detection after the first match. Often if a partition is reused with a new FS and is not all that full yet, much of the old FS can still be valid. I have seen this with ext2/fat. The fact that you have identified a valid FS on a partition doesn't mean there isn't an(almost) valid second FS that would yield additional files. Identifying doubly allocated space might in some cases also be relevant.
:: [[User:Joachim Metz|Joachim]] What your saying is that dealing with file system fragments should be part of the carving algorithm
* Allow use where filesystem based carving is done by other tool, and the tool is used as second stage on (sets of) unallocated block (pseudo) files and/or non FS partition (pseudo) files.
:: [[User:Joachim Metz|Joachim]] I would not opt for this. The tool would be dependent on other tools and their data format, which makes the tool difficult to maintain. I would opt to integrate the functionality of having multiple recovery phases (stages) and allow the tooling to run the phases after one and other or separately.
::[[User:Capibara|Rob J Meijer]] More generically, I feel a way should exist to communicate the 'left overs' a previous (non open, for example LE-only) tool left.
:: [[User:Joachim Metz|Joachim]] I guess if the tool is designed to handle multiple phases it should store its data somewhere. So it should be possible to convert results of such non open tooling to the format required. However I would opt to design the recovery functionality of these non-open tools into open tools. And not to limit ourselves making translators due to the design of these non-open tools.
* Ability to be used as a library instead of a tool. Ability to access metadata true library, and thus the ability to set metadata from the carving modules. This would be extremely usefull for integrating the project into a framework like ocfa.
:: [[User:Joachim Metz|Joachim]] I guess most of the code could be integrated into libraries, but I would not opt integrating tool functionality into a library
* A wild idea that I hope at least one person will have a liking for: It might be very interesting to look at the possibilities of using a multi process style of module support and combine it with a least authority design. On platforms that support AppArmor (or similar) and uid based firewall rules, this could make for the first true POLA (principle of least authority) based forensic tool ever. POLA based forensics tools should make for a strong integrity guard against many anti forensics. Alternatively we could look at integrating a capability secure language (E?) for implementation of at least validation modules. I don't expect this idea to make it, but mentioning it I hope might spark off less strong alternatives that at least partially address the integrity + anti-forensics problem. If we can in some way introduce POLA to a wider forensics public, other tools might also pick up on it what would be great.
:: [[User:Joachim Metz|Joachim]] Could you give an example of how you see this in action?
::::[[User:Capibara|Rob J Meijer]] I see two layers where using POLA could be applied. The best one would require one of the folowing as prerequisites:
::::* The libaff/libewf layer is moved to a fuse implementation (for example carvfs).
::::* Libewf/Libaff are updated to accept opened filhandles instead of demanding to open their own files.
::::If one of these is fulfilled, than the tool running as some user can just have the simple task of opening the image files, starting up the 'real' tool and handing over the appropriate file handles. If the real tool runs with a restrictive AppArmor profile, and is started suid to a tool specific user that also has its own iptables uid based filter, than the real tool will run with least authority.
:::: A second alternative, if neither of the first prerequisite could not be bet, would be to run the modules as confined processes and have a non confined process run as proxy for the first.
:::: A third probably far fetched alternative would be to embed an object capability language in the tool and make the module interface thus that modules are to be written in this ocap language.
::::A 4th alternative might include minorfs or plash, but I havn't geven those sufficient thinking hours yet.

* [[User:Mark Stam|Mark]] I think it would be very handy to have a CSV, TSV, XML or other delimited output (log)file with information about carved files. This output file can then be stored in a database or Excel sheet (report function)

== Format syntax specification ==
* Carving data structures. For example, extract all TCP headers from image by defining TCP header structure and some fields (e.g. source port > 1024, dest port = 80). This will extract all data matching the pattern and write a file with other fields. Another example is carving INFO2 structures and URL activity records from index.dat [[User:.FUF|.FUF]] 20:51, 28 October 2008 (UTC)
** This has the opportunity to be extended to the concept of "point at blob FOO and interpret it as BAR"
.FUF added:
The main idea is to allow users to define structures, for example (in pascal-like form):

<pre>
Field1: Byte = 123;
SomeTextLength: DWORD;
SomeText: string[SomeTextLength];
Field4: Char = 'r';
...
</pre>

This will produce something like this:
<pre>
Field1 = 123
SomeTextLength = 5
SomeText = 'abcd1'
Field4 = 'r'
</pre>

(In text or raw forms.)

Opinions?

Opinion: Simple pattern identification like that may not suffice, I think Simson's original intent was not only to identify but to allow for validation routines (plugins, as the original wording was). As such, the format syntax would need to implement a large chunk of some programming language in order to be sufficiently flexible. [[User:RB|RB]]

[[User:Joachim Metz|Joachim]]
In my option your example is too limited. Making the revit configuration I learned you'll need a near programming language to specify some file formats.
A simple descriptive language is too limiting. I would also go for 2 bytes with endianess instead of using terminology like WORD and small integer, it's much more clear. The configuration also needs to deal with aspects like cardinality, required and optional structures.
:: This is simply data structures carving, see ideas above. Somebody (I cannot track so many changes per day) separated the original text. There is no need to count and join different structures. [[User:.FUF|.FUF]] 19:53, 31 October 2008 (UTC)
:::: [[User:Joachim Metz|Joachim]] This was probably me is the text back in it's original form?
:::: I started it by moving your Revit07 comment to the validator/plugin section in [http://www.forensicswiki.org/index.php?title=Carver_2.0_Planning_Page&diff=prev&oldid=7583 this edit], since I was still at that point thinking operational configuration for that section, not parser configurations. [[User:RB|RB]]
:::: [[User:Joachim Metz|Joachim]] I renamed the title to format syntax, clarity is important ;-)

Please take a look at the revit07 format syntax specification (configuration). It's not there yet but goes a far way. Some things currently missing:
* bitwise alignment
* handling encapsulated streams (MPEG/capture files)
* handling content based formats (MBOX)

=Caving algorithm =
[[User:Joachim Metz|Joachim]]
* should we allow for multiple carving phases (runs/stages)?
:: I opt yes (separation of concern)
* should we allow for multiple carving algorithms?
:: I opt yes, this allows testing of different approaches
* Should the algorithm try to do as much in 1 run over the input data? To reduce IO?
:: I opt that the tool should allow for multiple and single run over the input data to minimize the IO or the CPU as bottleneck
* Interaction between algorithm and validators
** does the algorithm passes data blocks to the validators?
** does a validator need to maintain a state?
** does a validator need to revert a state?
** How do we deal with embedded files and content validation? Do the validators call another validator?
* do we use the assumption that a data block can be used by a single file (with the exception of embedded/encapsulated files)?
* Revit07 allows for multiple concurrent result files states to deal with fragmentation. One has the attribute of being active (the preferred) and the other passive. Do we want/need something similar? The algorithm adds block of input data (offsets) to these result files states.
** if so what info would these result files states require (type, list of input data blocks)
* how do we deal with file system remainders?
** Can we abstract them and compare them against available file system information?
* Do we carve file systems in files?
:: I opt that at least the validator uses this information

==Caving scenarios ==
[[User:Joachim Metz|Joachim]]
* normal file (file structure, loose text based structure (more a content structure?))
* fragmented file (the file entirely exist)
* a file fragment (the file does not entirely exist)
* intertwined file
* encapsulated file (MPEG/network capture)
* embedded file (JPEG thumbnail)
* obfuscation ('encrypted' PFF) this also entails encryption and/or compression
* file system in file

=File System Awareness =
==Background: Why be File System Aware?==
Advantages of being FS aware:
* You can pick up sector allocation sizes
:: [[User:Joachim Metz|Joachim]] do you mean file system block sizes?
* Some file systems may store things off sector boundaries. (ReiserFS with tail packing)
* Increasingly file systems have compression (NTFS compression)
* Carve just the sectors that are not in allocated files.

==Tasks that would be required==

==Discussion==
:: As noted above, TSK should be utilized as much as possible, particularly the filesystem-aware portion. If we want to identify filesystems outside of its supported set, it would be more worth our time to work on implementing them there than in the carver itself. [[User:RB|RB]]

:::: I guess this tool operates like [[Selective file dumper]] and can recover files in both ways (or not?). Recovering files by using carving can recover files in situations where sleuthkit does nothing (e.g. file on NTFS was deleted using ntfs-3g, or filesystem was destroyed or just unknown). And we should build the list of filesystems supported by carver, not by TSK. [[User:.FUF|.FUF]] 07:08, 29 October 2008 (UTC)

:: This tool is still in the early planning stages (requirements discovery), hence few operational details (like precise modes of operation) have been fleshed out - those will and should come later. The justification for strictly using TSK for the filesystem-sensitive approach is simple: TSK has good filesystem APIs, and it would be foolish to create yet another standalone, incompatible implementation of filesystem(foo) when time would be better spent improving those in TSK, aiding other methods of analysis as well. This is the same reason individuals that have implemented several other carvers are participating: de-duplication of effort. [[User:RB|RB]]

:: [[User:Joachim Metz|Joachim]] A design problem might be that TSK currently is a single library operating on multiple layers (storage media IO, volume/partition analysis and file system analysis). I'm not aware how easily the parts can be used separately. But I estimate that for the carver we want to use these 3 layers differently than TSK currently does.

[[User:Joachim Metz|Joachim]] I would like to have the carver (recovery tool) also do recovery using file allocation data or remainders of file allocation data.

[[User:Joachim Metz|Joachim]]
I would go as far to ask you all to look beyond the carver as a tool and look from the perspective of the carver as part of the forensic investigation process. In my eyes certain information needed/acquired by the carver could be also very useful investigative information i.e. what part of a hard disk contains empty sectors.

=Supportive tooling=
[[User:Joachim Metz|Joachim]]
* validator (definitions) tester (detest in revit07)
* tool to make configuration based definitions
* post carving validation
* the carver needs to provide support for fuse mount of carved files (carvfs)

=Testing =
[[User:Joachim Metz|Joachim]]
* automated testing
* test data

=Validator Construction=
Options:
* Write validators in C/C++
** [[User:Joachim Metz|Joachim]] you mean dedicated validators
* Have a scripting language for writing them (python? Perl?) our own?
** [[User:Joachim Metz|Joachim]] use easy to embed programming languages i.e. Phyton or Lua
* Use existing programs (libjpeg?) as plug-in validators?
** [[User:Joachim Metz|Joachim]] define a file structure api for this

=Existing Code that we have=
[[User:Joachim Metz|Joachim]]
Please add any missing links

Documentation/Articles
* DFRWS2006/2007 carving challenge results
* DFRWS2008 paper on carving

Carvers
* DFRWS2006/2007 carving challenge results
* photorec (http://www.cgsecurity.org/wiki/PhotoRec)
* revit06 and revit07 (http://sourceforge.net/projects/revit/)
* s3/scarve

Possible file structure validator libraries
* divers existing file support libraries
* libole2 (inhouse experimental code of OLE2 support)
* libpff (alpha release for PFF (PST/OST) file support) (http://sourceforge.net/projects/libpff/)

Input support
* AFF (http://www.afflib.org/)
* EWF (http://sourceforge.net/projects/libewf/)
* TSK device & raw & split raw (http://www.sleuthkit.org/)

Volume/Partition support
* disktype (http://disktype.sourceforge.net/)
* testdisk (http://www.cgsecurity.org/wiki/TestDisk)
* TSK

File system support
* TSK
* photorec FS code
* implementations of FS in Linux/BSD

Content support

Zero storage support
* libcarvpath ( http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=210704 )
* carvfs ( http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=210954 )
* tsk-cp ( http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=267227 )
* carvfsmodewf (http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=268256 )
POLA
* joe-e (java) ( http://code.google.com/p/joe-e/ )
* Emily (ocaml) ( http://erights.org/download/emily/ )
* the E language ( http://www.erights.org/ )
* AppArmor
* iptables/ipfw
* minorfs ( http://polacanthus.net/minorfs.html )
* plash ( http://plash.beasts.org/wiki/ )

=Implementation Timeline=
# gather the available resources/ideas/wishes/needs etc. (I guess we're in this phase)
# start discussing a high level design (in terms of algorithm, facilities, information needed)
## input formats facility
## partition/volume facility
## file system facility
## file format facility
## content facility
## how to deal with fragment detection (do the validators allow for fragment detection?)
## how to deal with recombination of fragments
## do we want multiple carving phases in light of speed/precision tradeoffs
# start detailing parts of the design
## Discuss options for a grammar driven validator?
## Hard-coded plug-ins?
## Which existing code can we use?
# start building/assembling parts of the tooling for a prototype
## Implement simple file carving with validation.
## Implement gap carving
# Initial Release
# Implement the ''threaded carving'' that [[User:.FUF|.FUF]] is describing above.

[[User:Joachim Metz|Joachim]] Shouldn't multi threaded carving (MTC) not be part of the 1st version?
The MT approach makes for different design decisions
: It is virtually impossible to turn a non-MT application into an MT application .[[User:Simsong|Simsong]] 06:37, 3 November 2008 (UTC)

Carver 2.0 Planning Page

2008-11-03T15:40:12Z

Capibara: /* Requirements */

This page is for planning Carver 2.0.

Please, do not delete text (ideas) here. Use something like this:

<pre>
<s>bad idea</s>
:: good idea
</pre>

This will look like:

<s>bad idea</s>
:: good idea

= License =

BSD-3.
:: [[User:Joachim Metz|Joachim]] library based validators could require other licenses
::: Make the other libraries plug-able. If you them, you use them. [[User:Simsong|Simsong]] 06:34, 3 November 2008 (UTC)

= OS =

Linux/FreeBSD/MacOS
: Shouldn't this just match what the underlying afflib & sleuthkit cover? [[User:RB|RB]]
:: Yes, but you need to test and validate on each. Question: Do we want to support windows? [[User:Simsong|Simsong]] 21:09, 30 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] I think we would do wise to design with windows support from the start this will improve the platform independence from the start
:::: Agreed; I would even settle at first for being able to run against Cygwin. Note that I don't even own or use a copy of Windows, but the vast majority of forensic investigators do. [[User:RB|RB]] 14:01, 31 October 2008 (UTC)
:: [[User:Capibara|Rob J Meijer]] Leaning heavily on the autotools might be the way to go. I do however feel that support requirements for windows would not be essential. Being able to run from a virtual machine with the main storage mounted over cifs should however be tested and if possible tuned extensively.
:::: [[User:Joachim Metz|Joachim]] You'll need more than autotools to do native Windows support i.e. file access, UTF-16 support, wrap some basic system functions or have them available otherwise
::::::[[User:Capibara|Rob J Meijer]] That´s exactly my point, windows support as in being able to build and run on windows natively is much more trouble than its worth. Better make for a lean and mean autotools based build with little dependencies and no or little recursion, and better spent effort on a lean POLA design on POSIX based systems than on supporting building and running on non POSIX systems.

= Name tooling =

* [[User:Joachim Metz|Joachim]] A name for the tooling I propose coldcut
:: How about 'butcher'? ;) [[User:RB|RB]] 14:20, 31 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] cleaver ( scalpel on steroids ;-) )
* I would like to propose Gouge or Chisel :-) [[User:Capibara|Rob J Meijer]]

= Requirements =

[[User:Joachim Metz|Joachim]] Could we do a MoSCoW evaluation of these.
* AFF and EWF file images supported from scratch. ([[User:Joachim Metz|Joachim]] I would like to have raw/split raw and device access as well)
:: If we base our image i/o on afflib, we get all three with one interface. [[User:RB|RB]] Instead of letting the tools use afflib, better to write an afflib module for carvfs, and update the libewf module. The tool could than be oblivious of the file format. [[User:Capibara|Rob J Meijer]]
:::: [[User:Simsong|Simsong]] 06:29, 3 November 2008 (UTC) The problem with using carvfs is that this adds another dependency. Do you really want to require that people install carvfs in order to run the carver? What about having the thing ported to Windows?
:::::: [[User:Capibara|Rob J Meijer]] I would support adding one build dependency (libcarvpath) and removing two (libewf/libaff) by moving them to a layer more suited for them (carvfs) that would possibly allow some form of file handle (as cap) based POLA design. I am a proponent of making small things that do one and do one thing right, and to stack those to do what you need. In my view that would lead ideally to the following (simplified) chain:
::::::* recursive-forensic-framework (ocfa/pyflag)
::::::** The-(pola-based)-carving-tool
::::::*** The-carving-lib working on open fd's.
::::::**** libcarvpath
::::::***** carvfs (Over cifs/nfs-v4 on platforms that don't support Fuse).
::::::****** libewf
::::::****** libaff
::::::*** AppArmor (on supporting platforms)
::::::*** suid (on supporting platforms)
::::::*** iptables/ipfw (on supporting platforms)
:::::: As fow windows support, I would imagine making carvfs run over smb would come a long way, that is for as far as windows support is all that relevant.
:::::: There are two advantages to using libcarvpath and carvfs instead of libaff/libewf t this layer:
::::::* storage requirements for doing carving. Beyond what sleuthkit or alternatives provide I have seen many situations where carving was not done due to storage limitations.
::::::* File handles are like object capabilities. You can often do pretty simple POLA based implementations using file handles and something like AppArmor. POLA could IMHO be a strong weapon against the more nasty forms of anti forensics.
::::::Next to this, I would consider making different tools for different stages instead of one semi recursive one, and looking at how to integrate these tools into existing frameworks (ocfa/pyflag).
::::::Keep things simple but rigid and try to easily integrate things into existing frameworks as effectively as possible I would suggest.
::::::Please note, I am not ptoposing the lib/tool should be useless without libcarvpath, only that usage without carvfs should limit the
::::::supported image formats to raw images, and that libewf/libaff should be abstracted at the Fuse level or below and not at the tool level.
:::::::[[User:Joachim Metz|Joachim]] do you have an idea what the performance impact of this approach would be? It might be wise to do a proof of concept for this approach first.
::::::::[[User:Capibara|Rob J Meijer]] It would I think depend greatly on behavior of the carving lib/tool. Small 512 byte reads are relatively very expensive, 128kb reads have negligible impact. Here are some numbers from ntfs-3g: http://article.gmane.org/gmane.comp.file-systems.fuse.devel/6397/match=ntfs+3g+performance+ext3 that might be relevant. More relevant than performance might be library footprint. For example, using OCFA, we often would want to keep e few hundred images that total to tens of TB of projected storage size fuse mounted. If libewf/libaff have a big combined memory footprint in such cases, this can be a major issue for this approach.
:::: [[User:Joachim Metz|Joachim]] this layer should support multi threaded decompression of compressed image types, this speeds up IO
* [[User:Joachim Metz|Joachim]] volume/partition aware layer (what about carving unpartioned space)
* File system aware layer. This could be or make use of tsk-cp.
** By default, files are not carved. (clarify: only identified? [[User:RB|RB]]; I guess that it operates like [[Selective file dumper]] [[User:.FUF|.FUF]] 07:00, 29 October 2008 (UTC)). Alternatively, the tool could use libcarvpath and output carvpaths or create a directory with symlinks to carvpaths that point into a carvfs mountpoint [[User:Capibara|Rob J Meijer]].
* Plug-in architecture for identification/validation.
** [[User:Joachim Metz|Joachim]] support for multiple types of validators
*** dedicated validator
*** validator based on file library (i.e. we could specify/implement a file structure API for these)
*** configuration based validator (Can handle config files,like Revit07, to enter different file formats used by the carver.)

[[User:Joachim Metz|Joachim]]
Moderator: Could we limit the requirements for prototype version 1 of the tool to get a working version up and running ASAP?
And keep discussing future options?

I think the following set will be large enough to handle:
Input facilities
* IO support (AFF, device, EWF, RAW and split RAW)
:: Abstraction of input format and multi threaded decompression (spin-off code out of afflib?)
* Volume/Partitions support
:: at least for DOS based layout and GPT (spin-off code out of TSK/Photorec?)
* File system support
:: VFAT/NTFS (spin-off code out of TSK/Photorec?)

Carving facilities
* File format support using plug-able validator model (use dedicated validators Photorec/Scarve and/or wrap revit07 file format as validator?)
* Content support using plug-able validator model (to handle text/mbox base64)
* File system carving support (to handle file system fragments, could be linked to file system support layer?)
* Basic fragment handling

Output facilities
* audit/analysis/debug log
* extraction of result files

==Supported File Formats==
* Ship with validators for:
[[User:Joachim Metz|Joachim]] I think we should distinguish between file format validators and content validators
** JPEG
** PNG
** GIF
** MSOLE
** ZIP
** TAR (gz/bz2)

[[User:Joachim Metz|Joachim]] For a production carver we need at least the following formats
** Grapical Images
*** JPEG (the 3 different types with JFIF/EXIF support)
*** PNG
*** GIF
*** BMP
*** TIFF
** Office documents
*** OLE2 (Word/Excell content support)
*** PDF
*** Open Office/Office 2007 (ZIP+XML)
:: Extension validation? AFAIK, MS Office 2007 [[DOCX]] format uses plain ZIP (or not?), and carved files will (or not?) have .zip extension instead of DOCX. Is there any way to fix this (may be using the file list in zip)? [[User:.FUF|.FUF]] 20:25, 31 October 2008 (UTC)
:: [[User:Joachim Metz|Joachim]] Addition: Office 2007 also has a binary file format which is also a ZIP-ed data

==Archive Files==
** Archive files
*** ZIP
*** 7z
*** gzip
*** bzip2
*** tar
*** RAR
** E-mail files
*** PFF (PST/OST)
*** MBOX (text based format, base64 content support)
** Audio/Video files
*** MPEG
*** MP2/MP3
*** AVI
*** ASF/WMV
*** QuickTime
*** MKV
** Printer spool files
*** EMF (if I remember correctly)
** Internet history files
*** index.dat
*** firefox (sqllite 3)
** Other files
*** thumbs.db
*** pagefile?

==Carving Strategies==
[[User:Joachim Metz|Joachim]] Note to moderator could this section be merged with the carving algorithm section?

* Simple fragment recovery carving using gap carving.
** [[User:Joachim Metz|Joachim]] have hook in for more advanced fragment recovery?
* Recovering of individual ZIP sections and JPEG icons that are not sector aligned.
** [[User:Joachim Metz|Joachim]] I would propose a generic fragment detection and recovery
* Autonomous operation (some mode of operation should be completely non-interactive, requiring no human intervention to complete [[User:RB|RB]])
** [[User:Joachim Metz|Joachim]] as much as possible, but allow to be overwritten by user
* [[User:Joachim Metz|Joachim]] When the tool output files the filenames should contain the offset in the input data (in hexadecimal?)
:: [[User:Mark Stam|Mark]] I really like the fact carved files are named after the physical or logical sector in which the file is found (photorec)
:::: [[User:Joachim Metz|Joachim]] This naming schema might cause duplicate name problem for extracting embedded files and extracting files from non sector aligned file systems.
* [[User:Joachim Metz|Joachim]] Should the tool allow to export embedded files?
* [[User:Joachim Metz|Joachim]] Should the tool allow to export fragments separately?
* [[User:Mark Stam|Mark]] I personally use photorec often for carving files in the whole volume (not only unallocated clusters), so I can store information about all potential interesting files in MySQL
:: [[User:Joachim Metz|Joachim]] interesting, Bas Kloet and me have been discussing to use information about allocated files in the recovery process, i.e. recovered fragments could be part of allocated files. Do we want to be able to extract them? Or could we rebuild the file from the fragments and the allocated files.
* [[User:Mark Stam|Mark]] It would also be nice if the files can be hashed immediately (MD5) so looking for them in other tools (for example Encase) is a snap

==Performance Requirements==
* Tested on 500GB-sized images. Should be able to carve a 500GB image in roughly 50% longer than it takes to read the image.
** Perhaps allocate a percentage budget per-validator (i.e. each validator adds N% to the carving time) [[User:RB|RB]]
** [[User:Joachim Metz|Joachim]] have multiple carving phases for precision/speed trade off?
* Parallelizable
** [[User:Joachim Metz|Joachim]] tunable for different architectures
* Configuration:
** Capability to parse some existing carvers' configuration files, either on-the-fly or as a one-way converter.
** Disengage internal configuration structure from configuration files, create parsers that present the expected structure
** [[User:Joachim Metz|Joachim]] The validator should deal with the file structure the carving algorithm should not know anything about the file structure (as in revit07 design)
** Either extend Scalpel/Foremost syntaxes for extended features or use a tertiary syntax ([[User:Joachim Metz|Joachim]] I would prefer a derivative of the revit07 configuration syntax which already has encountered some problems of dealing with defining file structure in a configuration file)

==Output==
* Can output audit.txt file.
* [[User:Joachim Metz|Joachim]] Can output database with offset analysis values i.e. for visualization tooling
* [[User:Joachim Metz|Joachim]] Can output debug log for debugging the algorithm/validation
* Easy integration into ascription software.
:: [[User:Joachim Metz|Joachim]] I'm no native speaker what do you mean with "ascription software"?
::: I think this was another non-native requesting easy scriptability. [[User:RB|RB]] 14:20, 31 October 2008 (UTC)
:::: [[User:Joachim Metz|Joachim]] that makes sense ;-)
::::: Incorrect. Ascription software is software that determines who the owner of a file is. [[User:Simsong|Simsong]] 06:36, 3 November 2008 (UTC)

= Ideas =
* Use as much TSK if possible. Don't carry your own FS implementation the way photorec does.
:: [[User:Joachim Metz|Joachim]] using TSK as much as possible would not allow to add your own file system support (i.e. mobile phones, memory structures, cap files) I would propose wrapping TSK and using it as much as possible but allow to integrate own FS implementations.
* Extracting/carving data from [[Thumbs.db]]? I've used [[foremost]] for it with some success. [[Vinetto]] has some critical bugs :( [[User:.FUF|.FUF]] 19:18, 28 October 2008 (UTC)

==Recursive Carving==
[[User:Joachim Metz|Joachim]] do we want to support (let's call it) 'recursive in file carving' (for now) this is different from embedded files because there is a file system structure in the file and not just another file structure
* Is it just me, or do a lot of the above (and below) ideas somewhat skirt around the fact that many of us want recursive carving? Can we bend back to that instead of discussing object particulars? I think this can be distilled down to three requirements:
** Simple recursion: once an object is identified, have the ability to re-carve it for internal structures
** Directed recursion: the carver should be able to be directed at arbitrary blobs and told to carve it as a specified type. This allows programmatically more simple methods of dealing with unidentifiably compressed or encrypted data. Or filesystem fragments.
** Export: the ability to export an object (recognized or not) for later or external "recursion". Should go without saying for a carver, but...
:--[[User:RB|RB]] 18:45, 2 November 2008 (UTC)
:: [[User:Simsong|Simsong]] 06:30, 3 November 2008 (UTC) pyflag already does recursive carving. Are we just going to reimplement pyflag as a single executable?

==Library Dependencies==
[[User:Capibara|Rob J Meijer]] :
* Use libcarvpath whenever possible and by default to avoid high storage requirements.
:: [[User:Joachim Metz|Joachim]] For easy deployment I would not opt for making an integral part of the tool solely dependant on a single external library or the library must be integrated in the package
::[[User:Capibara|Rob J Meijer]] Integrating libraries (libtsk,libaff.libewf,libcarvpath etc) is bad practice, autotools are your friend IMO.
:: [[User:Joachim Metz|Joachim]] I'm not talking about integrating (shared) libraries. I'm talking about that an integral part of a tool should be part of it's package. Why can't the tool package contain shared or static libraries for local use? A far worse thing to do is to have a large set of dependencies and making the tool difficult to install for most users. The tool package should contain the most necessary code. afflib/libewf support could be detected by the autotools a neat separation of functionality.
::: From a packager's standpoint, [[User:Joachim Metz|Joachim]]'s other libraries do a really good job of this, carrying around what they need but using a system-global version if available. [[User:RB|RB]]

==Filesystem Detection==
* Dont stop with filesystem detection after the first match. Often if a partition is reused with a new FS and is not all that full yet, much of the old FS can still be valid. I have seen this with ext2/fat. The fact that you have identified a valid FS on a partition doesn't mean there isn't an(almost) valid second FS that would yield additional files. Identifying doubly allocated space might in some cases also be relevant.
:: [[User:Joachim Metz|Joachim]] What your saying is that dealing with file system fragments should be part of the carving algorithm
* Allow use where filesystem based carving is done by other tool, and the tool is used as second stage on (sets of) unallocated block (pseudo) files and/or non FS partition (pseudo) files.
:: [[User:Joachim Metz|Joachim]] I would not opt for this. The tool would be dependent on other tools and their data format, which makes the tool difficult to maintain. I would opt to integrate the functionality of having multiple recovery phases (stages) and allow the tooling to run the phases after one and other or separately.
::[[User:Capibara|Rob J Meijer]] More generically, I feel a way should exist to communicate the 'left overs' a previous (non open, for example LE-only) tool left.
:: [[User:Joachim Metz|Joachim]] I guess if the tool is designed to handle multiple phases it should store its data somewhere. So it should be possible to convert results of such non open tooling to the format required. However I would opt to design the recovery functionality of these non-open tools into open tools. And not to limit ourselves making translators due to the design of these non-open tools.
* Ability to be used as a library instead of a tool. Ability to access metadata true library, and thus the ability to set metadata from the carving modules. This would be extremely usefull for integrating the project into a framework like ocfa.
:: [[User:Joachim Metz|Joachim]] I guess most of the code could be integrated into libraries, but I would not opt integrating tool functionality into a library
* A wild idea that I hope at least one person will have a liking for: It might be very interesting to look at the possibilities of using a multi process style of module support and combine it with a least authority design. On platforms that support AppArmor (or similar) and uid based firewall rules, this could make for the first true POLA (principle of least authority) based forensic tool ever. POLA based forensics tools should make for a strong integrity guard against many anti forensics. Alternatively we could look at integrating a capability secure language (E?) for implementation of at least validation modules. I don't expect this idea to make it, but mentioning it I hope might spark off less strong alternatives that at least partially address the integrity + anti-forensics problem. If we can in some way introduce POLA to a wider forensics public, other tools might also pick up on it what would be great.
:: [[User:Joachim Metz|Joachim]] Could you give an example of how you see this in action?
::::[[User:Capibara|Rob J Meijer]] I see two layers where using POLA could be applied. The best one would require one of the folowing as prerequisites:
::::* The libaff/libewf layer is moved to a fuse implementation (for example carvfs).
::::* Libewf/Libaff are updated to accept opened filhandles instead of demanding to open their own files.
::::If one of these is fulfilled, than the tool running as some user can just have the simple task of opening the image files, starting up the 'real' tool and handing over the appropriate file handles. If the real tool runs with a restrictive AppArmor profile, and is started suid to a tool specific user that also has its own iptables uid based filter, than the real tool will run with least authority.
:::: A second alternative, if neither of the first prerequisite could not be bet, would be to run the modules as confined processes and have a non confined process run as proxy for the first.
:::: A third probably far fetched alternative would be to embed an object capability language in the tool and make the module interface thus that modules are to be written in this ocap language.
::::A 4th alternative might include minorfs or plash, but I havn't geven those sufficient thinking hours yet.

* [[User:Mark Stam|Mark]] I think it would be very handy to have a CSV, TSV, XML or other delimited output (log)file with information about carved files. This output file can then be stored in a database or Excel sheet (report function)

== Format syntax specification ==
* Carving data structures. For example, extract all TCP headers from image by defining TCP header structure and some fields (e.g. source port > 1024, dest port = 80). This will extract all data matching the pattern and write a file with other fields. Another example is carving INFO2 structures and URL activity records from index.dat [[User:.FUF|.FUF]] 20:51, 28 October 2008 (UTC)
** This has the opportunity to be extended to the concept of "point at blob FOO and interpret it as BAR"
.FUF added:
The main idea is to allow users to define structures, for example (in pascal-like form):

<pre>
Field1: Byte = 123;
SomeTextLength: DWORD;
SomeText: string[SomeTextLength];
Field4: Char = 'r';
...
</pre>

This will produce something like this:
<pre>
Field1 = 123
SomeTextLength = 5
SomeText = 'abcd1'
Field4 = 'r'
</pre>

(In text or raw forms.)

Opinions?

Opinion: Simple pattern identification like that may not suffice, I think Simson's original intent was not only to identify but to allow for validation routines (plugins, as the original wording was). As such, the format syntax would need to implement a large chunk of some programming language in order to be sufficiently flexible. [[User:RB|RB]]

[[User:Joachim Metz|Joachim]]
In my option your example is too limited. Making the revit configuration I learned you'll need a near programming language to specify some file formats.
A simple descriptive language is too limiting. I would also go for 2 bytes with endianess instead of using terminology like WORD and small integer, it's much more clear. The configuration also needs to deal with aspects like cardinality, required and optional structures.
:: This is simply data structures carving, see ideas above. Somebody (I cannot track so many changes per day) separated the original text. There is no need to count and join different structures. [[User:.FUF|.FUF]] 19:53, 31 October 2008 (UTC)
:::: [[User:Joachim Metz|Joachim]] This was probably me is the text back in it's original form?
:::: I started it by moving your Revit07 comment to the validator/plugin section in [http://www.forensicswiki.org/index.php?title=Carver_2.0_Planning_Page&diff=prev&oldid=7583 this edit], since I was still at that point thinking operational configuration for that section, not parser configurations. [[User:RB|RB]]
:::: [[User:Joachim Metz|Joachim]] I renamed the title to format syntax, clarity is important ;-)

Please take a look at the revit07 format syntax specification (configuration). It's not there yet but goes a far way. Some things currently missing:
* bitwise alignment
* handling encapsulated streams (MPEG/capture files)
* handling content based formats (MBOX)

=Caving algorithm =
[[User:Joachim Metz|Joachim]]
* should we allow for multiple carving phases (runs/stages)?
:: I opt yes (separation of concern)
* should we allow for multiple carving algorithms?
:: I opt yes, this allows testing of different approaches
* Should the algorithm try to do as much in 1 run over the input data? To reduce IO?
:: I opt that the tool should allow for multiple and single run over the input data to minimize the IO or the CPU as bottleneck
* Interaction between algorithm and validators
** does the algorithm passes data blocks to the validators?
** does a validator need to maintain a state?
** does a validator need to revert a state?
** How do we deal with embedded files and content validation? Do the validators call another validator?
* do we use the assumption that a data block can be used by a single file (with the exception of embedded/encapsulated files)?
* Revit07 allows for multiple concurrent result files states to deal with fragmentation. One has the attribute of being active (the preferred) and the other passive. Do we want/need something similar? The algorithm adds block of input data (offsets) to these result files states.
** if so what info would these result files states require (type, list of input data blocks)
* how do we deal with file system remainders?
** Can we abstract them and compare them against available file system information?
* Do we carve file systems in files?
:: I opt that at least the validator uses this information

==Caving scenarios ==
[[User:Joachim Metz|Joachim]]
* normal file (file structure, loose text based structure (more a content structure?))
* fragmented file (the file entirely exist)
* a file fragment (the file does not entirely exist)
* intertwined file
* encapsulated file (MPEG/network capture)
* embedded file (JPEG thumbnail)
* obfuscation ('encrypted' PFF) this also entails encryption and/or compression
* file system in file

=File System Awareness =
==Background: Why be File System Aware?==
Advantages of being FS aware:
* You can pick up sector allocation sizes
:: [[User:Joachim Metz|Joachim]] do you mean file system block sizes?
* Some file systems may store things off sector boundaries. (ReiserFS with tail packing)
* Increasingly file systems have compression (NTFS compression)
* Carve just the sectors that are not in allocated files.

==Tasks that would be required==

==Discussion==
:: As noted above, TSK should be utilized as much as possible, particularly the filesystem-aware portion. If we want to identify filesystems outside of its supported set, it would be more worth our time to work on implementing them there than in the carver itself. [[User:RB|RB]]

:::: I guess this tool operates like [[Selective file dumper]] and can recover files in both ways (or not?). Recovering files by using carving can recover files in situations where sleuthkit does nothing (e.g. file on NTFS was deleted using ntfs-3g, or filesystem was destroyed or just unknown). And we should build the list of filesystems supported by carver, not by TSK. [[User:.FUF|.FUF]] 07:08, 29 October 2008 (UTC)

:: This tool is still in the early planning stages (requirements discovery), hence few operational details (like precise modes of operation) have been fleshed out - those will and should come later. The justification for strictly using TSK for the filesystem-sensitive approach is simple: TSK has good filesystem APIs, and it would be foolish to create yet another standalone, incompatible implementation of filesystem(foo) when time would be better spent improving those in TSK, aiding other methods of analysis as well. This is the same reason individuals that have implemented several other carvers are participating: de-duplication of effort. [[User:RB|RB]]

:: [[User:Joachim Metz|Joachim]] A design problem might be that TSK currently is a single library operating on multiple layers (storage media IO, volume/partition analysis and file system analysis). I'm not aware how easily the parts can be used separately. But I estimate that for the carver we want to use these 3 layers differently than TSK currently does.

[[User:Joachim Metz|Joachim]] I would like to have the carver (recovery tool) also do recovery using file allocation data or remainders of file allocation data.

[[User:Joachim Metz|Joachim]]
I would go as far to ask you all to look beyond the carver as a tool and look from the perspective of the carver as part of the forensic investigation process. In my eyes certain information needed/acquired by the carver could be also very useful investigative information i.e. what part of a hard disk contains empty sectors.

=Supportive tooling=
[[User:Joachim Metz|Joachim]]
* validator (definitions) tester (detest in revit07)
* tool to make configuration based definitions
* post carving validation
* the carver needs to provide support for fuse mount of carved files (carvfs)

=Testing =
[[User:Joachim Metz|Joachim]]
* automated testing
* test data

=Validator Construction=
Options:
* Write validators in C/C++
** [[User:Joachim Metz|Joachim]] you mean dedicated validators
* Have a scripting language for writing them (python? Perl?) our own?
** [[User:Joachim Metz|Joachim]] use easy to embed programming languages i.e. Phyton or Lua
* Use existing programs (libjpeg?) as plug-in validators?
** [[User:Joachim Metz|Joachim]] define a file structure api for this

=Existing Code that we have=
[[User:Joachim Metz|Joachim]]
Please add any missing links

Documentation/Articles
* DFRWS2006/2007 carving challenge results
* DFRWS2008 paper on carving

Carvers
* DFRWS2006/2007 carving challenge results
* photorec (http://www.cgsecurity.org/wiki/PhotoRec)
* revit06 and revit07 (http://sourceforge.net/projects/revit/)
* s3/scarve

Possible file structure validator libraries
* divers existing file support libraries
* libole2 (inhouse experimental code of OLE2 support)
* libpff (alpha release for PFF (PST/OST) file support) (http://sourceforge.net/projects/libpff/)

Input support
* AFF (http://www.afflib.org/)
* EWF (http://sourceforge.net/projects/libewf/)
* TSK device & raw & split raw (http://www.sleuthkit.org/)

Volume/Partition support
* disktype (http://disktype.sourceforge.net/)
* testdisk (http://www.cgsecurity.org/wiki/TestDisk)
* TSK

File system support
* TSK
* photorec FS code
* implementations of FS in Linux/BSD

Content support

Zero storage support
* libcarvpath ( http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=210704 )
* carvfs ( http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=210954 )
* tsk-cp ( http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=267227 )
* carvfsmodewf (http://sourceforge.net/project/showfiles.php?group_id=170249&package_id=268256 )
POLA
* joe-e (java) ( http://code.google.com/p/joe-e/ )
* Emily (ocaml) ( http://erights.org/download/emily/ )
* the E language ( http://www.erights.org/ )
* AppArmor
* iptables/ipfw
* minorfs ( http://polacanthus.net/minorfs.html )
* plash ( http://plash.beasts.org/wiki/ )

=Implementation Timeline=
# gather the available resources/ideas/wishes/needs etc. (I guess we're in this phase)
# start discussing a high level design (in terms of algorithm, facilities, information needed)
## input formats facility
## partition/volume facility
## file system facility
## file format facility
## content facility
## how to deal with fragment detection (do the validators allow for fragment detection?)
## how to deal with recombination of fragments
## do we want multiple carving phases in light of speed/precision tradeoffs
# start detailing parts of the design
## Discuss options for a grammar driven validator?
## Hard-coded plug-ins?
## Which existing code can we use?
# start building/assembling parts of the tooling for a prototype
## Implement simple file carving with validation.
## Implement gap carving
# Initial Release
# Implement the ''threaded carving'' that [[User:.FUF|.FUF]] is describing above.

[[User:Joachim Metz|Joachim]] Shouldn't multi threaded carving (MTC) not be part of the 1st version?
The MT approach makes for different design decisions
: It is virtually impossible to turn a non-MT application into an MT application .[[User:Simsong|Simsong]] 06:37, 3 November 2008 (UTC)