Difference between pages "ASR Data's Expert Witness Compression Format" and "AFF"

From ForensicsWiki
(Difference between pages)
Jump to: navigation, search
 
m
 
Line 1: Line 1:
<b>This page is intended to preserve the original material, including errors and typos. It is wiki-style formatted as closely as possible to the original.</b>
+
The '''Advanced Forensics Format''' ('''AFF''') is an extensible open format for the storage of [[disk image]]s and related forensic [[metadata]]. It was originally developed by [[Simson Garfinkel]] and [[Basis Technology]]. The last version of AFF is implemented in the [[AFFLIBv3]] library, which can be found on [https://github.com/simsong/AFFLIBv3 github].  [[AFF4]] builds upon many of the concepts developed in AFF.  AFF4 was developed by [[Michael Cohen]], Simson Garfinkel and Bradley Schatz. That version can be downloaded from [https://code.google.com/p/aff4/ Google Code].
  
Expert Witness File Format Specification
+
[[Sleuthkit]], [[Autopsy]] , [[OSFMount]], [[Xmount]], [[FTK Imager]] and [[FTK]] support the AFFv3 image format.
  
Revised: April 7, 2002
+
==AFF Background==
 +
AFF was created to be an open and extensible file format to store disk images and associated metadata. The goal was to create a disk imaging format that would not lock users into a proprietary format that may limit how he or she may analyze it. An open standard enables investigators to quickly and efficiently use their preferred tools to solve crimes, gather intelligence, and resolve security incidents. The format was implemented in AFFLIB which was distributed with an open source license.
  
Original URL: http://www.asrdata.com/SMART/whitepaper.html
+
After AFFLIB was published, [[Joachim Metz]] published [[libewf]], an open source implementation of the EnCase Expert Witness format. Later, Guidance Software modified its format to allow single disk volumes larger than 4GiB. Together these two changes significantly decreased the need for AFF and AFFLIB.
  
Redistributed with permission
+
In 2009 Cohen, Garfinkel and Schatz published an article on AFF4, a new file format that incorporated and expanded on the underlying AFF ideas. AFF4 provides for multiple data views within a single data archives and allows links between archives. As a result, AFF4 natively supports selective imaging, logical file volumes, hash-based imaging, and a variety of case-management scenarios.
  
= Introduction =
+
==AFFv3 Extensions==
  
Developed by ASR Data, the Expert Witness file format (aka E01 format aka EnCase file format) is an industry standard format for storing "forensic" images.
+
The original AFF format is a single file that contains segments with drive data and metadata. Its contents can be compressed, but it can be quite large as the data on modern hard disks often reach 100GB in size.
The format allows a user to access arbitrary offsets in the uncompressed data without requiring decompression of the entire data stream.
+
AFFv3 supported three file extensions --– AFF, AFD and AFM –-- and provided a tool to easily convert between the variations.
The specification does <b>NOT</b> provide for quantifyable assurance of integrity, it is up to the implementation to provide meaningful authentication
+
for <b>any</b> data contained in an "evidence file".
+
* Overview
+
* File Header
+
* The Section
+
* Section Types
+
** 'header' section
+
** 'volume' section
+
** 'table' section
+
** 'next' and 'done' sections
+
  
= Overview =
+
For ease of transfer, large AFF files can be broken into multiple AFD format files. The smaller AFD files can be readily moved around a FAT32 file system which limits files to 2GB or stored on DVDs, which have similar size restrictions. The AFM format stores the metadata in an AFF file, and the disk data in a separate raw file. This format allows analysis tools that support the raw format to access the data, but without losing the metadata.
The Expert Witness Compression format can store a single image in one or more segment files. Each file consists of a standard 13-byte header, followed by a series of sections. The sections are typically arranged back-to-back. A section cannot span two files.
+
  
A <i>Chunk</i> is a 32k run of data (64 standard sectors).
+
===Compression and Encryption===
All offsets are relative to the beginning of the segment file, unless otherwise noted.
+
AFF supports two compression algorithms: zlib, which is fast and reasonably efficient, and LZMA, which is slower but dramatically more efficient. zlib is the same compression algorithm used by EnCase. As a result, AFF files compressed with zlib are roughly the same size as the equivalent EnCase file. AFF files can be recompressed using the LZMA algorithm. These files are anywhere from 1/2 to 1/10th the size of the original AFF/EnCase file.
  
File Header
+
AFF2.0 supports encryption of disk images. Unlike the password implemented by EnCase, encrypted images cannot be accessed without the necessary encryption key. FTK Imager/FTK added support for this encryption  in version 3.0 and are able to create and access AFF encrypted images.
Each file begins with the following 13-byte header. (This is not to be confused with the header section, below.)
+
  
<i>Signature Part (8 bytes)...</i>
+
=== AFFLIBv3 Tools ===
{| class="wikitable"
+
|-
+
| align="left"| Bytes
+
| 3
+
| 1
+
| 1
+
| 1
+
| 1
+
| 1
+
|-
+
| Data
+
| "EVF"
+
| 0x09
+
| 0x0d
+
| 0x0a
+
| 0xff
+
| 0x00
+
|}
+
  
<i>Fields Part (5 bytes)...</i>
+
* [[aimage]]
{| class="wikitable"
+
* [[ident]]
|-
+
* [[afcat]]
| align="left"| Bytes
+
* [[afcompare]]
| 1
+
* [[afconvert]]
| 2
+
* [[affix]]
| 2
+
* [[affuse]]
|-
+
* [[afinfo]]
| Data
+
* [[afstats]]
| 0x01
+
* [[afxml]]
| 1 or higher
+
* [[afsegment]]
| 0x0000
+
|-
+
| Meaning
+
|
+
| Segment Number
+
|
+
|}
+
  
= The Section =
+
= See Also =
Every <i>section</i> begins with the same standard data, with the following layout.
+
  
{| class="wikitable"
+
* [[AFF Developers Guide]] --- A guide for programmers on how to use the AFF
|-
+
* [[AFF Development Task List]] --- Want to help with AFF? Here is a list of things that need to be done.
| align="left"| Offset:
+
| Bytes:
+
| Data:
+
| Meaning:
+
|-
+
| 0 (0x0)
+
| 16
+
| "volume", "header", etc.
+
| Section type string
+
|-
+
| 16 (0x10)
+
| 8
+
|
+
| 64-bit offset in current file to the next section
+
|-
+
| 24 (0x18)
+
| 8
+
|
+
| 64-bit byte-size of the section
+
|-
+
| 32 (0x20)
+
| 40
+
| 0x00...
+
| Padding
+
|-
+
| 72 (0x48)
+
| 4
+
|
+
| CRC of all previous setion data
+
|}
+
  
= Section Types =
+
== External Links ==
Expert Witness Compression uses the following section types: <tt>header</tt>, <tt>volume</tt>, <tt>table</tt>, <tt>next</tt>, and <tt>done</tt>. Some of these section types have unique data that begins directly after the standard section structure above.
+
* [http://www.basistech.com/digital-forensics/aff.html Basis Technology's AFF website]
 +
* [http://www.osforensics.com/tools/mount-disk-images.html OSFMount - 3rd party tool for mounting AFF disk images with a drive letter]
  
== 'header' section... ==
+
[[Category:Forensics File Formats]]
{| class="wikitable"
+
[[Category:Open Source Tools]]
|-
+
| align="left"| Offset:
+
| Bytes:
+
| Data:
+
| Meaning:
+
|-
+
| 76 (0x4c)
+
| to end of section
+
| zlib compress()'ed data
+
| Comments structure (see below)
+
|}
+
 
+
<i>Comment structure is simply a text string in the following tab- and newline-delimited format. (The data in each cell is separated by a tab character, and each row is separated by a newline character.) The first three lines are standard and must not change. The characters in the third line serve as reminders for the content of the fields in the fourth line. (The fourth line is the only line that needs to be customized.)</i>
+
 
+
{| class="wikitable"
+
|-
+
| align="left"| 1
+
|-
+
| main
+
|-
+
| c
+
| n
+
| a
+
| e
+
| t
+
| m
+
| u
+
| p
+
| r
+
|-
+
| Case Number
+
| Evidence Number
+
| Unique Description
+
| Examiner Name
+
| Notes
+
| Acquired Date
+
| System Date
+
| pwhash
+
| char
+
|}
+
 
+
* <i>Case Number, Evidence Number, Unique Description, Examiner Name, and Notes are free-form (provided they don't contain tab or newline characters).</i>
+
* <i>Acquired Date and System Date are in the form of: "2002 3 4 10 19 59" (March 4, 2002 10:19:59).</i>
+
* <i>pwhash should simply be the character '0'.</i>
+
* <i>char should be the one of these three characters: 'b', 'f', or 'n'. This represents "best", "fastest", or "no compression". Expert Witness Compression uses 'f'.</i>
+
 
+
The header section should appear in the first segment file only.
+
 
+
== 'volume' section... ==
+
{| class="wikitable"
+
|-
+
| align="left"| Offset:
+
| Bytes:
+
| Data:
+
| Meaning:
+
|-
+
| 76 (0x4c)
+
| 4
+
| 1
+
| Reserved
+
|-
+
| 80 (0x50)
+
| 4
+
|
+
| Chunk Count
+
|-
+
| 84 (0x54)
+
| 4
+
| 64
+
| Sectors per Chunk
+
|-
+
| 88 (0x58)
+
| 4
+
| 512
+
| Bytes per Sector
+
|-
+
| 92 (0x5c)
+
| 4
+
|
+
| Sector Count
+
|-
+
| 96 (0x60)
+
| 20
+
| 0x00...
+
| Reserved
+
|-
+
| 116 (0x74)
+
| 45
+
| 0x00...
+
| Padding
+
|-
+
| 161 (0xa1)
+
| 5
+
|
+
| Reserved (Signature)
+
|-
+
| 166 (0xa6)
+
| 4
+
|
+
| CRC of all previous 'volume' data, starting with offset 76
+
|}
+
 
+
The volume section should appear in the first segment file only.
+
 
+
== 'table' section... ==
+
{| class="wikitable"
+
|-
+
| align="left"| Offset:
+
| Bytes:
+
| Data:
+
| Meaning:
+
|-
+
| 76 (0x4c)
+
| 4
+
| 1
+
| Chunk Count (for this table)
+
|-
+
| 80 (0x50)
+
| 16
+
| 0x00...
+
| Padding
+
|-
+
| 96 (0x60)
+
| 4
+
|
+
| CRC of all previous 'table' data, starting with offset 76
+
|-
+
| 100 (0x64)
+
| as long as necessary
+
|
+
| Offset array (see below)
+
|-
+
| from end of offset array
+
| to end of section
+
|
+
| zlib compress()'ed data Chunks
+
|}
+
 
+
<i>The offset array is a series of back-to-back 4-byte unsigned integer values. Each entry is an offset to the start of a compressed 'Chunk'. The high bit of each value must be set! There must be one entry per Chunk.</i>
+
 
+
Each table section can hold 16375 entries. If more entries are needed, you must create multiple table sections per file.
+
 
+
== 'next' and 'done' sections... ==
+
Each file ends with a 'next' or 'done' section. If the file is the last segment in an Expert Witness compressed image, the section will be named 'done'. Otherwise, the section will be named 'next' to indicate that another segment file must be read. The "next section" pointer for these section types points to the beginning of the section itself, since it is the last section in a file. These section types have no unique data.
+

Revision as of 08:04, 8 April 2013

The Advanced Forensics Format (AFF) is an extensible open format for the storage of disk images and related forensic metadata. It was originally developed by Simson Garfinkel and Basis Technology. The last version of AFF is implemented in the AFFLIBv3 library, which can be found on github. AFF4 builds upon many of the concepts developed in AFF. AFF4 was developed by Michael Cohen, Simson Garfinkel and Bradley Schatz. That version can be downloaded from Google Code.

Sleuthkit, Autopsy , OSFMount, Xmount, FTK Imager and FTK support the AFFv3 image format.

AFF Background

AFF was created to be an open and extensible file format to store disk images and associated metadata. The goal was to create a disk imaging format that would not lock users into a proprietary format that may limit how he or she may analyze it. An open standard enables investigators to quickly and efficiently use their preferred tools to solve crimes, gather intelligence, and resolve security incidents. The format was implemented in AFFLIB which was distributed with an open source license.

After AFFLIB was published, Joachim Metz published libewf, an open source implementation of the EnCase Expert Witness format. Later, Guidance Software modified its format to allow single disk volumes larger than 4GiB. Together these two changes significantly decreased the need for AFF and AFFLIB.

In 2009 Cohen, Garfinkel and Schatz published an article on AFF4, a new file format that incorporated and expanded on the underlying AFF ideas. AFF4 provides for multiple data views within a single data archives and allows links between archives. As a result, AFF4 natively supports selective imaging, logical file volumes, hash-based imaging, and a variety of case-management scenarios.

AFFv3 Extensions

The original AFF format is a single file that contains segments with drive data and metadata. Its contents can be compressed, but it can be quite large as the data on modern hard disks often reach 100GB in size. AFFv3 supported three file extensions --– AFF, AFD and AFM –-- and provided a tool to easily convert between the variations.

For ease of transfer, large AFF files can be broken into multiple AFD format files. The smaller AFD files can be readily moved around a FAT32 file system which limits files to 2GB or stored on DVDs, which have similar size restrictions. The AFM format stores the metadata in an AFF file, and the disk data in a separate raw file. This format allows analysis tools that support the raw format to access the data, but without losing the metadata.

Compression and Encryption

AFF supports two compression algorithms: zlib, which is fast and reasonably efficient, and LZMA, which is slower but dramatically more efficient. zlib is the same compression algorithm used by EnCase. As a result, AFF files compressed with zlib are roughly the same size as the equivalent EnCase file. AFF files can be recompressed using the LZMA algorithm. These files are anywhere from 1/2 to 1/10th the size of the original AFF/EnCase file.

AFF2.0 supports encryption of disk images. Unlike the password implemented by EnCase, encrypted images cannot be accessed without the necessary encryption key. FTK Imager/FTK added support for this encryption in version 3.0 and are able to create and access AFF encrypted images.

AFFLIBv3 Tools

See Also

External Links