Difference between pages "ASR Data's Expert Witness Compression Format" and "Encase image file format"

From ForensicsWiki
(Difference between pages)
Jump to: navigation, search
(Created page with "Expert Witness File Format Specification Revised: April 7, 2002 Original URL: http://www.asrdata.com/SMART/whitepaper.html Redistributed with permission This page is intended...")
 
(History)
 
Line 1: Line 1:
Expert Witness File Format Specification
+
The Encase image file format is used by [[EnCase]] used to store various types of digital evidence e.g.
Revised: April 7, 2002
+
* disk image (physical bitstream of an acquired disk)
Original URL: http://www.asrdata.com/SMART/whitepaper.html
+
* volume image
Redistributed with permission
+
* memory
 +
* logical files
  
This page is intended to preserve the original material, including errors and typos. It is wiki-style formatted as closely as possible to the original.
+
== History ==
 +
Expert Witness (for Windows) was the original name for EnCase (dating back to 1998). More info about this can be found on the Internet Archive [http://web.archive.org/web/19980504153628/http://guidancesoftware.com/] including a demo of the original software [http://web.archive.org/web/19980504153759/http://guidancesoftware.com/data/ewsetup.exe].
  
= Introduction =
+
(presumably) the product was renamed because it intruded the Expert Wittness trademark held by ASR Data [http://www.asrdata.com/wp-content/themes/asr/pdf/ruling.pdf].
  
Developed by ASR Data, the Expert Witness file format (aka E01 format aka EnCase file format) is an industry standard format for storing "forensic" images.  
+
The Encase image file format therefore is also referred to as the Expert Witness (Compression) Format.
The format allows a user to access arbitrary offsets in the uncompressed data without requiring decompression of the entire data stream.
+
The specification does <b>NOT</b> provide for quantifyable assurance of integrity, it is up to the implementation to provide meaningful authentication
+
for <b>any</b> data contained in an "evidence file".
+
* Overview
+
* File Header
+
* The Section
+
* Section Types
+
** 'header' section
+
** 'volume' section
+
** 'table' section
+
** 'next' and 'done' sections
+
  
= Overview =
 
The Expert Witness Compression format can store a single image in one or more segment files. Each file consists of a standard 13-byte header, followed by a series of sections. The sections are typically arranged back-to-back. A section cannot span two files.
 
  
A <i>Chunk</i> is a 32k run of data (64 standard sectors).
+
Currently there are 2 versions of the format:
All offsets are relative to the beginning of the segment file, unless otherwise noted.
+
* version 1 is (reportedly) based on [[ASR Data's Expert Witness Compression Format]]
 +
* version 2 was introduced in EnCase 7, for which a format specification (at least non-encrypted Ex01) is available, but requires registration.
  
File Header
+
The libewf project indicates that the January 2012 version of the version 2 format specification, besides Lx01 not being specified, is sufficient to read non-encrypted Ex01 files.
Each file begins with the following 13-byte header. (This is not to be confused with the header section, below.)
+
  
<i>Signature Part (8 bytes)...</i>
 
{| class="wikitable"
 
|-
 
| align="left"| Bytes
 
| 3
 
| 1
 
| 1
 
| 1
 
| 1
 
| 1
 
|-
 
| Data
 
| "EVF"
 
| 0x09
 
| 0x0d
 
| 0x0a
 
| 0xff
 
| 0x00
 
|}
 
  
<i>Fields Part (5 bytes)...</i>
+
Although the format specification is not complete, at the moment Guidance Software is working on an update. This will not include:
{| class="wikitable"
+
* encrypted Ex01
|-
+
* Lx01
| align="left"| Bytes
+
| 1
+
| 2
+
| 2
+
|-
+
| Data
+
| 0x01
+
| 1 or higher
+
| 0x0000
+
|-
+
| Meaning
+
|
+
| Segment Number
+
|
+
|}
+
  
= The Section =
 
Every <i>section</i> begins with the same standard data, with the following layout.
 
  
{| class="wikitable"
+
So in contrast to other claims [http://tech.groups.yahoo.com/group/linux_forensics/message/3555] both versions of the EWF file format are:
|-
+
* proprietary
| align="left"| Offset:
+
* partially open specification
| Bytes:
+
| Data:
+
| Meaning:
+
|-
+
| 0 (0x0)
+
| 16
+
| "volume", "header", etc.
+
| Section type string
+
|-
+
| 16 (0x10)
+
| 8
+
|
+
| 64-bit offset in current file to the next section
+
|-
+
| 24 (0x18)
+
| 8
+
|
+
| 64-bit byte-size of the section
+
|-
+
| 32 (0x20)
+
| 40
+
| 0x00...
+
| Padding
+
|-
+
| 72 (0x48)
+
| 4
+
|
+
| CRC of all previous setion data
+
|}
+
  
= Section Types =
+
For more information about these definitions see: [[File formats]]
Expert Witness Compression uses the following section types: <tt>header</tt>, <tt>volume</tt>, <tt>table</tt>, <tt>next</tt>, and <tt>done</tt>. Some of these section types have unique data that begins directly after the standard section structure above.
+
  
== 'header' section... ==
+
== Version 1 ==
{| class="wikitable"
+
The media data can be stored in multiple evidence files, which are called segment files.
|-
+
Each segment file consist of multiple sections, which has a distinct section start definition containing a section type.
| align="left"| Offset:
+
Up to EnCase 5 the segment file were limited to 2 GiB, due to the internal 31-bit file offset representation. This limitation was lifted by adding a base offset value in EnCase 6.
| Bytes:
+
| Data:
+
| Meaning:
+
|-
+
| 76 (0x4c)
+
| to end of section
+
| zlib compress()'ed data
+
| Comments structure (see below)
+
|}
+
  
<i>Comment structure is simply a text string in the following tab- and newline-delimited format. (The data in each cell is separated by a tab character, and each row is separated by a newline character.) The first three lines are standard and must not change. The characters in the third line serve as reminders for the content of the fields in the fourth line. (The fourth line is the only line that needs to be customized.)</i>
 
  
{| class="wikitable"
+
EnCase allows to store the data compressed either using a fast or best level of the deflate compression method.
|-
+
EnCase 7 no longer distinguishes between fast or best compression and just provides for either uncompressed or compressed.
| align="left"| 1
+
|-
+
| main
+
|-
+
| c
+
| n
+
| a
+
| e
+
| t
+
| m
+
| u
+
| p
+
| r
+
|-
+
| Case Number
+
| Evidence Number
+
| Unique Description
+
| Examiner Name
+
| Notes
+
| Acquired Date
+
| System Date
+
| pwhash
+
| char
+
|}
+
  
* <i>Case Number, Evidence Number, Unique Description, Examiner Name, and Notes are free-form (provided they don't contain tab or newline characters).</i>
 
* <i>Acquired Date and System Date are in the form of: "2002 3 4 10 19 59" (March 4, 2002 10:19:59).</i>
 
* <i>pwhash should simply be the character '0'.</i>
 
* <i>char should be the one of these three characters: 'b', 'f', or 'n'. This represents "best", "fastest", or "no compression". Expert Witness Compression uses 'f'.</i>
 
  
The header section should appear in the first segment file only.  
+
Besides digital evidence the evidence files, or segment files, contain a header containing case information.
 +
The case information which entails date and time of acquisition, an examiner's name, notes on the acquisition, and an optional password.
 +
* In EnCase 3 the case information header is stored in the "header" section, which is defined twice within the file and contain the same information.
 +
* As of EnCase 4 an additional "header2" section was added. The "header" section now appears only once, but the new "header2" section twice.
  
== 'volume' section... ==
 
{| class="wikitable"
 
|-
 
| align="left"| Offset:
 
| Bytes:
 
| Data:
 
| Meaning:
 
|-
 
| 76 (0x4c)
 
| 4
 
| 1
 
| Reserved
 
|-
 
| 80 (0x50)
 
| 4
 
|
 
| Chunk Count
 
|-
 
| 84 (0x54)
 
| 4
 
| 64
 
| Sectors per Chunk
 
|-
 
| 88 (0x58)
 
| 4
 
| 512
 
| Bytes per Sector
 
|-
 
| 92 (0x5c)
 
| 4
 
|
 
| Sector Count
 
|-
 
| 96 (0x60)
 
| 20
 
| 0x00...
 
| Reserved
 
|-
 
| 116 (0x74)
 
| 45
 
| 0x00...
 
| Padding
 
|-
 
| 161 (0xa1)
 
| 5
 
|
 
| Reserved (Signature)
 
|-
 
| 166 (0xa6)
 
| 4
 
|
 
| CRC of all previous 'volume' data, starting with offset 76
 
|}
 
  
The volume section should appear in the first segment file only.  
+
The format adds error detection by storing the data with checksums (Adler32), for both the metadata as the data blocks, which are by default 64 x 512 byte sectors (32 KiB).
 +
As of EnCase 5 the number of sectors per block (chunk) can vary.
 +
EnCase 3F introduced an "error2" section that it uses to record the location and number of bad sector chunks. The way it handles the sections it can't read is that those areas are filled with zero.
 +
Then EnCase displays to the user the areas that could not be read when the image was acquired. The granularity of unreadable chunks appears to be 32K.
 +
As of EnCase 5 the granularity of unreadable chunks can vary.
  
== 'table' section... ==
 
{| class="wikitable"
 
|-
 
| align="left"| Offset:
 
| Bytes:
 
| Data:
 
| Meaning:
 
|-
 
| 76 (0x4c)
 
| 4
 
| 1
 
| Chunk Count (for this table)
 
|-
 
| 80 (0x50)
 
| 16
 
| 0x00...
 
| Padding
 
|-
 
| 96 (0x60)
 
| 4
 
|
 
| CRC of all previous 'table' data, starting with offset 76
 
|-
 
| 100 (0x64)
 
| as long as necessary
 
|
 
| Offset array (see below)
 
|-
 
| from end of offset array
 
| to end of section
 
|
 
| zlib compress()'ed data Chunks
 
|}
 
  
<i>The offset array is a series of back-to-back 4-byte unsigned integer values. Each entry is an offset to the start of a compressed 'Chunk'. The high bit of each value must be set! There must be one entry per Chunk.</i>
+
EnCase 3 can store a one-way hash of the data. For a bitstream it does so by calculating e.g. a MD5 hash of the original media data and adds a hash section to the last of the segment file.
 +
As of EnCase 6 the option to store a SHA1 hash was added.
  
Each table section can hold 16375 entries. If more entries are needed, you must create multiple table sections per file.
 
  
== 'next' and 'done' sections... ==
+
EnCase 5 and later have the option to store '''single files''' into the EnCase Logical Evidence File (LEF) or EWF-L01.
Each file ends with a 'next' or 'done' section. If the file is the last segment in an Expert Witness compressed image, the section will be named 'done'. Otherwise, the section will be named 'next' to indicate that another segment file must be read. The "next section" pointer for these section types points to the beginning of the section itself, since it is the last section in a file. These section types have no unique data.
+
This format changed slightly in EnCase 6 and 7.
 +
 
 +
== Version 2 ==
 +
 
 +
In EnCase 7 the EWF format was succeeded by the EnCase Evidence File Format Version 2 (EWF2-EX01 and EWF2-LX01).
 +
EWF2-EX01 is at it's lower levels a different format then EWF-E01 and provides support for:
 +
* bzip2 compression
 +
* direct encryption (AES-256) of the section data
 +
 
 +
The same features are added to the new logical evidence file format (EWF2-LX01) with the exception of encryption.
 +
The actual encryption method and corresponding key derivation are, currently, not open.
 +
 
 +
EWF2-EX01, EWF2-LX01 are not backwards compatible with previous EnCase products.
 +
 
 +
== See Also ==
 +
 
 +
* [[:File:ASR Data's Expert Witness Compression Format.pdf|ASR Data's Expert Witness Compression Format]]
 +
* [[EnCase]]
 +
 
 +
== External Links ==
 +
 
 +
* [http://www.cfreds.nist.gov/v2/Basic_Mac_Image.html Sample image in EnCase, iLook, and dd format] - From the [[Computer Forensic Reference Data Sets]] Project, the E01 sample image dates from January 2005
 +
* [http://code.google.com/p/libewf/downloads/detail?name=Expert%20Witness%20Compression%20Format%20%28EWF%29.pdf Expert Witness Compression Format (EWF)], by the [[libewf|libewf project]], March 2006
 +
* [http://encase-enterprise-blog.guidancesoftware.com/2012/01/2nd-generation-encase-evidence-file.html 2nd Generation EnCase Evidence File Technical Specification now Available], Guidance Software, January 2012
 +
* Requires registration: [http://www.guidancesoftware.com/DocumentRegistration.aspx?did=1000018246 EnCase Evidence File Format Version 2], Guidance Software, January 2012
 +
* [http://www.guidancesoftware.com/Document.aspx?did=1000019161 ex01 Example], by [[Guidance Software]], March 2012
 +
* [http://code.google.com/p/libewf/downloads/detail?name=Expert%20Witness%20Compression%20Format%202%20%28EWF2%29.pdf Expert Witness Compression Format (EWF) version 2], by the [[libewf|libewf project]], July 2012
 +
 
 +
[[Category:Forensics File Formats]]

Revision as of 07:17, 29 March 2013

The Encase image file format is used by EnCase used to store various types of digital evidence e.g.

  • disk image (physical bitstream of an acquired disk)
  • volume image
  • memory
  • logical files

History

Expert Witness (for Windows) was the original name for EnCase (dating back to 1998). More info about this can be found on the Internet Archive [1] including a demo of the original software [2].

(presumably) the product was renamed because it intruded the Expert Wittness trademark held by ASR Data [3].

The Encase image file format therefore is also referred to as the Expert Witness (Compression) Format.


Currently there are 2 versions of the format:

  • version 1 is (reportedly) based on ASR Data's Expert Witness Compression Format
  • version 2 was introduced in EnCase 7, for which a format specification (at least non-encrypted Ex01) is available, but requires registration.

The libewf project indicates that the January 2012 version of the version 2 format specification, besides Lx01 not being specified, is sufficient to read non-encrypted Ex01 files.


Although the format specification is not complete, at the moment Guidance Software is working on an update. This will not include:

  • encrypted Ex01
  • Lx01


So in contrast to other claims [4] both versions of the EWF file format are:

  • proprietary
  • partially open specification

For more information about these definitions see: File formats

Version 1

The media data can be stored in multiple evidence files, which are called segment files. Each segment file consist of multiple sections, which has a distinct section start definition containing a section type. Up to EnCase 5 the segment file were limited to 2 GiB, due to the internal 31-bit file offset representation. This limitation was lifted by adding a base offset value in EnCase 6.


EnCase allows to store the data compressed either using a fast or best level of the deflate compression method. EnCase 7 no longer distinguishes between fast or best compression and just provides for either uncompressed or compressed.


Besides digital evidence the evidence files, or segment files, contain a header containing case information. The case information which entails date and time of acquisition, an examiner's name, notes on the acquisition, and an optional password.

  • In EnCase 3 the case information header is stored in the "header" section, which is defined twice within the file and contain the same information.
  • As of EnCase 4 an additional "header2" section was added. The "header" section now appears only once, but the new "header2" section twice.


The format adds error detection by storing the data with checksums (Adler32), for both the metadata as the data blocks, which are by default 64 x 512 byte sectors (32 KiB). As of EnCase 5 the number of sectors per block (chunk) can vary. EnCase 3F introduced an "error2" section that it uses to record the location and number of bad sector chunks. The way it handles the sections it can't read is that those areas are filled with zero. Then EnCase displays to the user the areas that could not be read when the image was acquired. The granularity of unreadable chunks appears to be 32K. As of EnCase 5 the granularity of unreadable chunks can vary.


EnCase 3 can store a one-way hash of the data. For a bitstream it does so by calculating e.g. a MD5 hash of the original media data and adds a hash section to the last of the segment file. As of EnCase 6 the option to store a SHA1 hash was added.


EnCase 5 and later have the option to store single files into the EnCase Logical Evidence File (LEF) or EWF-L01. This format changed slightly in EnCase 6 and 7.

Version 2

In EnCase 7 the EWF format was succeeded by the EnCase Evidence File Format Version 2 (EWF2-EX01 and EWF2-LX01). EWF2-EX01 is at it's lower levels a different format then EWF-E01 and provides support for:

  • bzip2 compression
  • direct encryption (AES-256) of the section data

The same features are added to the new logical evidence file format (EWF2-LX01) with the exception of encryption. The actual encryption method and corresponding key derivation are, currently, not open.

EWF2-EX01, EWF2-LX01 are not backwards compatible with previous EnCase products.

See Also

External Links