Difference between pages "Encase image file format" and "ASR Data's Expert Witness Compression Format"

From ForensicsWiki
(Difference between pages)
Jump to: navigation, search
(History)
 
 
Line 1: Line 1:
The Encase image file format is used by [[EnCase]] used to store various types of digital evidence e.g.
+
Expert Witness File Format Specification
* disk image (physical bitstream of an acquired disk)
+
* volume image
+
* memory
+
* logical files
+
  
== History ==
+
Revised: April 7, 2002
Expert Witness (for Windows) was the original name for EnCase (dating back to 1998). More info about this can be found on the Internet Archive [http://web.archive.org/web/19980504153628/http://guidancesoftware.com/] including a demo of the original software [http://web.archive.org/web/19980504153759/http://guidancesoftware.com/data/ewsetup.exe].
+
  
(presumably) the product was renamed because it intruded the Expert Wittness trademark held by ASR Data [http://www.asrdata.com/wp-content/themes/asr/pdf/ruling.pdf].
+
Original URL: http://www.asrdata.com/SMART/whitepaper.html
  
The Encase image file format therefore is also referred to as the Expert Witness (Compression) Format.
+
Redistributed with permission
  
 +
This page is intended to preserve the original material, including errors and typos. It is wiki-style formatted as closely as possible to the original.
  
Currently there are 2 versions of the format:
+
= Introduction =
* version 1 is (reportedly) based on [[ASR Data's Expert Witness Compression Format]]
+
* version 2 was introduced in EnCase 7, for which a format specification (at least non-encrypted Ex01) is available, but requires registration.
+
  
The libewf project indicates that the January 2012 version of the version 2 format specification, besides Lx01 not being specified, is sufficient to read non-encrypted Ex01 files.
+
Developed by ASR Data, the Expert Witness file format (aka E01 format aka EnCase file format) is an industry standard format for storing "forensic" images.
 +
The format allows a user to access arbitrary offsets in the uncompressed data without requiring decompression of the entire data stream.
 +
The specification does <b>NOT</b> provide for quantifyable assurance of integrity, it is up to the implementation to provide meaningful authentication
 +
for <b>any</b> data contained in an "evidence file".
 +
* Overview
 +
* File Header
 +
* The Section
 +
* Section Types
 +
** 'header' section
 +
** 'volume' section
 +
** 'table' section
 +
** 'next' and 'done' sections
  
 +
= Overview =
 +
The Expert Witness Compression format can store a single image in one or more segment files. Each file consists of a standard 13-byte header, followed by a series of sections. The sections are typically arranged back-to-back. A section cannot span two files.
  
Although the format specification is not complete, at the moment Guidance Software is working on an update. This will not include:
+
A <i>Chunk</i> is a 32k run of data (64 standard sectors).
* encrypted Ex01
+
All offsets are relative to the beginning of the segment file, unless otherwise noted.
* Lx01
+
  
 +
File Header
 +
Each file begins with the following 13-byte header. (This is not to be confused with the header section, below.)
  
So in contrast to other claims [http://tech.groups.yahoo.com/group/linux_forensics/message/3555] both versions of the EWF file format are:
+
<i>Signature Part (8 bytes)...</i>
* proprietary
+
{| class="wikitable"
* partially open specification
+
|-
 +
| align="left"| Bytes
 +
| 3
 +
| 1
 +
| 1
 +
| 1
 +
| 1
 +
| 1
 +
|-
 +
| Data
 +
| "EVF"
 +
| 0x09
 +
| 0x0d
 +
| 0x0a
 +
| 0xff
 +
| 0x00
 +
|}
  
For more information about these definitions see: [[File formats]]
+
<i>Fields Part (5 bytes)...</i>
 +
{| class="wikitable"
 +
|-
 +
| align="left"| Bytes
 +
| 1
 +
| 2
 +
| 2
 +
|-
 +
| Data
 +
| 0x01
 +
| 1 or higher
 +
| 0x0000
 +
|-
 +
| Meaning
 +
|
 +
| Segment Number
 +
|
 +
|}
  
== Version 1 ==
+
= The Section =
The media data can be stored in multiple evidence files, which are called segment files.
+
Every <i>section</i> begins with the same standard data, with the following layout.
Each segment file consist of multiple sections, which has a distinct section start definition containing a section type.
+
Up to EnCase 5 the segment file were limited to 2 GiB, due to the internal 31-bit file offset representation. This limitation was lifted by adding a base offset value in EnCase 6.
+
  
 +
{| class="wikitable"
 +
|-
 +
| align="left"| Offset:
 +
| Bytes:
 +
| Data:
 +
| Meaning:
 +
|-
 +
| 0 (0x0)
 +
| 16
 +
| "volume", "header", etc.
 +
| Section type string
 +
|-
 +
| 16 (0x10)
 +
| 8
 +
|
 +
| 64-bit offset in current file to the next section
 +
|-
 +
| 24 (0x18)
 +
| 8
 +
|
 +
| 64-bit byte-size of the section
 +
|-
 +
| 32 (0x20)
 +
| 40
 +
| 0x00...
 +
| Padding
 +
|-
 +
| 72 (0x48)
 +
| 4
 +
|
 +
| CRC of all previous setion data
 +
|}
  
EnCase allows to store the data compressed either using a fast or best level of the deflate compression method.
+
= Section Types =
EnCase 7 no longer distinguishes between fast or best compression and just provides for either uncompressed or compressed.
+
Expert Witness Compression uses the following section types: <tt>header</tt>, <tt>volume</tt>, <tt>table</tt>, <tt>next</tt>, and <tt>done</tt>. Some of these section types have unique data that begins directly after the standard section structure above.
  
 +
== 'header' section... ==
 +
{| class="wikitable"
 +
|-
 +
| align="left"| Offset:
 +
| Bytes:
 +
| Data:
 +
| Meaning:
 +
|-
 +
| 76 (0x4c)
 +
| to end of section
 +
| zlib compress()'ed data
 +
| Comments structure (see below)
 +
|}
  
Besides digital evidence the evidence files, or segment files, contain a header containing case information.
+
<i>Comment structure is simply a text string in the following tab- and newline-delimited format. (The data in each cell is separated by a tab character, and each row is separated by a newline character.) The first three lines are standard and must not change. The characters in the third line serve as reminders for the content of the fields in the fourth line. (The fourth line is the only line that needs to be customized.)</i>
The case information which entails date and time of acquisition, an examiner's name, notes on the acquisition, and an optional password.
+
* In EnCase 3 the case information header is stored in the "header" section, which is defined twice within the file and contain the same information.
+
* As of EnCase 4 an additional "header2" section was added. The "header" section now appears only once, but the new "header2" section twice.
+
  
 +
{| class="wikitable"
 +
|-
 +
| align="left"| 1
 +
|-
 +
| main
 +
|-
 +
| c
 +
| n
 +
| a
 +
| e
 +
| t
 +
| m
 +
| u
 +
| p
 +
| r
 +
|-
 +
| Case Number
 +
| Evidence Number
 +
| Unique Description
 +
| Examiner Name
 +
| Notes
 +
| Acquired Date
 +
| System Date
 +
| pwhash
 +
| char
 +
|}
  
The format adds error detection by storing the data with checksums (Adler32), for both the metadata as the data blocks, which are by default 64 x 512 byte sectors (32 KiB).
+
* <i>Case Number, Evidence Number, Unique Description, Examiner Name, and Notes are free-form (provided they don't contain tab or newline characters).</i>
As of EnCase 5 the number of sectors per block (chunk) can vary.
+
* <i>Acquired Date and System Date are in the form of: "2002 3 4 10 19 59" (March 4, 2002 10:19:59).</i>
EnCase 3F introduced an "error2" section that it uses to record the location and number of bad sector chunks. The way it handles the sections it can't read is that those areas are filled with zero.
+
* <i>pwhash should simply be the character '0'.</i>
Then EnCase displays to the user the areas that could not be read when the image was acquired. The granularity of unreadable chunks appears to be 32K.
+
* <i>char should be the one of these three characters: 'b', 'f', or 'n'. This represents "best", "fastest", or "no compression". Expert Witness Compression uses 'f'.</i>
As of EnCase 5 the granularity of unreadable chunks can vary.
+
  
 +
The header section should appear in the first segment file only.
  
EnCase 3 can store a one-way hash of the data. For a bitstream it does so by calculating e.g. a MD5 hash of the original media data and adds a hash section to the last of the segment file.
+
== 'volume' section... ==
As of EnCase 6 the option to store a SHA1 hash was added.
+
{| class="wikitable"
 +
|-
 +
| align="left"| Offset:
 +
| Bytes:
 +
| Data:
 +
| Meaning:
 +
|-
 +
| 76 (0x4c)
 +
| 4
 +
| 1
 +
| Reserved
 +
|-
 +
| 80 (0x50)
 +
| 4
 +
|
 +
| Chunk Count
 +
|-
 +
| 84 (0x54)
 +
| 4
 +
| 64
 +
| Sectors per Chunk
 +
|-
 +
| 88 (0x58)
 +
| 4
 +
| 512
 +
| Bytes per Sector
 +
|-
 +
| 92 (0x5c)
 +
| 4
 +
|
 +
| Sector Count
 +
|-
 +
| 96 (0x60)
 +
| 20
 +
| 0x00...
 +
| Reserved
 +
|-
 +
| 116 (0x74)
 +
| 45
 +
| 0x00...
 +
| Padding
 +
|-
 +
| 161 (0xa1)
 +
| 5
 +
|
 +
| Reserved (Signature)
 +
|-
 +
| 166 (0xa6)
 +
| 4
 +
|
 +
| CRC of all previous 'volume' data, starting with offset 76
 +
|}
  
 +
The volume section should appear in the first segment file only.
  
EnCase 5 and later have the option to store '''single files''' into the EnCase Logical Evidence File (LEF) or EWF-L01.
+
== 'table' section... ==
This format changed slightly in EnCase 6 and 7.
+
{| class="wikitable"
 +
|-
 +
| align="left"| Offset:
 +
| Bytes:
 +
| Data:
 +
| Meaning:
 +
|-
 +
| 76 (0x4c)
 +
| 4
 +
| 1
 +
| Chunk Count (for this table)
 +
|-
 +
| 80 (0x50)
 +
| 16
 +
| 0x00...
 +
| Padding
 +
|-
 +
| 96 (0x60)
 +
| 4
 +
|
 +
| CRC of all previous 'table' data, starting with offset 76
 +
|-
 +
| 100 (0x64)
 +
| as long as necessary
 +
|
 +
| Offset array (see below)
 +
|-
 +
| from end of offset array
 +
| to end of section
 +
|
 +
| zlib compress()'ed data Chunks
 +
|}
  
== Version 2 ==
+
<i>The offset array is a series of back-to-back 4-byte unsigned integer values. Each entry is an offset to the start of a compressed 'Chunk'. The high bit of each value must be set! There must be one entry per Chunk.</i>
  
In EnCase 7 the EWF format was succeeded by the EnCase Evidence File Format Version 2 (EWF2-EX01 and EWF2-LX01).
+
Each table section can hold 16375 entries. If more entries are needed, you must create multiple table sections per file.
EWF2-EX01 is at it's lower levels a different format then EWF-E01 and provides support for:
+
* bzip2 compression
+
* direct encryption (AES-256) of the section data
+
  
The same features are added to the new logical evidence file format (EWF2-LX01) with the exception of encryption.
+
== 'next' and 'done' sections... ==
The actual encryption method and corresponding key derivation are, currently, not open.
+
Each file ends with a 'next' or 'done' section. If the file is the last segment in an Expert Witness compressed image, the section will be named 'done'. Otherwise, the section will be named 'next' to indicate that another segment file must be read. The "next section" pointer for these section types points to the beginning of the section itself, since it is the last section in a file. These section types have no unique data.
 
+
EWF2-EX01, EWF2-LX01 are not backwards compatible with previous EnCase products.
+
 
+
== See Also ==
+
 
+
* [[:File:ASR Data's Expert Witness Compression Format.pdf|ASR Data's Expert Witness Compression Format]]
+
* [[EnCase]]
+
 
+
== External Links ==
+
 
+
* [http://www.cfreds.nist.gov/v2/Basic_Mac_Image.html Sample image in EnCase, iLook, and dd format] - From the [[Computer Forensic Reference Data Sets]] Project, the E01 sample image dates from January 2005
+
* [http://code.google.com/p/libewf/downloads/detail?name=Expert%20Witness%20Compression%20Format%20%28EWF%29.pdf Expert Witness Compression Format (EWF)], by the [[libewf|libewf project]], March 2006
+
* [http://encase-enterprise-blog.guidancesoftware.com/2012/01/2nd-generation-encase-evidence-file.html 2nd Generation EnCase Evidence File Technical Specification now Available], Guidance Software, January 2012
+
* Requires registration: [http://www.guidancesoftware.com/DocumentRegistration.aspx?did=1000018246 EnCase Evidence File Format Version 2], Guidance Software, January 2012
+
* [http://www.guidancesoftware.com/Document.aspx?did=1000019161 ex01 Example], by [[Guidance Software]], March 2012
+
* [http://code.google.com/p/libewf/downloads/detail?name=Expert%20Witness%20Compression%20Format%202%20%28EWF2%29.pdf Expert Witness Compression Format (EWF) version 2], by the [[libewf|libewf project]], July 2012
+
 
+
[[Category:Forensics File Formats]]
+

Revision as of 08:18, 29 March 2013

Expert Witness File Format Specification

Revised: April 7, 2002

Original URL: http://www.asrdata.com/SMART/whitepaper.html

Redistributed with permission

This page is intended to preserve the original material, including errors and typos. It is wiki-style formatted as closely as possible to the original.

Introduction

Developed by ASR Data, the Expert Witness file format (aka E01 format aka EnCase file format) is an industry standard format for storing "forensic" images. The format allows a user to access arbitrary offsets in the uncompressed data without requiring decompression of the entire data stream. The specification does NOT provide for quantifyable assurance of integrity, it is up to the implementation to provide meaningful authentication for any data contained in an "evidence file".

  • Overview
  • File Header
  • The Section
  • Section Types
    • 'header' section
    • 'volume' section
    • 'table' section
    • 'next' and 'done' sections

Overview

The Expert Witness Compression format can store a single image in one or more segment files. Each file consists of a standard 13-byte header, followed by a series of sections. The sections are typically arranged back-to-back. A section cannot span two files.

A Chunk is a 32k run of data (64 standard sectors). All offsets are relative to the beginning of the segment file, unless otherwise noted.

File Header Each file begins with the following 13-byte header. (This is not to be confused with the header section, below.)

Signature Part (8 bytes)...

Bytes 3 1 1 1 1 1
Data "EVF" 0x09 0x0d 0x0a 0xff 0x00

Fields Part (5 bytes)...

Bytes 1 2 2
Data 0x01 1 or higher 0x0000
Meaning Segment Number

The Section

Every section begins with the same standard data, with the following layout.

Offset: Bytes: Data: Meaning:
0 (0x0) 16 "volume", "header", etc. Section type string
16 (0x10) 8 64-bit offset in current file to the next section
24 (0x18) 8 64-bit byte-size of the section
32 (0x20) 40 0x00... Padding
72 (0x48) 4 CRC of all previous setion data

Section Types

Expert Witness Compression uses the following section types: header, volume, table, next, and done. Some of these section types have unique data that begins directly after the standard section structure above.

'header' section...

Offset: Bytes: Data: Meaning:
76 (0x4c) to end of section zlib compress()'ed data Comments structure (see below)

Comment structure is simply a text string in the following tab- and newline-delimited format. (The data in each cell is separated by a tab character, and each row is separated by a newline character.) The first three lines are standard and must not change. The characters in the third line serve as reminders for the content of the fields in the fourth line. (The fourth line is the only line that needs to be customized.)

1
main
c n a e t m u p r
Case Number Evidence Number Unique Description Examiner Name Notes Acquired Date System Date pwhash char
  • Case Number, Evidence Number, Unique Description, Examiner Name, and Notes are free-form (provided they don't contain tab or newline characters).
  • Acquired Date and System Date are in the form of: "2002 3 4 10 19 59" (March 4, 2002 10:19:59).
  • pwhash should simply be the character '0'.
  • char should be the one of these three characters: 'b', 'f', or 'n'. This represents "best", "fastest", or "no compression". Expert Witness Compression uses 'f'.

The header section should appear in the first segment file only.

'volume' section...

Offset: Bytes: Data: Meaning:
76 (0x4c) 4 1 Reserved
80 (0x50) 4 Chunk Count
84 (0x54) 4 64 Sectors per Chunk
88 (0x58) 4 512 Bytes per Sector
92 (0x5c) 4 Sector Count
96 (0x60) 20 0x00... Reserved
116 (0x74) 45 0x00... Padding
161 (0xa1) 5 Reserved (Signature)
166 (0xa6) 4 CRC of all previous 'volume' data, starting with offset 76

The volume section should appear in the first segment file only.

'table' section...

Offset: Bytes: Data: Meaning:
76 (0x4c) 4 1 Chunk Count (for this table)
80 (0x50) 16 0x00... Padding
96 (0x60) 4 CRC of all previous 'table' data, starting with offset 76
100 (0x64) as long as necessary Offset array (see below)
from end of offset array to end of section zlib compress()'ed data Chunks

The offset array is a series of back-to-back 4-byte unsigned integer values. Each entry is an offset to the start of a compressed 'Chunk'. The high bit of each value must be set! There must be one entry per Chunk.

Each table section can hold 16375 entries. If more entries are needed, you must create multiple table sections per file.

'next' and 'done' sections...

Each file ends with a 'next' or 'done' section. If the file is the last segment in an Expert Witness compressed image, the section will be named 'done'. Otherwise, the section will be named 'next' to indicate that another segment file must be read. The "next section" pointer for these section types points to the beginning of the section itself, since it is the last section in a file. These section types have no unique data.