Difference between pages "Linux Memory Analysis" and "Gzip"

From ForensicsWiki
(Difference between pages)
Jump to: navigation, search
 
 
Line 1: Line 1:
The output of a [[Tools:Memory_Imaging|memory acquisition tool]] is a memory image which contains the raw physical memory of a system.  A wide variety of tools can be used to search for strings or other patterns in a memory image, but to extract higher-level information about the state of the system a memory analysis tool is required.
+
{{expand}}
  
==Linux Memory Analysis Tools==
+
== File format ==
 +
The gzip file (.gz) format consists of:
 +
* a file header
 +
* optional headers
 +
** extra fields
 +
** original file name
 +
** comment
 +
** header checksum
 +
* a body, containing a DEFLATE-compressed payload
 +
* a file footer
  
Active Open Source Projects:
+
=== File header ===
* The [https://www.volatilesystems.com/default/volatility Volatility Framework] is a collection of tools, implemented in Python, for the extraction of digital artifacts from volatile memory (RAM) samples.  See the [http://code.google.com/p/volatility/wiki/LinuxMemoryForensics LinuxMemoryForensics] page on the Volatility wiki.  (Availability/License: GNU GPL)
+
The file header is 10 bytes in size and contains:
* The [http://people.redhat.com/anderson/ Red Hat Crash Utility] is an extensible Linux kernel core dump analysis program.  Although designed as a debugging tool, it also has been utilized for memory forensics.  See, for example, the [http://volatilesystems.blogspot.com/2008/07/linux-memory-analysis-one-of-major.html 2008 DFRWS challenge write-up by AAron Walters].  (Availability/License: GNU GPL)
+
{| class="wikitable"
 +
! align="left"| Offset
 +
! Size
 +
! Value
 +
! Description
 +
|-
 +
| 0
 +
| 2
 +
| 0x1f 0x8b
 +
| Signature (or identification byte 1 and 2)
 +
|-
 +
| 2
 +
| 1
 +
|
 +
| Compression Method
 +
|-
 +
| 3
 +
| 1
 +
|
 +
| Flags
 +
|-
 +
| 4
 +
| 4
 +
|
 +
| Last modification time <br> Contains a POSIX timestamp.
 +
|-
 +
| 8
 +
| 1
 +
|
 +
| Extra flags
 +
|-
 +
| 9
 +
| 1
 +
|
 +
| Operating system <br> Value that indicates on which operating system the gzip file was created.
 +
|}
  
Commercial Products:
+
==== Compression method ====
* [[Second Look]] provides memory acquisition and analysis tools for Linux incident response and enterprise security.  Its major differentiators versus Volatility are malware detection via integrity verification of the kernel and running processes, ease of use (automatic kernel version detection, a graphical user interface, etc.), and enterprise scalability (including live analysis of remote systems via a memory access agent).  (Availability/License: commercial)
+
  
Inactive Open Source and Research Projects:
+
{| class="wikitable"
* The [http://4tphi.net/fatkit/ Forensic Analysis Toolkit (FATKit)] is a cross-platform, modular, and extensible digital investigation framework for analyzing volatile system memory.  (Publication Date: 2006; Availability/License: not available)
+
! align="left"| Value
* [http://hysteria.sk/~niekt0/foriana/ Foriana] is tool for extraction of information such as the process and modules lists from a RAM image using logical relations between OS structures.  (Availability/License: GNU GPL)
+
! Identifier
* [http://code.google.com/p/draugr/ Draugr] is a Linux memory forensics tool written in Python.  (Availability/License: GNU GPL)
+
! Description
* [http://code.google.com/p/volatilitux/ Volatilitux] is another Linux memory forensics tool written in Python.  (Availability/License: GNU GPL)
+
|-
* Idetect (Linux) http://forensic.seccure.net/ is an older implementation of Linux memory analysis.
+
| 0 - 7
 +
|
 +
| Reserved
 +
|-
 +
| 8
 +
| "deflate"
 +
| zlib compressed data
 +
|}
  
==Linux Memory Analysis Challenges==
+
==== Flags ====
  
* The [[Digital Forensic Research Workshop]] [http://dfrws.org/2008/challenge/index.shtml 2008 Forensics Challenge] focused on the development of Linux memory analysis techniques and the fusion of evidence from memory, hard disk, and network.
+
{| class="wikitable"
* [http://communaute.sstic.org/ChallengeSSTIC2010 Challenge SSTIC 2010] (French) dealt with analysis of physical memory from a mobile device running Android.
+
! align="left"| Value
* [http://www.honeynet.org/challenges/2011_7_compromised_server Challenge 7 of the Honeynet Project's Forensic Challenge 2011] included forensic analysis of a memory image from a potentially compromised Linux server.
+
! Identifier
 +
! Description
 +
|-
 +
| 0x01
 +
| FTEXT
 +
| If set the uncompressed data needs to be treated as text instead of binary data. <br> This flag hints end-of-line conversion for cross-platform text files but does not enforce it.
 +
|-
 +
| 0x02
 +
| FHCRC
 +
| The file contains a header checksum (CRC-16)
 +
|-
 +
| 0x04
 +
| FEXTRA
 +
| The file contains extra fields
 +
|-
 +
| 0x08
 +
| FNAME
 +
| The file contains an original file name string
 +
|-
 +
| 0x10
 +
| FCOMMENT
 +
| The file contains comment
 +
|-
 +
| 0x20
 +
|
 +
| Reserved
 +
|-
 +
| 0x40
 +
|
 +
| Reserved
 +
|-
 +
| 0x80
 +
|
 +
| Reserved
 +
|}
  
==Linux Memory Images==
+
<b>Note:</b> The FHCRC bit was never set by versions of gzip up to 1.2.4, even though it was documented with a different meaning in gzip 1.2.4.
  
Aside from those in the challenges referenced above, sample Linux memory images can also be found on the Second Look web site at http://secondlookforensics.com/images.html.
+
==== Extra flags ====
 +
If compression method is 8 the following extra flags can be defined:
 +
{| class="wikitable"
 +
! align="left"| Value
 +
! Identifier
 +
! Description
 +
|-
 +
| 0x02
 +
|
 +
| compressor used maximum compression, slowest algorithm
 +
|-
 +
| 0x04
 +
|
 +
| compressor used fastest algorithm
 +
|}
  
==Linux Memory Analysis Bibliography==
+
==== Operating System ====
* [http://forensic.seccure.net/pdf/mburdach_digital_forensics_of_physical_memory.pdf Digital Forensics of the Physical Memory] M. Burdach, March 2005.
+
{| class="wikitable"
* [http://www.usenix.org/events/usenix05/tech/freenix/full_papers/movall/movall.pdf Linux Physical Memory Analysis], Paul Movall, Ward Nelson, Shaun Wetzstein; Usenix, 2005.
+
! align="left"| Value
* [http://cisr.nps.edu/downloads/theses/06thesis_urrea.pdf An Analysis Of Linux RAM Forensics], J.M. Urrea, Masters Thesis, Naval Postgraduate School, 2006.
+
! Identifier
* [http://volatilesystems.blogspot.com/2008/07/linux-memory-analysis-one-of-major.html Linux Memory Forensics for DFRWS Challenge 2008 using Volatility, Crash, and PyFlag], by AAron Walters on the Volatile Systems Blog.
+
! Description
* [http://www.dfrws.org/2008/proceedings/p65-case.pdf FACE: Automated digital evidence discovery and correlation], Andrew Case, Andrew Cristina, Lodovico Marziale, Golden G. Richard, Vassil Roussev, DFRWS 2008
+
|-
* [http://esiea-recherche.eu/~desnos/papers/slidesdraugr.pdf Linux Live Memory Forensics], a presentation by Desnos Anthony describing the implementation of draugr, 2009.
+
| 0
* [http://is.cuni.cz/studium/dipl_st/index.php?doo=detail&did=48540 Forensic RAM Dump Image Analyzer] by Ivor Kollar, describing the implementation of foriana, 2009.
+
|
* [http://www.dfrws.org/2010/proceedings/2010-305.pdf Treasure and tragedy in kmem_cache mining for live forensics investigation] by Andrew Case, Lodovico Marziale, Cris Neckar, Golden G. Richard III; Digital Investigation, Volume 7, Supplement 1, The Proceedings of the Tenth Annual DFRWS Conference, August 2010.  [http://www.dfrws.org/2010/proceedings/richard2.pdf (Presentation)]
+
| FAT filesystem (MS-DOS, OS/2, NT/Win32)
* [http://secondlookforensics.com/ Second Look Web Page]
+
|-
* [http://blackhat.com/html/bh-dc-11/bh-dc-11-archives.html#Case De-Anonymizing Live CDs through Physical Memory Analysis] ([https://media.blackhat.com/bh-dc-11/Case/BlackHat_DC_2011_Case_De-Anonymizing_Live_CDs-wp.pdf Whitepaper]) ([https://media.blackhat.com/bh-dc-11/Case/BlackHat_DC_2011_Case_De-Anonymizing%20Live%20CDs-Slides.pdf Slides]) Andrew Case; Blackhat DC 2011.
+
| 1
* [http://dfsforensics.blogspot.com/2011/03/bringing-linux-support-to-volatility.html Bringing Linux Support to Volatility], Andrew Case; Digital Forensics Solutions Blog, 2011.
+
|
* [http://blackhat.com/html/bh-us-11/bh-us-11-briefings.html#Case Workshop - Linux Memory Analysis with Volatility] ([http://www.digitalforensicssolutions.com/papers/blackhat-workshop-full-presentation.pdf Slides]) Andrew Case; Blackhat Vegas 2011.
+
| Amiga
 +
|-
 +
| 2
 +
|
 +
| VMS (or OpenVMS)
 +
|-
 +
| 3
 +
|
 +
| Unix
 +
|-
 +
| 4
 +
|
 +
| VM/CMS
 +
|-
 +
| 5
 +
|
 +
| Atari TOS
 +
|-
 +
| 6
 +
|
 +
| HPFS filesystem (OS/2, NT)
 +
|-
 +
| 7
 +
|
 +
| Macintosh
 +
|-
 +
| 8
 +
|
 +
| Z-System
 +
|-
 +
| 9
 +
|
 +
| CP/M
 +
|-
 +
| 10
 +
|
 +
| TOPS-20
 +
|-
 +
| 11
 +
|
 +
| NTFS filesystem (NT)
 +
|-
 +
| 12
 +
|
 +
| QDOS
 +
|-
 +
| 13
 +
|
 +
| Acorn RISCOS
 +
|-
 +
| 255
 +
|
 +
| unknown
 +
|}
  
Volatility Mailing List Threads on Support for Linux:
+
=== Optional headers ===
* http://lists.volatilesystems.com/pipermail/vol-users/2010-January/thread.html#143
+
==== Extra fields ====
* http://lists.volatilesystems.com/pipermail/vol-dev/2010-September/thread.html#112
+
<b>TODO: add description</b>
  
[[Category:Memory Analysis]]
+
The extra field are variable of size and contains:
 +
{| class="wikitable"
 +
! align="left"| Offset
 +
! Size
 +
! Value
 +
! Description
 +
|-
 +
| 0
 +
| 2
 +
|
 +
| Extra field data size <br> Value in bytes.
 +
|-
 +
| 2
 +
| ...
 +
|
 +
| Extra field data
 +
|}
 +
 
 +
==== Original file name ====
 +
This is the original name of the file being compressed, with any directory components removed, and, if the file being compressed is on a file system with case insensitive names, forced to lower case.
 +
 
 +
Contains an ISO 8859-1 (LATIN-1) string with end-of-string character.
 +
 
 +
==== Comment ====
 +
Contains an ISO 8859-1 (LATIN-1) string with end-of-string character. Line breaks should be denoted by a single line feed character.
 +
 
 +
==== Header checksum ====
 +
The header checksum contain a CRC-16 that consists of the two least significant bytes of the CRC-32 for all bytes of the gzip header up to and not including the CRC-16.
 +
 
 +
=== File footer ===
 +
The file footer is 8 bytes in size and contains:
 +
{| class="wikitable"
 +
! align="left"| Offset
 +
! Size
 +
! Value
 +
! Description
 +
|-
 +
| 0
 +
| 4
 +
|
 +
| Checksum (CRC-32)
 +
|-
 +
| 4
 +
| 4
 +
|
 +
| Uncompressed data size <br> Value in bytes.
 +
|}
 +
 
 +
== External Links ==
 +
 
 +
* [http://www.gzip.org/format.txt The gzip file format], by the [http://www.gzip.org/ gzip project]
 +
* [http://www.gzip.org/algorithm.txt The gzip compression algorithm], by the [http://www.gzip.org/ gzip project]
 +
* [http://tools.ietf.org/html/rfc1952 RFC1952: GZIP file format specification version 4.3], by [[IETF]]
 +
* [http://en.wikipedia.org/wiki/Gzip Wikipedia: gzip]
 +
 
 +
[[Category:File Formats]]

Revision as of 03:05, 28 November 2013

Information icon.png

Please help to improve this article by expanding it.
Further information might be found on the discussion page.

File format

The gzip file (.gz) format consists of:

  • a file header
  • optional headers
    • extra fields
    • original file name
    • comment
    • header checksum
  • a body, containing a DEFLATE-compressed payload
  • a file footer

File header

The file header is 10 bytes in size and contains:

Offset Size Value Description
0 2 0x1f 0x8b Signature (or identification byte 1 and 2)
2 1 Compression Method
3 1 Flags
4 4 Last modification time
Contains a POSIX timestamp.
8 1 Extra flags
9 1 Operating system
Value that indicates on which operating system the gzip file was created.

Compression method

Value Identifier Description
0 - 7 Reserved
8 "deflate" zlib compressed data

Flags

Value Identifier Description
0x01 FTEXT If set the uncompressed data needs to be treated as text instead of binary data.
This flag hints end-of-line conversion for cross-platform text files but does not enforce it.
0x02 FHCRC The file contains a header checksum (CRC-16)
0x04 FEXTRA The file contains extra fields
0x08 FNAME The file contains an original file name string
0x10 FCOMMENT The file contains comment
0x20 Reserved
0x40 Reserved
0x80 Reserved

Note: The FHCRC bit was never set by versions of gzip up to 1.2.4, even though it was documented with a different meaning in gzip 1.2.4.

Extra flags

If compression method is 8 the following extra flags can be defined:

Value Identifier Description
0x02 compressor used maximum compression, slowest algorithm
0x04 compressor used fastest algorithm

Operating System

Value Identifier Description
0 FAT filesystem (MS-DOS, OS/2, NT/Win32)
1 Amiga
2 VMS (or OpenVMS)
3 Unix
4 VM/CMS
5 Atari TOS
6 HPFS filesystem (OS/2, NT)
7 Macintosh
8 Z-System
9 CP/M
10 TOPS-20
11 NTFS filesystem (NT)
12 QDOS
13 Acorn RISCOS
255 unknown

Optional headers

Extra fields

TODO: add description

The extra field are variable of size and contains:

Offset Size Value Description
0 2 Extra field data size
Value in bytes.
2 ... Extra field data

Original file name

This is the original name of the file being compressed, with any directory components removed, and, if the file being compressed is on a file system with case insensitive names, forced to lower case.

Contains an ISO 8859-1 (LATIN-1) string with end-of-string character.

Comment

Contains an ISO 8859-1 (LATIN-1) string with end-of-string character. Line breaks should be denoted by a single line feed character.

Header checksum

The header checksum contain a CRC-16 that consists of the two least significant bytes of the CRC-32 for all bytes of the gzip header up to and not including the CRC-16.

File footer

The file footer is 8 bytes in size and contains:

Offset Size Value Description
0 4 Checksum (CRC-32)
4 4 Uncompressed data size
Value in bytes.

External Links