Difference between pages "Tools:Memory Imaging" and "File Carving"

From ForensicsWiki
(Difference between pages)
Jump to: navigation, search
(Mac OS X)
 
m (FIle -> File)
 
Line 1: Line 1:
The [[physical memory]] of computers can be imaged and analyzed using a variety of tools. Because the procedure for accessing physical memory varies between [[operating systems]], these tools are listed by operating system. Once memory has been imaged, it is subjected to [[memory analysis]] to ascertain the state of the system, extract artifacts, and so on.
+
'''File Carving,''' or sometimes simply '''Carving,''' is the practice of searching an input for files or other kinds of objects based on content, rather than on metadata. File carving is a powerful tool for recovering files and fragments of files when directory entries are corrupt or missing, as may be the case with old files that have been deleted or when performing an analysis on damaged media. Memory carving is a useful tool for analyzing physical and virtual memory dumps when the memory structures are unknown or have been overwritten.
  
One of the most vexing problems for memory imaging is verifying that the image has been created correctly.  That is, verifying that it reflects the actual contents of memory at the time of its creation. Because the contents of memory are constantly changing on a running system, the process can be repeated but the results will never--to a high degree of probability--be the same.  Thus, repeating the acquisition and comparing the results is not a feasible means of validating correct image creation.  [[Memory analysis]] can reveal whether the image's contents are consistent with the known layout and structure of a given operating system, as well as answering other questions, but it cannot answer the question as to whether the image accurately reflects the system from which it was taken at the time it was taken.
 
  
== Memory Imaging Techniques ==
+
Most file carvers operate by looking for file headers and/or footers, and then "carving out" the blocks between these two boundaries. [[Semantic Carving]] performs carving based on an analysis of the contents of the proposed files.
  
; Crash Dumps
+
File carving should be done on a [[disk image]], rather than on the original disk.
: When configured to create a full memory dump, [[Windows]] operating systems will automatically save an image of physical memory when a bugcheck (aka blue screen or kernel panic) occurs. [[Andreas Schuster]] has a [http://computer.forensikblog.de/en/2005/10/acquisition_2_crashdump.html blog post] describing this technique.
+
; LiveKd Dumps
+
: The [[Sysinternals]] tool [http://www.microsoft.com/technet/sysinternals/SystemInformation/LiveKd.mspx LiveKd] can be used to create an image of physical memory on a live machine in crash dump format. Once livekd is started, use the command ".dump -f [output file]"
+
; Hibernation Files
+
: [[Windows]] 98, 2000, XP, 2003, and Vista support a feature called [[hibernation]] that saves the machine's state to the disk when the computer is powered off. When the machine is turned on again, the state is restored and the user can return to the exact point where they left off. The machine's state, including a compressed image of [[physical memory]], is written to the disk on the system drive, usually C:, as [[hiberfil.sys]]. This file can be parsed and decompressed to obtain the memory image. Once [[hiberfil.sys]] has been obtained, [http://sandman.msuiche.net/ Sandman] can be used to convert it to a dd image.
+
: [[Mac OS X]] very kindly creates a file called '''/var/vm/sleepimage''' on any laptop that is suspended. This file is NOT erased when the machine starts up. It is unencrypted even if the user turns on [[File Vault]] and enables Secure Virtual Memory. [http://pc-eye.blogspot.com/2008/08/live-memory-dump-on-mac-laptops.html].
+
; Firewire
+
: It is possible for [[Firewire]] or IEEE1394 devices to directly access the memory of a computer. Using this capability has been suggested as a method for acquiring memory images for forensic analysis. Unfortunately, the method is not safe enough to be widely used yet. There are some published papers and tools, listed below, but they are not yet forensically sound. These tools do not work with all Firewire controllers and on other can cause system crashes. The technology holds promise for future development, in general should be avoided for now.
+
: At [[CanSec West 05]], [[Michael Becher]], [[Maximillian Dornseif]], and [[Christian N. Klein]] discussed an [[exploit]] which uses [[DMA]] to read arbitrary memory locations of a [[firewire]]-enabled system. The [http://md.hudora.de/presentations/firewire/2005-firewire-cansecwest.pdf paper] lists more details. The exploit is run on an [http://ipodlinux.org/Main_Page iPod running Linux]. This can be used to grab screen contents.
+
: This technique has been turned into a tool that you can download from:  http://www.storm.net.nz/projects/16
+
: The [http://digitalfire.ucd.ie/?page_id=430 Goldfish] tool automates this exploit for investigators needing to analyze the memory of a Mac.
+
; Virtual Machine Imaging
+
: There are numerous popular virtual machines that are in wide use such as xen, qemu or vmware. If the memory image is for a machine running in this kind of virtual environment, there are usually two methods for obtaining a memory image. The common method is to pause/suspend/stop the system and then collect the resulting memory image file, this has the disadvantage of taking the machine offline during the suspend time. Alternatively most of these systems support live dumping of a memory image. [http://www.qemu.org Qemu ] supports the pmemsave function, [http://www.xen.org Xen] has the xm dump-core command.
+
  
== Memory Imaging Tools ==
+
File carving tools are listed on the [[Tools:Data_Recovery]] wiki page.
===x86 Hardware===
+
  
; [http://www.windowsscope.com WindowsSCOPE] CaptureGUARD PCIe card (commercial) - desktops, servers
+
Many carving programs have an option to only look at or near sector boundaries where headers are found. However, searching the entire input can find files that have been embedded into other files, such as [[JPEG]]s being embedded into [[Microsoft]] [[DOC|Word documents]]. This may be considered an advantage or a disadvantage, depending on the circumstances.
: Publicly available, supports all Windows OS; windd and other formats.
+
: CaptureGUARD Gateway performs DRAM acquisition even on locked computers
+
: Inquire at http://www.windowsscope.com.
+
  
; [http://www.windowsscope.com WindowsSCOPE] CaptureGUARD ExpressCard (commercial) - laptop applications
+
Today most file carving programs will only recover files that are contiguous on the media.  
: Publicly available, supports all Windows OS; windd and other formats.
+
: CaptureGUARD Gateway performs DRAM acquisition even on locked computers
+
: Inquire at http://www.windowsscope.com.  
+
  
; Tribble PCI Card (research project)
+
== File Carving Taxonomy==
: http://www.digital-evidence.org/papers/tribble-preprint.pdf
+
[[Simson Garfinkel]] and [[Joachim Metz]] have proposed the following file carving taxonomy:
  
; CoPilot by Komoku
+
;Carving
: Komoku was acquired by Microsoft and the card was not made publicly available.
+
:General term for extracting data (files) out of undifferentiated blocks (raw data), like "carving" a sculpture out of soap stone.  
  
; Forensic RAM Extraction Device (FRED) by BBN
+
;Block Based Carving
: Not publicly available. http://www.ir.bbn.com/~vkawadia/
+
:Any carving method (algorithm) that analyzes the input on block-by-block basis to determine if a block is part of a possible output file. This method assumes that each block can only be part of a single file (or embedded file).
  
===[[Windows]] Software===
+
;Characteristic Based Carving
There are many Windows memory acquisition tools. Most of them will not work on Windows Vista or 7, as user programs have been denied access to the ''\Device\Physicalmemory'' object starting in Windows 2003 Service Pack 1 and Windows Vista. Modern tools acquire physical memory by first installing a device driver, so administrative privileges are needed.
+
:Any carving method (algorithm) that analyzes the input on characteristic basis (for example, entropy) to determine if the input is part of a possible output file.
  
We have edited this list so that it only includes current tools:
+
;Header/Footer Carving
 +
:A method for carving files out of raw data using a distinct header (start of file marker) and footer (end of file marker).
  
; WindowsSCOPE Pro and Ultimate, available at  http://www.windowsscope.com
+
;Header/Maximum (file) size Carving
: Can capture, analyze, graph in depth physical and virtual memory codes and structures
+
:A method for carving files out of raw data using a distinct header (start of file marker) and a maximum (file) size. This approach works because many file formats (e.g. JPEG, MP3) do not care if additional junk is appended to the end of a valid file.
: Proprietary and standard formats (windd), snapshot repository, snapshot comparison
+
: All Windows OSs (Xp, Vista, 7), 32 and 64 bit supported
+
: Phantom Probe USB based fetch
+
: CaptureGUARD PCIe card and ExpressCard for hardware-assisted DRAM acquisition
+
: CaptureGUARD Gateway for hardware-assisted DRAM acquisition of locked computers
+
: launched in 2011
+
  
; WindowsSCOPE Live
+
;Header/Embedded Length Carving
: available at http://www.windowsscope.com and Android market
+
:A method for carving files out of raw data using a distinct header and a file length (size) which is embedded in the file format
: allows live memory analysis of Windows computers from Android phones and tablets
+
: launched in 2011
+
  
; winen.exe (Guidance Software - included with Encase 6.11 and higher)
+
;File structure based Carving
: included on [http://www.e-fense.com/helix/ Helix 2.0]
+
:A method for carving files out of raw data using a certain level of knowledge of the internal structure of file types. Garfinkel called this approach "Semantic Carving" in his DFRWS2006 carving challenge submission, while Metz and Mora called the approach "Deep Carving."
: http://forensiczone.blogspot.com/2008/06/winenexe-ram-imaging-tool-included-in.html
+
  
; [[Mdd]] (Memory DD) ([[ManTech]])
+
;Semantic Carving
: http://sourceforge.net/projects/mdd
+
:A method for carving files based on a linguistic analysis of the file's content. For example, a semantic carver might conclude that six blocks of french in the middle of a long HTML file written in English is a fragment left from a previous allocated file, and not from the English-language HTML file.
  
; MANDIANT Memoryze
+
;Carving with Validation
: Can capture and analyze memory. Supports reading dumps (raw/dd format) from other tools.
+
:A method for carving files out of raw data where the carved files are validated using a file type specific validator.
: http://www.mandiant.com/software/memoryze.htm
+
  
; [[Kntdd]]
+
;Fragment Recovery Carving
: http://www.gmgsystemsinc.com/knttools/
+
:A carving method in which two or more fragments are reassembled to form the original file or object. Garfinkel previously called this approach "Split Carving."
  
;[[Moonsols]]: [[DumpIt]]
+
== File Carving challenges and test images ==
: This utility is used to generate a physical memory dump of Windows machines. It works with both x86 (32-bits) and x64 (64-bits) machines.
+
: The raw memory dump is generated in the current directory, only a confirmation question is prompted before starting.
+
: Perfect to deploy the executable on USB keys, for quick incident responses needs.
+
: http://www.moonsols.com/wp-content/plugins/download-monitor/download.php?id=7
+
  
;[[HBGary]]: Fastdump and Fastdump Pro
+
[http://www.dfrws.org/2006/challenge/ File Carving Challenge] - [[Digital Forensic Research Workshop|DFRWS]] 2006
:[[Fastdump]] (free with registration) Can acquire physical memory on Windows 2000 through Windows XP 32 bit but not Windows 2003 or Vista.
+
:[[Fastdump Pro]] Can acquire physical memory on Windows 2000 through Windows 2008, all service packs.  Additionally, Fastdump Pro supports:
+
:-32 bit and 64 bit architectures
+
:-Acquisitions of greater than 4GB
+
:-Fast acquisitions through the use of larger page sizes (1024KB) but also supports a strict mode that enforces 4KB page sizes.
+
:-Process probing which allows for a more complete memory image of a process of interest.
+
:-Acquisition of the system page file during physical memory acquisition.  This allows for a more complete memory analysis.
+
  
;[[FTK Imager]]: FTK Imager
+
[http://www.dfrws.org/2007/challenge/ File Carving Challenge] - [[Digital Forensic Research Workshop|DFRWS]] 2007
:http://accessdata.com/support/adownloads#FTKImager
+
:FTK Imager can acquire live memory and paging file on 32bit and 64bit systems.
+
  
;[[OSForensics]]: OSForensics
+
[http://dftt.sourceforge.net/test6/index.html FAT Undelete Test #1] - Digital Forensics Tool Testing Image (dftt #6)
:http://www.osforensics.com/
+
:OSForensics can acquire live memory on 32bit and 64bit systems. A dump of an individual process's memory space or physical memory dump can be done. Output can be a straight dump or a Microsoft crash dump file, for use with Micrsoft's WinDbg debugger.
+
  
 +
[http://dftt.sourceforge.net/test7/index.html NTFS Undelete (and leap year) Test #1] - Digital Forensics Tool Testing Image (dftt #7)
  
===Linux===
+
[http://dftt.sourceforge.net/test11/index.html Basic Data Carving Test - fat32], Nick Mikus - Digital Forensics Tool Testing Image (dftt #11)
;[[/dev/mem]]
+
: On older Linux systems, the program [[dd]] can be used to read the contents of [[physical memory]] from the device file <tt>/dev/mem</tt>. On recent Linux systems, however, /dev/mem provides access only to a restricted range of addresses, rather than the full physical memory of a system.  On other systems it may not be available at all. Throughout the 2.6 series of the Linux kernel, the trend was to reduce direct access to memory via pseudo-device files.  See, for example, the message accompanying this patch: http://lwn.net/Articles/267427/.
+
;[[/dev/crash]]
+
:On Red Hat systems (and those running related distros such as Fedora or CentOS), the crash driver can be loaded to create pseudo-device /dev/crash for raw physical memory access (via command "modprobe crash"). This module can also be compiled for other Linux distributions with minor effort (see, for example, http://gleeda.blogspot.com/2009/08/devcrash-driver.html). When the crash driver is modified, compiled, and loaded on other systems, the resulting memory access device is not safe to image in its entirety. Care must be taken to avoid addresses that are not RAM-backed. On Linux, /proc/iomem exposes the correct address ranges to image, marked with "System RAM".
+
;[http://secondlookforensics.com Second Look: Linux Memory Forensics]
+
: This commercial memory forensics product ships with a modified version of the crash driver and a script for safely dumping memory using the original or modified driver on any given Linux system.
+
;[http://hysteria.sk/~niekt0/foriana/fmem_current.tgz fmem]
+
: fmem is kernel module that creates device /dev/fmem, similar to /dev/mem but without limitations. This device (physical RAM) can be copied using dd or other tool. Works on 2.6 Linux kernels. Under GNU GPL.
+
;[http://code.google.com/p/lime-forensics/ LiME]
+
: Linux Memory Extractor (LiME) is a Loadable Kernel Module (LKM), which allows the acquisition of volatile memory from Linux and Linux-based devices, such as those powered by Android. The tool supports dumping memory either to the file system of the device or over the network.
+
  
===Mac OS X===
+
[http://dftt.sourceforge.net/test12/index.html Basic Data Carving Test - ext2], Nick Mikus - Digital Forensics Tool Testing Image (dftt #12)
;[http://digitalfire.ucd.ie/?page_id=430 Goldfish]
+
:Goldfish is a [[Mac OS X]] live forensic tool. Its main purpose is to provide an easy to use interface to dump the system RAM of a target machine via a [[Firewire]] connection. It then automatically extracts the current user login password and any open AOL Instant Messenger conversation fragments that may be available. Please see [http://digitalfire.ucd.ie/?page_id=430] for more information.
+
;[http://cybermarshal.atc-nycorp.com/index.php/cyber-marshal-utilities/mac-memory-reader Mac Memory Reader]
+
:Mac Memory Reader is a simple command-line utility to capture the contents of physical RAM. Results are stored in a Mach-O binary file.  Mac Memory Reader is available free of charge.  It executes directly on 32- and 64-bit target machines running Mac OS X 10.4 through 10.7 and requires a PowerPC G4 or newer, or any Intel processor.
+
  
===Virtual===
+
== See also ==
; Qemu
+
* [[Tools:Data_Recovery#Carving | File Carving Tools]]
: Qemu allows you to dump the memory of a running image using pmemsave.
+
* [[File Carving Bibliography]]
: e.g. pmemsave 0 0x20000000 /tmp/dumpfile
+
* [[Carver 2.0 Planning Page]]
; Xen
+
: Xen allows you to live dump the memory of a guest domain using the dump-core command.
+
: You can list the available machines to find the host machine you care about using xm list and see the configuration.
+
: Dumping is a matter of sudo xm dump-core -L /tmp/dump-core-6 6
+
  
==See Also==
+
=Memory Carving=
* [[Windows Memory Analysis]]
+
* [[Linux Memory Analysis]]
+
 
+
== External Links ==
+
* [http://www.syngress.com/book_catalog/sample_159749156X.PDF Windows Memory Analysis (Sample Chapter)]
+
* [http://blogs.23.nu/RedTeam/0000/00/antville-5201/ RedTeam: FireWire round-up],
+
* [http://www.friendsglobal.com/papers/FireWire%20Memory%20Dump%20of%20Windows%20XP.pdf FireWire Memory Dump of a Windows XP Computer: A Forensic Approach], by [[Antonio Martin]], 2007
+
 
+
[[Category:Tools]]
+

Revision as of 15:46, 26 October 2008

File Carving, or sometimes simply Carving, is the practice of searching an input for files or other kinds of objects based on content, rather than on metadata. File carving is a powerful tool for recovering files and fragments of files when directory entries are corrupt or missing, as may be the case with old files that have been deleted or when performing an analysis on damaged media. Memory carving is a useful tool for analyzing physical and virtual memory dumps when the memory structures are unknown or have been overwritten.


Most file carvers operate by looking for file headers and/or footers, and then "carving out" the blocks between these two boundaries. Semantic Carving performs carving based on an analysis of the contents of the proposed files.

File carving should be done on a disk image, rather than on the original disk.

File carving tools are listed on the Tools:Data_Recovery wiki page.

Many carving programs have an option to only look at or near sector boundaries where headers are found. However, searching the entire input can find files that have been embedded into other files, such as JPEGs being embedded into Microsoft Word documents. This may be considered an advantage or a disadvantage, depending on the circumstances.

Today most file carving programs will only recover files that are contiguous on the media.

File Carving Taxonomy

Simson Garfinkel and Joachim Metz have proposed the following file carving taxonomy:

Carving
General term for extracting data (files) out of undifferentiated blocks (raw data), like "carving" a sculpture out of soap stone.
Block Based Carving
Any carving method (algorithm) that analyzes the input on block-by-block basis to determine if a block is part of a possible output file. This method assumes that each block can only be part of a single file (or embedded file).
Characteristic Based Carving
Any carving method (algorithm) that analyzes the input on characteristic basis (for example, entropy) to determine if the input is part of a possible output file.
Header/Footer Carving
A method for carving files out of raw data using a distinct header (start of file marker) and footer (end of file marker).
Header/Maximum (file) size Carving
A method for carving files out of raw data using a distinct header (start of file marker) and a maximum (file) size. This approach works because many file formats (e.g. JPEG, MP3) do not care if additional junk is appended to the end of a valid file.
Header/Embedded Length Carving
A method for carving files out of raw data using a distinct header and a file length (size) which is embedded in the file format
File structure based Carving
A method for carving files out of raw data using a certain level of knowledge of the internal structure of file types. Garfinkel called this approach "Semantic Carving" in his DFRWS2006 carving challenge submission, while Metz and Mora called the approach "Deep Carving."
Semantic Carving
A method for carving files based on a linguistic analysis of the file's content. For example, a semantic carver might conclude that six blocks of french in the middle of a long HTML file written in English is a fragment left from a previous allocated file, and not from the English-language HTML file.
Carving with Validation
A method for carving files out of raw data where the carved files are validated using a file type specific validator.
Fragment Recovery Carving
A carving method in which two or more fragments are reassembled to form the original file or object. Garfinkel previously called this approach "Split Carving."

File Carving challenges and test images

File Carving Challenge - DFRWS 2006

File Carving Challenge - DFRWS 2007

FAT Undelete Test #1 - Digital Forensics Tool Testing Image (dftt #6)

NTFS Undelete (and leap year) Test #1 - Digital Forensics Tool Testing Image (dftt #7)

Basic Data Carving Test - fat32, Nick Mikus - Digital Forensics Tool Testing Image (dftt #11)

Basic Data Carving Test - ext2, Nick Mikus - Digital Forensics Tool Testing Image (dftt #12)

See also

Memory Carving