Difference between pages "File Carving" and "SQLite database format"

From ForensicsWiki
(Difference between pages)
Jump to: navigation, search
m (See also)
 
(Use Cases)
 
Line 1: Line 1:
'''Carving''' is the practice of searching an input for files or other kinds of objects based on content, rather than on metadata. File carving is a powerful tool for recovering files and fragments of files when directory entries are corrupt or missing, as may be the case with old files that have been deleted or when performing an analysis on damaged media. Memory carving is a useful tool for analyzing physical and virtual memory dumps when the memory structures are unknown or have been overwritten.
+
{{expand}}
  
 +
SQLite databases are used by many programs including several forensics tools, e.g. [[Autopsy]] 3.
 +
SQLite 3 is current and older SQLite packages cannot use sqlite3 databases so use sqlite3 tools.
  
=File Carving=
+
== SQLite3 ==
  
Most file carvers operate by looking for file headers and/or footers, and then "carving out" the blocks between these two boundaries. [[Semantic Carving]] performs carving based on an analysis of the contents of the proposed files.  
+
SQLite version 3 uses a page-based storage where the pages are used for various types of data e.g. there are:
 +
* lock-byte pages
 +
* freelist pages
 +
** freelist trunk pages
 +
** freelist leaf pages
 +
* B-tree pages
 +
** table B-tree interior pages
 +
** table B-tree leaf pages
 +
** index B-tree interior pages
 +
** index B-tree leaf pages
 +
* payload overflow pages
 +
* pointer map pages
  
File carving should be done on a [[disk image]], rather than on the original disk.
+
=== Write-Ahead Log (WAL) ===
 +
The default method by which SQLite implements atomic commit and rollback is a rollback journal. In version 3.7.0 a "Write-Ahead Log" option was added.
  
File carving tools are listed on the [[Tools:Data_Recovery]] wiki page.
+
== Temporary sqlite files ==
 +
Seen in e.g.
 +
<pre>
 +
/Users/%USERNAME%/AppData/Local/Temp/etilqs_%RANDOM%
 +
</pre>
  
Many carving programs have an option to only look at or near sector boundaries where headers are found. However, searching the entire input can find files that have been embedded into other files, such as [[JPEG]]s being embedded into [[Microsoft]] [[DOC|Word documents]]. This may be considered an advantage or a disadvantage, depending on the circumstances.
+
Where "etilqs" is "sqlite" in reverse
  
Today most file carving programs will only recover files that are contiguous on the media.  
+
== Use Cases ==
 +
=== Web Browser Data ===
 +
[[Mozilla Firefox]] and [[Google Chrome]] both use SQLite version 3 databases for user data such as history, downloaded files.
  
== File Carving Taxonomy==
+
=== Mobile OS ===
[[Simson Garfinkel]] and [[Joachim Metz]] have proposed the following file carving taxonomy:
+
[[Google Android]] and [[Apple iOS]] use SQLite3 databases for many system applications. Phone data including calls, messages, and credentials are all stored in SQLite3.
  
;Carving
+
== External Links ==
:General term for extracting data (files) out of undifferentiated blocks (raw data), like "carving" a sculpture out of soap stone.  
+
* [http://sqlite.org/fileformat2.html The SQLite Database File Format], by the [[SQLite|SQLite project]]
 +
* [http://sqlite.org/wal.html Write-Ahead Logging], by the [[SQLite|SQLite project]]
 +
* [http://forensicsfromthesausagefactory.blogspot.com/2011/04/carving-sqlite-databases-from.html Carving SQLite databases from unallocated clusters], by Richard Drinkwater, April 27, 2011
 +
* [http://linuxsleuthing.blogspot.ch/2013/09/recovering-data-from-deleted-sqlite.html Recovering Data from Deleted SQLite Records: Redux], by [[John Lehr]], September 13, 2013
  
;Block Based Carving
+
== Tools ==
:Any carving method (algorithm) that analyzes the input on block-by-block basis to determine if a block is part of a possible output file. This method assumes that each block can only be part of a single file (or embedded file).
+
* [[SQLite]]
 +
* [[SQLite Forensic Reporter]]
  
;Characteristic Based Carving
+
[[Category:File Formats]]
:Any carving method (algorithm) that analyzes the input on characteristic basis (for example, entropy) to determine if the input is part of a possible output file.
+
 
+
;Header/Footer Carving
+
:A method for carving files out of raw data using a distinct header (start of file marker) and footer (end of file marker).
+
 
+
;Header/Maximum (file) size Carving
+
:A method for carving files out of raw data using a distinct header (start of file marker) and a maximum (file) size. This approach works because many file formats (e.g. JPEG, MP3) do not care if additional junk is appended to the end of a valid file.
+
 
+
;Header/Embedded Length Carving
+
:A method for carving files out of raw data using a distinct header and a file length (size) which is embedded in the file format
+
 
+
;File structure based Carving
+
:A method for carving files out of raw data using a certain level of knowledge of the internal structure of file types. Garfinkel called this approach "Semantic Carving" in his DFRWS2006 carving challenge submission, while Metz and Mora called the approach "Deep Carving."
+
 
+
;Semantic Carving
+
:A method for carving files based on a linguistic analysis of the file's content. For example, a semantic carver might conclude that six blocks of french in the middle of a long HTML file written in English is a fragment left from a previous allocated file, and not from the English-language HTML file.
+
 
+
;Carving with Validation
+
:A method for carving files out of raw data where the carved files are validated using a file type specific validator.
+
 
+
;Fragment Recovery Carving
+
:A carving method in which two or more fragments are reassembled to form the original file or object. Garfinkel previously called this approach "Split Carving."
+
 
+
== File Carving challenges and test images ==
+
 
+
[http://www.dfrws.org/2006/challenge/]
+
File Carving Challenge - [[Digital Forensic Research Workshop|DFRWS]] 2006
+
 
+
[http://dftt.sourceforge.net/test6/index.html]
+
FAT Undelete Test #1 - Digital Forensics Tool Testing Image (dftt #6)
+
 
+
[http://dftt.sourceforge.net/test7/index.html]
+
NTFS Undelete (and leap year) Test #1 - Digital Forensics Tool Testing Image (dftt #7)
+
 
+
[http://dftt.sourceforge.net/test11/index.html]
+
Basic Data Carving Test - fat32 (by Nick Mikus) - Digital Forensics Tool Testing Image (dftt #11)
+
 
+
[http://dftt.sourceforge.net/test12/index.html]
+
Basic Data Carving Test - ext2 (by Nick Mikus) - Digital Forensics Tool Testing Image (dftt #12)
+
 
+
==File Carving Bibliography==
+
 
+
Mikus, Nicholas A. "An analysis of disc carving techniques," Master's Thesis, Naval Postgraduate School. March 2005. http://handle.dtic.mil/100.2/ADA432468
+
 
+
Garfinkel, S., "Carving Contiguous and Fragmented Files with Fast Object Validation", Digital Forensics Workshop (DFRWS 2007), Pittsburgh, PA, August 2007.  http://www.simson.net/clips/academic/2007.DFRWS.pdf
+
 
+
== See also ==
+
* [[Tools:Data_Recovery#Carving | FIle Carving Tools]]
+
* [[File Carving Bibliography]]
+
 
+
=Memory Carving=
+

Revision as of 11:15, 26 June 2014

Information icon.png

Please help to improve this article by expanding it.
Further information might be found on the discussion page.

SQLite databases are used by many programs including several forensics tools, e.g. Autopsy 3. SQLite 3 is current and older SQLite packages cannot use sqlite3 databases so use sqlite3 tools.

SQLite3

SQLite version 3 uses a page-based storage where the pages are used for various types of data e.g. there are:

  • lock-byte pages
  • freelist pages
    • freelist trunk pages
    • freelist leaf pages
  • B-tree pages
    • table B-tree interior pages
    • table B-tree leaf pages
    • index B-tree interior pages
    • index B-tree leaf pages
  • payload overflow pages
  • pointer map pages

Write-Ahead Log (WAL)

The default method by which SQLite implements atomic commit and rollback is a rollback journal. In version 3.7.0 a "Write-Ahead Log" option was added.

Temporary sqlite files

Seen in e.g.

/Users/%USERNAME%/AppData/Local/Temp/etilqs_%RANDOM%

Where "etilqs" is "sqlite" in reverse

Use Cases

Web Browser Data

Mozilla Firefox and Google Chrome both use SQLite version 3 databases for user data such as history, downloaded files.

Mobile OS

Google Android and Apple iOS use SQLite3 databases for many system applications. Phone data including calls, messages, and credentials are all stored in SQLite3.

External Links

Tools