Difference between pages "Google Chrome" and "Windows SuperFetch Format"

From ForensicsWiki
(Difference between pages)
Jump to: navigation, search
(Disk Cache)
 
 
Line 1: Line 1:
Google Chrome is a [[Web Browser|web browser]] developed by Google Inc.
+
{{expand}}
  
== Configuration ==
+
SuperFetch, is a memory management scheme that enhances the least-recently accessed approach with historical information and proactive memory management. [http://technet.microsoft.com/en-us/magazine/2007.03.vistakernel.aspx]
The Google Chrome configuration can be found in the '''Preferences''' file.
+
  
On Linux
+
<b>Note that the following format specification are incomplete.</b>
<pre>
+
/home/$USER/.config/google-chrome/Default/Preferences
+
</pre>
+
  
On MacOS-X
+
== SuperFetch DB files ==
 +
The <tt>Ag*.db</tt> files are of the SuperFetch file format. E.g.
 
<pre>
 
<pre>
/Users/$USER/Library/Application Support/Google/Chrome/Default/Preferences
+
AgAppLaunch.db
 +
AgCx_SC*.db
 +
AgGlFaultHistory.db
 +
AgGlFgAppHistory.db
 +
AgGlGlobalHistory.db
 +
AgGlUAD_%SID%.db
 +
AgGlUAD_P_%SID%.db
 +
AgRobust.db
 
</pre>
 
</pre>
  
On Windows XP
+
The SuperFetch DB files can be stored in uncompressed or compressed form, where different version of Windows use different compressed forms:
<pre>
+
* Compressed SuperFetch DB - MEMO file format; Windows Vista
C:\Documents and Settings\%USERNAME%\Local Settings\Application Data\Google\Chrome\User Data\Default\Preferences
+
* Compressed SuperFetch DB - MEM0 file format; Windows  7
</pre>
+
* Compressed SuperFetch DB - MAM file format; Windows 8
  
On Windows Vista and later
+
=== Compressed SuperFetch DB - MEMO file format ===
<pre>
+
The MEM file consists of:
C:\Users\%USERNAME%\AppData\Local\Google\Chrome\User Data\Default\Preferences
+
* file header
</pre>
+
* compressed blocks
  
Or for '''Chromium'''
+
This format uses the LZNT1 compression method
  
On Linux
+
==== File header ====
<pre>
+
The file header is 84 bytes of size and consists of:
/home/$USER/.config/chromium/Default/Preferences
+
{| class="wikitable"
</pre>
+
|-
 +
! Offset
 +
! Size
 +
! Value
 +
! Description
 +
|-
 +
| 0
 +
| 4
 +
| "MEMO" (0x4d, 0x45, 0x4d, 0x4f)
 +
| Signature
 +
|-
 +
| 4
 +
| 4
 +
|
 +
| Uncompressed (total) data size
 +
|-
 +
|}
  
On MacOS-X
+
==== Compressed blocks ====
<pre>
+
The compressed block size is the chunk data size, which is part of the LZNT1 compressed data, + 2 bytes for the size of the chunk header itself.
/Users/$USER/Library/Application Support/Chromium/Default/Preferences
+
</pre>
+
  
On Windows XP
+
The uncompressed block size is 4096 (0x1000) or the remaining uncompressed data size for the last block.
<pre>
+
C:\Documents and Settings\%USERNAME%\Local Settings\Application Data\Chromium\User Data\Default\Preferences
+
</pre>
+
  
On Windows Vista and later
+
=== Compressed SuperFetch DB - MEM0 file format ===
<pre>
+
The MEM file consists of:
C:\Users\%USERNAME%\AppData\Local\Chromium\User Data\Default\Preferences
+
* file header
</pre>
+
* compressed blocks
  
=== Plugins ===
+
This format uses the LZXPRESS Huffman compression method
  
Information about plugins can be found under the "plugins section" of the Preferences file.
+
==== File header ====
 +
The file header is 8 bytes of size and consists of:
 +
{| class="wikitable"
 +
|-
 +
! Offset
 +
! Size
 +
! Value
 +
! Description
 +
|-
 +
| 0
 +
| 4
 +
| "MEM0" (0x4d, 0x45, 0x4d, 0x30)
 +
| Signature
 +
|-
 +
| 4
 +
| 4
 +
|
 +
| Uncompressed (total) data size
 +
|-
 +
|}
  
=== DNS Prefetching ===
+
==== Compressed blocks ====
 +
The file header is followed by compressed blocks:
 +
{| class="wikitable"
 +
|-
 +
! Offset
 +
! Size
 +
! Value
 +
! Description
 +
|-
 +
| 0
 +
| 4
 +
|
 +
| Compressed data size
 +
|-
 +
| 4
 +
| ...
 +
|
 +
| Compressed data
 +
|-
 +
|}
  
DNS is prefetched for related sites, e.g. links on the page.
+
The uncompressed block size is 65536 (0x10000) or the remaining uncompressed data size for the last block.
This behavior is controlled by the setting "Predict network actions to improve page load performance", which is enabled by default.
+
  
If enabled the Preferences file contains:
+
=== Compressed SuperFetch DB - MAM file format ===
<pre>
+
The MAM file consists of:
  "dns_prefetching": {
+
* file header
      "enabled": true,
+
* compressed blocks
</pre>
+
  
If disabled the Preferences file contains:
+
This format uses the LZXPRESS Huffman compression method.
<pre>
+
  "dns_prefetching": {
+
      "enabled": false,
+
</pre>
+
  
== Start-up DNS queries ==
+
==== File header ====
 +
The file header is 12 bytes of size and consists of:
 +
{| class="wikitable"
 +
|-
 +
! Offset
 +
! Size
 +
! Value
 +
! Description
 +
|-
 +
| 0
 +
| 4
 +
| "MAM\x84" (0x4d, 0x41, 0x4d, 0x84)
 +
| Signature
 +
|-
 +
| 4
 +
| 4
 +
|
 +
| Uncompressed (total) data size
 +
|-
 +
| 8
 +
| 4
 +
|
 +
| Unknown (checksum?)
 +
|-
 +
|}
  
When Chrome starts it queries for several non-existing hostnames that consists of a 10 random characters, E.g.
+
==== Compressed blocks ====
<pre>
+
<b>TODO</b>
ttrgoiknff.mydomain.com
+
bxjhgftsyu.mydomain.com
+
yokjbjiagd.mydomain.com
+
</pre>
+
  
This is used to determine if your ISP is hijacking NXDOMAIN results [http://www.google.com/support/forum/p/Chrome/thread?tid=3511015c72a7b314&hl=en].
+
=== Uncompressed SuperFetch DB format ===
 +
<b>TODO</b>
  
== Disk Cache ==
+
==== File header ====
Google Chrome uses multiple caches, from [http://src.chromium.org/viewvc/chrome/trunk/src/net/base/cache_type.h?view=markup]:
+
<b>TODO</b>
<pre>
+
// The types of caches that can be created.
+
enum CacheType {
+
    DISK_CACHE,  // Disk is used as the backing storage.
+
    MEMORY_CACHE,  // Data is stored only in memory.
+
    MEDIA_CACHE,  // Optimized to handle media files.
+
    APP_CACHE,  // Backing store for an AppCache.
+
    SHADER_CACHE, // Backing store for the GL shader cache.
+
    PNACL_CACHE, // Backing store the PNaCl translation cache
+
};
+
</pre>
+
 
+
The Google Chrome disk cache can be found in:
+
 
+
On Linux
+
  
 +
{| class="wikitable"
 +
|-
 +
! Offset
 +
! Size
 +
! Value
 +
! Description
 +
|-
 +
| 0
 +
| 4
 +
| 0x0000000e
 +
| Unknown (Database type or signature?)
 +
|-
 +
| 4
 +
| 4
 +
|
 +
| Uncompressed (total) data size
 +
|-
 +
|}
 +
== TRX files ==
 +
The <tt>Ag*.db.trx</tt> files are of the TRX file format. E.g.
 
<pre>
 
<pre>
/home/$USER/.cache/google-chrome/Default/Cache/
+
AgCx_SC*.db.trx
 
</pre>
 
</pre>
  
<pre>
+
<b>Note that the following format specification is incomplete.</b>
/home/$USER/.cache/google-chrome/Default/Media Cache/
+
</pre>
+
  
<pre>
+
=== File header ===
/home/$USER/.config/google-chrome/Default/Application Cache/Cache/
+
The file header is variable of size and consists of:
</pre>
+
{| class="wikitable"
 +
|-
 +
! Offset
 +
! Size
 +
! Value
 +
! Description
 +
|-
 +
| 0
 +
| 4
 +
| 1
 +
| Unknown (Version?)
 +
|-
 +
| 4
 +
| 4
 +
|
 +
| Unknown
 +
|-
 +
| 8
 +
| 4
 +
|
 +
| File size
 +
|-
 +
| 12
 +
| 4
 +
|
 +
| Maximum number of records (of the record offsets array)
 +
|-
 +
| 16
 +
| 4
 +
|
 +
| Number of records
 +
|-
 +
| 20
 +
| ...
 +
|
 +
| Record offsets array, where the record offset is a 32-bit integer. Unused record offset are set to 0.
 +
|-
 +
|}
  
<pre>
+
=== Record ===
/home/$USER/.config/google-chrome/Default/Cache/
+
<b>TODO describe</b>
</pre>
+
 
+
On MacOS-X
+
<pre>
+
/Users/$USER/Caches/Google/Chrome/Default/Cache/
+
</pre>
+
 
+
On Windows XP
+
<pre>
+
C:\Documents and Settings\%USERNAME%\Local Settings\Application Data\Google\Chrome\User Data\Cache\
+
</pre>
+
 
+
On Windows Vista and later
+
<pre>
+
C:\Users\%USERNAME%\AppData\Local\Google\Chrome\User Data\Default\Cache\
+
</pre>
+
 
+
The Chrome Cache contains different files with the following file names:
+
* index
+
* data_#; where # contains a decimal digit.
+
* f_######; where # contains a hexadecimal digit.
+
 
+
For more info see Chrome developers site [http://www.chromium.org/developers/design-documents/network-stack/disk-cache].
+
 
+
== History ==
+
Chrome stores the history of visited sites in a file named '''History'''. This file uses the [[SQLite database format]].
+
 
+
The '''History''' file can be found in same location as the '''Preferences''' file.
+
 
+
There is also '''Archived History''' that predates information in the '''History''' file.
+
Note that the '''Archived History''' only contains visits.
+
 
+
=== Timestamps ===
+
The '''History''' file uses the different timestamps.
+
 
+
==== visits.visit_time ====
+
 
+
The '''visits.visit_time''' is in (the number of) microseconds since January 1, 1601 UTC
+
 
+
Some Python code to do the conversion into human readable format:
+
<pre>
+
date_string = datetime.datetime( 1601, 1, 1 )
+
            + datetime.timedelta( microseconds=timestamp )
+
</pre>
+
 
+
Note that this timestamp is not the same as a Windows filetime which is (the number of) 100 nanoseconds since January 1, 1601 UTC
+
 
+
==== downloads.start_time ====
+
 
+
The '''downloads.start_time''' is in (the number of) seconds since January 1, 1970 UTC
+
 
+
Some Python code to do the conversion into human readable format:
+
<pre>
+
date_string = datetime.datetime( 1970, 1, 1 )
+
            + datetime.timedelta( seconds=timestamp )
+
</pre>
+
 
+
=== Example queries ===
+
Some example queries:
+
 
+
To get an overview of the visited sites:
+
<pre>
+
SELECT datetime(((visits.visit_time/1000000)-11644473600), "unixepoch"), urls.url, urls.title FROM urls, visits WHERE urls.id = visits.url;
+
</pre>
+
 
+
Note that the visit_time conversion looses precision.
+
 
+
To get an overview of the downloaded files:
+
<pre>
+
SELECT datetime(downloads.start_time, "unixepoch"), downloads.url, downloads.full_path, downloads.received_bytes, downloads.total_bytes FROM downloads;
+
</pre>
+
 
+
How the information of the downloaded files is stored in the database can vary per version of Chrome as of version 26:
+
<pre>
+
SELECT datetime(((downloads.start_time/1000000)-11644473600), "unixepoch"), downloads.target_path, downloads_url_chains.url, downloads.received_bytes, downloads.total_bytes \
+
FROM downloads, downloads_url_chains WHERE downloads.id = downloads_url_chains.id;
+
</pre>
+
  
 
== See Also ==
 
== See Also ==
 
+
* [[SuperFetch]]
* [[SQLite database format]]
+
  
 
== External Links ==
 
== External Links ==
* [http://en.wikipedia.org/wiki/Google_Chrome Wikipedia article on Google Chrome]
+
* [http://technet.microsoft.com/en-us/magazine/2007.03.vistakernel.aspx Inside the Windows Vista Kernel: Part 2], by [[Mark Russinovich]], March 2007
* [http://www.chromium.org/user-experience/user-data-directory The Chromium Projects - User Data Directory]
+
* [http://blog.rewolf.pl/blog/?p=214 Windows SuperFetch file format – partial specification], by ReWolf, October 5, 2011
* [http://www.chromium.org/developers/design-documents/network-stack/disk-cache Chrome Disk Cache]
+
* [http://www.google.com/support/forum/p/Chrome/thread?tid=3511015c72a7b314&hl=en Chrome support forum article random 10 character hostnames on startup]
+
* [http://www.useragentstring.com/pages/Chrome/ Chrome User Agent strings]
+
* [http://computer-forensics.sans.org/blog/2010/01/21/google-chrome-forensics/ Google Chrome Forensics] by [[Kristinn Guðjónsson]], January 21, 2010
+
* [http://linuxsleuthing.blogspot.ch/2013/02/cashing-in-on-google-chrome-cache.html?m=1 Cashing in on the Google Chrome Cache], [[John Lehr]], February 24, 2013
+
* [http://www.obsidianforensics.com/blog/history-index-files-removed-from-chrome/ History Index files removed from Chrome v30], by Ryan Benson, October 2, 2013
+
* [https://hindsight-internet-history.googlecode.com/files/Evolution_of_Chrome_Databases.png Evolution of Chrome Databases], by Ryan Benson, November 12, 2013
+
 
+
== Tools ==
+
=== Open Source ===
+
* [https://code.google.com/p/hindsight-internet-history/ hindsight-internet-history]
+
  
[[Category:Applications]]
+
[[Category:File Formats]]
[[Category:Web Browsers]]
+

Revision as of 23:33, 11 May 2014

Information icon.png

Please help to improve this article by expanding it.
Further information might be found on the discussion page.

SuperFetch, is a memory management scheme that enhances the least-recently accessed approach with historical information and proactive memory management. [1]

Note that the following format specification are incomplete.

SuperFetch DB files

The Ag*.db files are of the SuperFetch file format. E.g.

AgAppLaunch.db
AgCx_SC*.db
AgGlFaultHistory.db
AgGlFgAppHistory.db
AgGlGlobalHistory.db
AgGlUAD_%SID%.db
AgGlUAD_P_%SID%.db
AgRobust.db

The SuperFetch DB files can be stored in uncompressed or compressed form, where different version of Windows use different compressed forms:

  • Compressed SuperFetch DB - MEMO file format; Windows Vista
  • Compressed SuperFetch DB - MEM0 file format; Windows 7
  • Compressed SuperFetch DB - MAM file format; Windows 8

Compressed SuperFetch DB - MEMO file format

The MEM file consists of:

  • file header
  • compressed blocks

This format uses the LZNT1 compression method

File header

The file header is 84 bytes of size and consists of:

Offset Size Value Description
0 4 "MEMO" (0x4d, 0x45, 0x4d, 0x4f) Signature
4 4 Uncompressed (total) data size

Compressed blocks

The compressed block size is the chunk data size, which is part of the LZNT1 compressed data, + 2 bytes for the size of the chunk header itself.

The uncompressed block size is 4096 (0x1000) or the remaining uncompressed data size for the last block.

Compressed SuperFetch DB - MEM0 file format

The MEM file consists of:

  • file header
  • compressed blocks

This format uses the LZXPRESS Huffman compression method

File header

The file header is 8 bytes of size and consists of:

Offset Size Value Description
0 4 "MEM0" (0x4d, 0x45, 0x4d, 0x30) Signature
4 4 Uncompressed (total) data size

Compressed blocks

The file header is followed by compressed blocks:

Offset Size Value Description
0 4 Compressed data size
4 ... Compressed data

The uncompressed block size is 65536 (0x10000) or the remaining uncompressed data size for the last block.

Compressed SuperFetch DB - MAM file format

The MAM file consists of:

  • file header
  • compressed blocks

This format uses the LZXPRESS Huffman compression method.

File header

The file header is 12 bytes of size and consists of:

Offset Size Value Description
0 4 "MAM\x84" (0x4d, 0x41, 0x4d, 0x84) Signature
4 4 Uncompressed (total) data size
8 4 Unknown (checksum?)

Compressed blocks

TODO

Uncompressed SuperFetch DB format

TODO

File header

TODO

Offset Size Value Description
0 4 0x0000000e Unknown (Database type or signature?)
4 4 Uncompressed (total) data size

TRX files

The Ag*.db.trx files are of the TRX file format. E.g.

AgCx_SC*.db.trx

Note that the following format specification is incomplete.

File header

The file header is variable of size and consists of:

Offset Size Value Description
0 4 1 Unknown (Version?)
4 4 Unknown
8 4 File size
12 4 Maximum number of records (of the record offsets array)
16 4 Number of records
20 ... Record offsets array, where the record offset is a 32-bit integer. Unused record offset are set to 0.

Record

TODO describe

See Also

External Links