Difference between pages "Windows SuperFetch Format" and "SuperFetch"

From ForensicsWiki
(Difference between pages)
Jump to: navigation, search
(External Links)
 
(Components)
 
Line 1: Line 1:
{{expand}}
+
{{Expand}}
  
SuperFetch, is a memory management scheme that enhances the least-recently accessed approach with historical information and proactive memory management. [http://technet.microsoft.com/en-us/magazine/2007.03.vistakernel.aspx]
+
SuperFetch is a performance enhancement introduced in [[Microsoft]] [[Windows|Windows Vista]] to reduce the time necessary to launch applications. SuperFetch works with the memory manager service in Windows to analyze memory usage patterns over time to determine the optimal memory content for a given user for a date or time of day. This differs from the [[Prefetch]] technique used in Microsoft Windows XP, which preloads data into memory without analyzing usage patterns.
  
<b>Note that the following format specification are incomplete.</b>
+
From [http://msdn.microsoft.com/en-us/library/windows/hardware/dn653317(v=vs.85).aspx]: SuperFetch prioritizes the following kinds of pages to remain in memory:
 +
* Pages of applications that are used most frequently overall.
 +
* Pages of applications that are commonly used when resuming:
 +
** After extensive hibernation (for example, first thing in the morning).
 +
** After shorter periods of sleep or hibernation (for example, after lunch).  
  
== SuperFetch DB files ==
+
If SuperFetch detects that the system drive is a fast SSD (as measured by Windows Experience Index Disk score), then SuperFetch turns off [[ReadyBoot]], [[ReadyBoost]], and the SuperFetch service itself.
The <tt>Ag*.db</tt> files are of the SuperFetch file format. E.g.
+
<pre>
+
AgAppLaunch.db
+
AgCx_SC*.db
+
AgGlFaultHistory.db
+
AgGlFgAppHistory.db
+
AgGlGlobalHistory.db
+
AgGlUAD_%SID%.db
+
AgGlUAD_P_%SID%.db
+
AgRobust.db
+
</pre>
+
  
The SuperFetch DB files can be stored in uncompressed or compressed form, where different version of Windows use different compressed forms:
+
== Components ==
* Compressed SuperFetch DB - MEMO file format; Windows Vista
+
=== Robust performance ===
* Compressed SuperFetch DB - MEM0 file format; Windows  7
+
Robust performance (or robustness) is a component of SuperFetch to watch for specific file I/O access that might harm system performance by populating the standby lists with unneeded data.
* Compressed SuperFetch DB - MAM file format; Windows 8
+
  
=== Compressed SuperFetch DB - MEMO file format ===
+
== Scenarios ==
The MEM file consists of:
+
SuperFetch distinguishes between different scenarios to accurately measure performance.
* file header
+
* compressed blocks
+
  
This format uses the LZNT1 compression method
+
=== Cold scenario ===
 +
In a cold scenario, the test applications are not already in memory when the test begins. Cold scenarios measure performance either after a state transition, such as boot or resume from hibernation, or after another application claims most of the available memory, such as after launching and quitting a game.
  
==== File header ====
+
=== Warm scenario ===
The file header is 84 bytes of size and consists of:
+
In a warm scenario, some or all the scenario contents are in memory before measurement. This usually means that the test has run at least once during this logon session.
{| class="wikitable"
+
|-
+
! Offset
+
! Size
+
! Value
+
! Description
+
|-
+
| 0
+
| 4
+
| "MEMO" (0x4d, 0x45, 0x4d, 0x4f)
+
| Signature
+
|-
+
| 4
+
| 4
+
|
+
| Uncompressed (total) data size
+
|-
+
|}
+
  
==== Compressed blocks ====
+
== Configuration ==
The compressed block size is the chunk data size, which is part of the LZNT1 compressed data, + 2 bytes for the size of the chunk header itself.
+
  
The uncompressed block size is 4096 (0x1000) or the remaining uncompressed data size for the last block.
+
Because SuperFetch appears to leave a system with no available memory, some users turn it off to create the appearance of having more free memory. The feature can be configured by changing the [[Registry]] value [http://www.codinghorror.com/blog/archives/000688.html]:
 +
<pre>
 +
Key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PrefetchParameters
 +
Value: EnableSuperfetch
 +
</pre>
  
=== Compressed SuperFetch DB - MEM0 file format ===
+
A value of zero disables SuperFetch, one enables it for booting only, two for applications, and three for both applications and boot. This setting can also be changed using the Services console, <tt>services.msc</tt> [http://tiredblogger.wordpress.com/2007/03/27/superfetch-not-so-super-for-gaming/].
The MEM file consists of:
+
* file header
+
* compressed blocks
+
  
This format uses the LZXPRESS Huffman compression method
+
== File Formats ==
  
==== File header ====
+
Data for SuperFetch is gathered by the <tt>%SystemRoot%\System32\Sysmain.dll</tt>, part of the Service Host process, <tt>%SystemRoot%\System32\Svchost.exe</tt>, and stored in a series of files in the <tt>%SystemRoot%\Prefetch</tt> directory [http://www.microsoft.com/technet/technetmag/issues/2007/03/VistaKernel/]. These files appear to start with the prefix <tt>Ag</tt> and have a <tt>.db</tt> extension. Note that there are likely more SuperFetch database files named differently, presumably all using the .db extension.
The file header is 8 bytes of size and consists of:
+
{| class="wikitable"
+
|-
+
! Offset
+
! Size
+
! Value
+
! Description
+
|-
+
| 0
+
| 4
+
| "MEM0" (0x4d, 0x45, 0x4d, 0x30)
+
| Signature
+
|-
+
| 4
+
| 4
+
|
+
| Uncompressed (total) data size
+
|-
+
|}
+
  
==== Compressed blocks ====
+
The format of the SuperFetch database files is not fully known, there is available unofficial partial specification [http://blog.rewolf.pl/blog/?p=214] and open source (GPL) dumper for .db files [http://code.google.com/p/rewolf-superfetch-dumper/]. For more information see [[Windows SuperFetch Format|SuperFetch Format]].
The file header is followed by compressed blocks:
+
{| class="wikitable"
+
|-
+
! Offset
+
! Size
+
! Value
+
! Description
+
|-
+
| 0
+
| 4
+
|
+
| Compressed data size
+
|-
+
| 4
+
| ...
+
|
+
| Compressed data
+
|-
+
|}
+
  
The uncompressed block size is 65536 (0x10000) or the remaining uncompressed data size for the last block.
+
The SuperFetch feature is seeded with some basic usage patterns when the operating system is installed [http://channel9.msdn.com/showpost.aspx?postid=242429].
  
=== Compressed SuperFetch DB - MAM file format ===
+
The SuperFetch service is managed by the File Information FS MiniFilter service. It appears that most of the SuperFetch database files are updated (written) when the service is shut down. AgAppLaunch.db is also written when the service starts.
The MAM file consists of:
+
* file header
+
* compressed blocks
+
 
+
This format uses the LZXPRESS Huffman compression method.
+
 
+
==== File header ====
+
The file header is 12 bytes of size and consists of:
+
{| class="wikitable"
+
|-
+
! Offset
+
! Size
+
! Value
+
! Description
+
|-
+
| 0
+
| 4
+
| "MAM\x84" (0x4d, 0x41, 0x4d, 0x84)
+
| Signature
+
|-
+
| 4
+
| 4
+
|
+
| Uncompressed (total) data size
+
|-
+
| 8
+
| 4
+
|
+
| Unknown (checksum?)
+
|-
+
|}
+
 
+
==== Compressed blocks ====
+
The uncompressed block size is 65536 (0x10000) or the remaining uncompressed data size for the last block.
+
 
+
Note the LZExpress Huffman variant used differs from the one used in the MEM0 format. This variant can refer to uncompressed data stored in the previous compressed block. The compressed blocks are essentially a single compressed stream.
+
 
+
=== Uncompressed SuperFetch DB format ===
+
<b>TODO</b>
+
 
+
==== File header ====
+
<b>TODO</b>
+
 
+
{| class="wikitable"
+
|-
+
! Offset
+
! Size
+
! Value
+
! Description
+
|-
+
| 0
+
| 4
+
| 0x0000000e
+
| Unknown (Database type or signature?)
+
|-
+
| 4
+
| 4
+
|
+
| Uncompressed (total) data size
+
|-
+
|}
+
== TRX files ==
+
The <tt>Ag*.db.trx</tt> files are of the TRX file format. E.g.
+
<pre>
+
AgCx_SC*.db.trx
+
</pre>
+
 
+
<b>Note that the following format specification is incomplete.</b>
+
 
+
=== File header ===
+
The file header is variable of size and consists of:
+
{| class="wikitable"
+
|-
+
! Offset
+
! Size
+
! Value
+
! Description
+
|-
+
| 0
+
| 4
+
| 1
+
| Unknown (Version?)
+
|-
+
| 4
+
| 4
+
|
+
| Unknown
+
|-
+
| 8
+
| 4
+
|
+
| File size
+
|-
+
| 12
+
| 4
+
|
+
| Maximum number of records (of the record offsets array)
+
|-
+
| 16
+
| 4
+
|
+
| Number of records
+
|-
+
| 20
+
| ...
+
|
+
| Record offsets array, where the record offset is a 32-bit integer. Unused record offset are set to 0.
+
|-
+
|}
+
 
+
=== Record ===
+
<b>TODO describe</b>
+
  
 
== See Also ==
 
== See Also ==
* [[SuperFetch]]
+
* [[Prefetch]]
 +
* [[ReadyBoost]]
 +
* [[ReadyBoot]]
 +
* [[Windows SuperFetch Format|SuperFetch Format]]
 +
* [[Windows]]
  
 
== External Links ==
 
== External Links ==
* [http://blog.rewolf.pl/blog/?p=214 Windows SuperFetch file format – partial specification], by ReWolf, October 5, 2011
+
* [http://technet.microsoft.com/en-us/magazine/2007.03.vistakernel.aspx Inside the Windows Vista Kernel: Part 2], by [[Mark Russinovich]], March 2007
* [https://googledrive.com/host/0B3fBvzttpiiSWFFkSTBOUHdjOVU/Windows%20SuperFetch%20(DB)%20format.pdf Windows SuperFetch database format], by the [[libagdb| libagdb project]], April 2014
+
* [http://download.microsoft.com/download/7/E/7/7E7662CF-CBEA-470B-A97E-CE7CE0D98DC2/Win7Perf.docx Performance Testing Guide for Windows], by [[Microsoft]], August 18, 2009 
 +
* [http://en.wikipedia.org/wiki/Windows_Vista_I/O_technologies#SuperFetch Wikipedia: Windows Vista I/O technologies - SuperFetch]
 +
* [http://channel9.msdn.com/showpost.aspx?postid=242429 Channel 9 Interview with Michael Fortin of Microsoft on SuperFetch]
 +
* [http://www.informationweek.com/news/showArticle.jhtml?articleID=196902178 Microsoft Predicts The Future With Vista's SuperFetch] from Information Week
 +
* [http://jessekornblum.com/presentations/dodcc08-2.pdf DC3 Presentation: My You Look SuperFetching], by Jesse Kornblum
 +
 
 +
== Tools ==
 +
=== Open Source ===
 +
* [https://code.google.com/p/rewolf-superfetch-dumper/ rewolf-superfetch-dumper]
  
[[Category:File Formats]]
+
[[Category:Windows]]

Revision as of 07:10, 28 June 2014

Information icon.png

Please help to improve this article by expanding it.
Further information might be found on the discussion page.

SuperFetch is a performance enhancement introduced in Microsoft Windows Vista to reduce the time necessary to launch applications. SuperFetch works with the memory manager service in Windows to analyze memory usage patterns over time to determine the optimal memory content for a given user for a date or time of day. This differs from the Prefetch technique used in Microsoft Windows XP, which preloads data into memory without analyzing usage patterns.

From [1]: SuperFetch prioritizes the following kinds of pages to remain in memory:

  • Pages of applications that are used most frequently overall.
  • Pages of applications that are commonly used when resuming:
    • After extensive hibernation (for example, first thing in the morning).
    • After shorter periods of sleep or hibernation (for example, after lunch).

If SuperFetch detects that the system drive is a fast SSD (as measured by Windows Experience Index Disk score), then SuperFetch turns off ReadyBoot, ReadyBoost, and the SuperFetch service itself.

Components

Robust performance

Robust performance (or robustness) is a component of SuperFetch to watch for specific file I/O access that might harm system performance by populating the standby lists with unneeded data.

Scenarios

SuperFetch distinguishes between different scenarios to accurately measure performance.

Cold scenario

In a cold scenario, the test applications are not already in memory when the test begins. Cold scenarios measure performance either after a state transition, such as boot or resume from hibernation, or after another application claims most of the available memory, such as after launching and quitting a game.

Warm scenario

In a warm scenario, some or all the scenario contents are in memory before measurement. This usually means that the test has run at least once during this logon session.

Configuration

Because SuperFetch appears to leave a system with no available memory, some users turn it off to create the appearance of having more free memory. The feature can be configured by changing the Registry value [2]:

Key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PrefetchParameters
Value: EnableSuperfetch

A value of zero disables SuperFetch, one enables it for booting only, two for applications, and three for both applications and boot. This setting can also be changed using the Services console, services.msc [3].

File Formats

Data for SuperFetch is gathered by the %SystemRoot%\System32\Sysmain.dll, part of the Service Host process, %SystemRoot%\System32\Svchost.exe, and stored in a series of files in the %SystemRoot%\Prefetch directory [4]. These files appear to start with the prefix Ag and have a .db extension. Note that there are likely more SuperFetch database files named differently, presumably all using the .db extension.

The format of the SuperFetch database files is not fully known, there is available unofficial partial specification [5] and open source (GPL) dumper for .db files [6]. For more information see SuperFetch Format.

The SuperFetch feature is seeded with some basic usage patterns when the operating system is installed [7].

The SuperFetch service is managed by the File Information FS MiniFilter service. It appears that most of the SuperFetch database files are updated (written) when the service is shut down. AgAppLaunch.db is also written when the service starts.

See Also

External Links

Tools

Open Source