Difference between pages "Carver 2.0 Planning Page" and "Mac OS X"

From ForensicsWiki
(Difference between pages)
Jump to: navigation, search
(Discussion)
 
(Launch Agents)
 
Line 1: Line 1:
This page is for planning Carver 2.0.
+
{{Expand}}
  
Please, do not delete text (ideas) here. Use something like this:
+
Apple Inc.'s Macintosh OS X (pronounced "'''OS Ten'''") is the operating system distributed with Apple computers. It includes heavily used several programs by default, including [[Apple Mail]], a web browser called [[Apple Safari | Safari]], and an [[Apple Address Book]], and [[iCal]].  
  
 +
== Disk image types ==
 +
 +
Mac OS X has support for various disk image types build-in, some of which are:
 +
* read-write disk image (.dmg) some of which use the [[Raw Image Format]]
 +
* [[Sparse Image format|Sparse disk image (.spareimage)]]
 +
* [[Sparse Bundle Image format|Sparse bundle disk image (.sparsebundle)]]
 +
 +
== Burn Folder ==
 +
 +
Mac OS X Burn Folder:
 
<pre>
 
<pre>
<s>bad idea</s>
+
$NAME.fpbf
:: good idea
+
 
</pre>
 
</pre>
  
This will look like:
+
This folder normally contains [[Mac OS X Alias Files|alias files]] (similar to LNK files under Windows). Which should have the following signature.
 +
<pre>
 +
00000000  62 6f 6f 6b 00 00 00 00  6d 61 72 6b 00 00 00 00  |book....mark....|
 +
</pre>
  
<s>bad idea</s>
+
These [[Mac OS X Alias Files|alias files]] contain additional date and time values.
:: good idea
+
  
= License =
+
Also check the following files for references to deleted .fpbf paths:
 +
<pre>
 +
/Users/$USERNAME/Library/Preferences/com.apple.finder.plist
 +
/Users/$USERNAME/Library/Preferences/com.apple.sidebarlists.plist
 +
</pre>
  
BSD-3.
+
Actual burning of optical media is logged in:
 +
<pre>
 +
/var/log/system.log
 +
/Users/$USERNAME/Library/Logs/DiscRecording.log
 +
/private/var/.logs_exporter/cache/Users/$USERNAME/Library/Logs/DiscRecording.log
 +
</pre>
  
= OS =
+
== HFS/HFS+ date and time values ==
  
Linux/FreeBSD/MacOS
+
In HFS+ date and time values are stored in an unsigned 32-bit integer containing the number of seconds since January 1, 1904 at 00:00:00 (midnight) UTC (GMT). This is slightly different from HFS where the date and time value are stored using the local time. The maximum representable date is February 6, 2040 at 06:28:15 UTC (GMT). The date values do not account for leap seconds. They do include a leap day in every year that is evenly divisible by four. This is sufficient given that the range of representable dates does not contain 1900 or 2100, neither of which have leap days. Also see: [http://web.archive.org/web/20090214212148/http://developer.apple.com/technotes/tn/tn1150.html Technical Note TN1150 - HFS Plus Volume Format]
: (shouldn't this just match what the underlying afflib & sleuthkit cover? [[User:RB|RB]])
+
:: Yes, but you need to test and validate on each. Question: Do we want to support windows? [[User:Simsong|Simsong]] 21:09, 30 October 2008 (UTC)
+
:: [[User:Joachim Metz|Joachim]] I think we would do wise to design with windows support from the start this will improve the platform independence from the start
+
  
= Requirements =
+
Converting HFS/HFS+ date and time values with Python:
 +
<pre>
 +
import datetime
  
* [[User:Joachim Metz|Joachim]] A name for the tooling I propose coldcut
+
print datetime.datetime( 1904, 1, 1 ) + datetime.timedelta( seconds=0xCBDAF25B )
 +
</pre>
  
[[User:Joachim Metz|Joachim]] Could we do a MoSCoW evaluation of these.
+
== Launch Agents ==
 +
System-wide:
 +
<pre>
 +
/Library/LaunchAgents
 +
/System/Library/LaunchAgents
 +
</pre>
  
* AFF and EWF file images supported from scratch. ([[User:Joachim Metz|Joachim]] I would like to have raw/split raw and device access as well)
+
Per user:
* [[User:Joachim Metz|Joachim]] volume/partition aware layer (what about carving unpartioned space)
+
<pre>
* File system aware layer.
+
/Users/$USERNAME/Library/LaunchAgents
** By default, files are not carved. (clarify: only identified? [[User:RB|RB]]; I guess that it operates like [[Selective file dumper]] [[User:.FUF|.FUF]] 07:00, 29 October 2008 (UTC))
+
</pre>
* Plug-in architecture for identification/validation.
+
** [[User:Joachim Metz|Joachim]] support for multiple types of validators
+
*** dedicated validator
+
*** validator based on file library (i.e. we could specify/implement a file structure for these)
+
*** configuration based validator (Can handle config files,like Revit07, to enter different file formats used by the carver.)
+
* Ship with validators for:
+
[[User:Joachim Metz|Joachim]] I think we should distinguish between file format validators and content validators
+
** JPEG
+
** PNG
+
** GIF
+
** MSOLE
+
** ZIP
+
** TAR (gz/bz2)
+
  
[[User:Joachim Metz|Joachim]] For a production carver we need at least the following formats
+
These directories contain  [[Property list (plist)]] files.
** Grapical Images
+
*** JPEG (the 3 different types with JFIF/EXIF support)
+
*** PNG
+
*** GIF
+
*** BMP
+
*** TIFF
+
** Office documents
+
*** OLE2 (Word/Excell content support)
+
*** PDF
+
*** Open Office/Office 2007 (ZIP+XML)
+
** Archive files
+
*** ZIP
+
** E-mail files
+
*** PFF (PST/OST)
+
*** MBOX (text based format, base64 content support)
+
** Audio/Video files
+
*** MPEG
+
*** MP2/MP3
+
*** AVI
+
*** ASF/WMV
+
*** QuickTime
+
** Printer spool files
+
*** EMF (if I remember correctly)
+
** Internet history files
+
*** index.dat
+
*** firefox (sqllite 3)
+
** Other files
+
*** thumbs.db
+
  
* Simple fragment recovery carving using gap carving.
+
== Launch Daemons ==
** [[User:Joachim Metz|Joachim]] have hook in for more advanced fragment recovery?
+
System-wide:
* Recovering of individual ZIP sections and JPEG icons that are not sector aligned.
+
<pre>
** [[User:Joachim Metz|Joachim]] I would propose a generic fragment detection and recovery
+
/Library/LaunchDaemons
* Autonomous operation (some mode of operation should be completely non-interactive, requiring no human intervention to complete [[User:RB|RB]])
+
/System/Library/LaunchDaemons
** [[User:Joachim Metz|Joachim]] as much as possible, but allow to be overwritten by user
+
</pre>
* Tested on 500GB-sized images. Should be able to carve a 500GB image in roughly 50% longer than it takes to read the image.
+
** Perhaps allocate a percentage budget per-validator (i.e. each validator adds N% to the carving time)
+
** [[User:Joachim Metz|Joachim]] have multiple carving phases for precision/speed trade off?
+
* Parallelizable
+
** [[User:Joachim Metz|Joachim]] tunable for different architectures
+
* Configuration:
+
** Capability to parse some existing carvers' configuration files, either on-the-fly or as a one-way converter.
+
** Disengage internal configuration structure from configuration files, create parsers that present the expected structure
+
** [[User:Joachim Metz|Joachim]] The validator should deal with the file structure the carving algorithm should not know anything about the file structure (as in revit07 design)
+
**  Either extend Scalpel/Foremost syntaxes for extended features or use a tertiary syntax ([[User:Joachim Metz|Joachim]] I would prefer a derivative of the revit07 configuration syntax which already has encountered some problems of dealing with defining file structure in a configuration file)
+
* Can output audit.txt file.
+
* [[User:Joachim Metz|Joachim]] Can output database with offset analysis values i.e. for visualization tooling
+
* [[User:Joachim Metz|Joachim]] Can output debug log for debugging the algorithm/validation
+
* Easy integration into ascription software.
+
** [[User:Joachim Metz|Joachim]] I'm no native speaker what do you mean with "ascription software"?
+
  
= Ideas =
+
These directories contain [[Property list (plist)]] files.
* Use as much TSK if possible. Don't carry your own FS implementation the way photorec does.
+
** [[User:Joachim Metz|Joachim]] using TSK as much as possible would not allow to add your own file system support (i.e. mobile phones, memory structures, cap files)
+
I would propose wrapping TSK and using it as much as possible but allow to integrate own FS implementations.
+
* Extracting/carving data from [[Thumbs.db]]? I've used [[foremost]] for it with some success. [[Vinetto]] has some critical bugs :( [[User:.FUF|.FUF]] 19:18, 28 October 2008 (UTC)
+
* Carving data structures. For example, extract all TCP headers from image by defining TCP header structure and some fields (e.g. source port > 1024, dest port = 80). This will extract all data matching the pattern and write a file with other fields. Another example is carving INFO2 structures and URL activity records from index.dat [[User:.FUF|.FUF]] 20:51, 28 October 2008 (UTC)
+
** This has the opportunity to be extended to the concept of "point at blob FOO and interpret it as BAR"
+
  
.FUF added:
+
== Startup Items ==
The main idea is to allow users to define structures, for example (in pascal-like form):
+
<pre>
 +
/Library/StartupItems/
 +
/System/Library/StartupItems/
 +
</pre>
  
 +
== Crash Reporter ==
 
<pre>
 
<pre>
Field1: Byte = 123;
+
/Library/Application Support/CrashReporter
SomeTextLength: DWORD;
+
SomeText: string[SomeTextLength];
+
Field4: Char = 'r';
+
...
+
 
</pre>
 
</pre>
  
This will produce something like this:
+
== Diagnostic Reports ==
 
<pre>
 
<pre>
Field1 = 123
+
/Library/Logs/DiagnosticReports
SomeTextLength = 5
+
SomeText = 'abcd1'
+
Field4 = 'r'
+
 
</pre>
 
</pre>
  
(In text or raw forms.)
+
== Quarantine event database ==
 +
See [http://menial.co.uk/blog/2011/06/16/mac-quarantine-event-database/]
  
Opinions?
+
Snow Leopard and earlier
 +
<pre>
 +
/Users/$USER/Library/Preferences/com.apple.LaunchServices.QuarantineEvents
 +
</pre>
  
Opinion: Simple pattern identification like that may not suffice, I think Simson's original intent was not only to identify but to allow for validation routines (plugins, as the original wording was).  As such, the format syntax would need to implement a large chunk of some programming language in order to be sufficiently flexible. [[User:RB|RB]]
+
<pre>
 +
SELECT datetime(LSQuarantineTimeStamp + 978307200, "unixepoch") as LSQuarantineTimeStamp, LSQuarantineAgentName, LSQuarantineOriginURLString, LSQuarantineDataURLString from LSQuarantineEvent;
 +
</pre>
  
=File System Awareness =
+
Lion and later
==Background: Why be File System Aware?==
+
<pre>
Advantages of being FS aware:
+
/Users/$USER/Library/Preferences/com.apple.LaunchServices.QuarantineEventsV2
* You can pick up sector allocation sizes ([[User:Joachim Metz|Joachim]] do you mean file system block sizes?)
+
</pre>
* Some file systems may store things off sector boundaries. (ReiserFS with tail packing)
+
* Increasingly file systems have compression (NTFS compression)
+
* Carve just the sectors that are not in allocated files.
+
  
==Tasks that would be required==
+
== sleepimage ==
 
+
This file is similar to the hibernation file on Windows.
==Discussion==
+
<pre>
:: As noted above, TSK should be utilized as much as possible, particularly the filesystem-aware portion.  If we want to identify filesystems outside of its supported set, it would be more worth our time to work on implementing them there than in the carver itself. [[User:RB|RB]]
+
/private/var/vm/sleepimage
 
+
</pre>
[[User:Joachim Metz|Joachim]] I would like to have the carver (recovery tool) also do recovery using file allocation data or remainders of file allocation data.
+
 
+
:::: I guess this tool operates like [[Selective file dumper]] and can recover files in both ways (or not?). Recovering files by using carving can recover files in situations where sleuthkit does nothing (e.g. file on NTFS was deleted using ntfs-3g, or filesystem was destroyed or just unknown). And we should build the list of filesystems supported by carver, not by TSK. [[User:.FUF|.FUF]] 07:08, 29 October 2008 (UTC)
+
  
:: This tool is still in the early planning stages (requirements discovery), hence few operational details (like precise modes of operation) have been fleshed out - those will and should come later. The justification for strictly using TSK for the filesystem-sensitive approach is simple: TSK has good filesystem APIs, and it would be foolish to create yet another standalone, incompatible implementation of filesystem(foo) when time would be better spent improving those in TSK, aiding other methods of analysis as well.  This is the same reason individuals that have implemented several other carvers are participating: de-duplication of effort.  [[User:RB|RB]]
+
Also see: [http://osxdaily.com/2010/10/11/sleepimage-mac/]
  
[[User:Joachim Metz|Joachim]]
+
== Package Files (.PKG) ==
I would go as far to ask you all to look beyond the carver as a tool and look from the perspective of the carver as part of the forensic investigation process. In my eyes certain information needed/acquired by the carver could be also very useful investigative information i.e. what part of a hard disk contains empty sectors.
+
Package Files (.PKG) are XAR archives [http://en.wikipedia.org/wiki/Xar_(archiver)] that contain a cpio archive and metadata [http://s.sudre.free.fr/Stuff/Ivanhoe/FLAT.html].
  
[[User:Joachim Metz|Joachim]]
+
== Also see ==
I'm missing a part on the page about the carving challenges (scenarios)
+
* [[MacOS Process Monitoring]]
* normal file (file structure, loose text based structure (more a content structure?))
+
* [[Acquiring a MacOS System with Target Disk Mode]]
* fragmented file (the file entirely exist)
+
* [[Converting Binary Plists]]
* a file fragment (the file does not entirely exist)
+
* [[FileVault Disk Encryption]]
* intertwined file
+
* [[File Vault]]
* encapsulated file (MPEG/network capture)
+
* embedded file (JPEG thumbnail)
+
  
[[User:Joachim Metz|Joachim]]  
+
=== Formats ===
I'm missing a part on the page about the carving algorithm
+
* [[Basic Security Module (BSM) file format]]
* should we allow for multiple runs?
+
* [[Property list (plist)]]
* should we allow for multiple algorithms?
+
* does the algorithm passes data blocks to the validators?
+
* does a validator need to maintain a state?
+
* does a validator need to revert a state?
+
* do we use the assumption that a data block can be used by a single file (with the exception of embedded/encapsulated files)?
+
  
=Validator Construction=
+
== External Links ==
Options:
+
* [http://www.apple.com/macosx/ Official website]
* Write validators in C/C++
+
* [http://en.wikipedia.org/wiki/OS_X Wikipedia entry on OS X]
* Have a scripting language for writing them (python? Perl?) our own?
+
* [http://menial.co.uk/blog/2011/06/16/mac-quarantine-event-database/ Quarantine event database]
** [[User:Joachim Metz|Joachim]] use easy to embed programming languages i.e. Phyton or Lua
+
* [http://www2.tech.purdue.edu/cit/Courses/cit556/readings/MacForensicsCraiger.pdf Mac Forensics: Mac OS X and the HFS+ File System] by P. Craiger
* Use existing programs (libjpeg?) as plug-in validators?
+
* [http://web.me.com/driley/iWeb/Previous_files/Directory_Services_Overview.pdf Mac OS X Directory Services Integration including Active Directory]
** [[User:Joachim Metz|Joachim]] define a file structure api for this
+
* [http://digitalinvestigation.wordpress.com/2012/04/04/geek-post-nskeyedarchiver-files-what-are-they-and-how-can-i-use-them/ NSKeyedArchiver files – what are they, and how can I use them?]
 +
* [http://krypted.com/mac-os-x/command-line-alf-on-mac-os-x/ Command Line ALF on Mac OS X]
 +
* [http://newosxbook.com/DMG.html Demystifying the DMG File Format]
 +
* [https://code.google.com/p/mac-security-tips/wiki/ALL_THE_TIPS mac-security-tips]
  
=Existing Code that we have=
+
=== Apple Examiner ===
 +
* [http://www.appleexaminer.com/ The Apple Examiner]
 +
* [http://www.appleexaminer.com/MacsAndOS/Analysis/USBOSX/USBOSX.html USB Entries on OS X]
 +
* [http://www.appleexaminer.com/Downloads/MacForensics.pdf Macintosh Forensics - A Guide for the Forensically Sound Examination of a Macintosh Computer] by Ryan R. Kubasiak
  
[[User:Joachim Metz|Joachim]]
+
=== iCloud ===
* DFRWS2006/2007 carving challenge results
+
* [http://support.apple.com/kb/HT4865?viewlocale=en_US&locale=en_US iCloud: iCloud security and privacy overview]
* photorec
+
* revit06 and revit07
+
* s3/scarve
+
  
=Implementation Timeline=
+
[[Category:Mac OS X]]
# gather the available resources/ideas/wishes/needs etc. (I guess we're in this phase)
+
[[Category:Operating systems]]
# start discussing a high level design (in terms of algorithm, facilities, information needed)
+
## input formats facility
+
## partition/volume facility
+
## file system facility
+
## file format facility
+
## content facility
+
## how to deal with fragment detection (do the validators allow for fragment detection?)
+
## how to deal with recombination of fragments
+
## do we want multiple carving phases in light of speed/precision tradeoffs
+
# start detailing parts of the design
+
## Discuss options for a grammar driven validator?
+
## Hard-coded plug-ins?
+
## Which exsisting code can we use?
+
# start building/assembling parts of the tooling for a prototype
+
## Implement simple file carving with validation.
+
## Implement gap carving
+
# Initial Release
+
# Implement the ''threaded carving'' that [[User:.FUF|.FUF]] is describing above.
+

Revision as of 13:08, 12 June 2014

Information icon.png

Please help to improve this article by expanding it.
Further information might be found on the discussion page.

Apple Inc.'s Macintosh OS X (pronounced "OS Ten") is the operating system distributed with Apple computers. It includes heavily used several programs by default, including Apple Mail, a web browser called Safari, and an Apple Address Book, and iCal.

Disk image types

Mac OS X has support for various disk image types build-in, some of which are:

Burn Folder

Mac OS X Burn Folder:

$NAME.fpbf

This folder normally contains alias files (similar to LNK files under Windows). Which should have the following signature.

00000000  62 6f 6f 6b 00 00 00 00  6d 61 72 6b 00 00 00 00  |book....mark....|

These alias files contain additional date and time values.

Also check the following files for references to deleted .fpbf paths:

/Users/$USERNAME/Library/Preferences/com.apple.finder.plist
/Users/$USERNAME/Library/Preferences/com.apple.sidebarlists.plist

Actual burning of optical media is logged in:

/var/log/system.log
/Users/$USERNAME/Library/Logs/DiscRecording.log
/private/var/.logs_exporter/cache/Users/$USERNAME/Library/Logs/DiscRecording.log

HFS/HFS+ date and time values

In HFS+ date and time values are stored in an unsigned 32-bit integer containing the number of seconds since January 1, 1904 at 00:00:00 (midnight) UTC (GMT). This is slightly different from HFS where the date and time value are stored using the local time. The maximum representable date is February 6, 2040 at 06:28:15 UTC (GMT). The date values do not account for leap seconds. They do include a leap day in every year that is evenly divisible by four. This is sufficient given that the range of representable dates does not contain 1900 or 2100, neither of which have leap days. Also see: Technical Note TN1150 - HFS Plus Volume Format

Converting HFS/HFS+ date and time values with Python:

import datetime

print datetime.datetime( 1904, 1, 1 ) + datetime.timedelta( seconds=0xCBDAF25B )

Launch Agents

System-wide:

/Library/LaunchAgents
/System/Library/LaunchAgents

Per user:

/Users/$USERNAME/Library/LaunchAgents

These directories contain Property list (plist) files.

Launch Daemons

System-wide:

/Library/LaunchDaemons
/System/Library/LaunchDaemons

These directories contain Property list (plist) files.

Startup Items

/Library/StartupItems/
/System/Library/StartupItems/

Crash Reporter

/Library/Application Support/CrashReporter

Diagnostic Reports

/Library/Logs/DiagnosticReports

Quarantine event database

See [1]

Snow Leopard and earlier

/Users/$USER/Library/Preferences/com.apple.LaunchServices.QuarantineEvents
SELECT datetime(LSQuarantineTimeStamp + 978307200, "unixepoch") as LSQuarantineTimeStamp, LSQuarantineAgentName, LSQuarantineOriginURLString, LSQuarantineDataURLString from LSQuarantineEvent;

Lion and later

/Users/$USER/Library/Preferences/com.apple.LaunchServices.QuarantineEventsV2

sleepimage

This file is similar to the hibernation file on Windows.

/private/var/vm/sleepimage

Also see: [2]

Package Files (.PKG)

Package Files (.PKG) are XAR archives [3] that contain a cpio archive and metadata [4].

Also see

Formats

External Links

Apple Examiner

iCloud