Difference between pages "Bulk extractor" and "Mac OS X"

From ForensicsWiki
(Difference between pages)
Jump to: navigation, search
m
 
m (EFI boot)
 
Line 1: Line 1:
== Overview ==
+
{{Expand}}
'''bulk_extractor''' is a computer forensics tool that scans a disk image, a file, or a directory of files and extracts useful information without parsing the file system or file system structures. The results can be easily inspected, parsed, or processed with automated tools. '''bulk_extractor''' also created a histograms of features that it finds, as features that are more common tend to be more important. The program can be used for law enforcement, defense, intelligence, and cyber-investigation applications.
+
  
bulk_extractor is distinguished from other forensic tools by its speed and thoroughness. Because it ignores file system structure, bulk_extractor can process different parts of the disk in parallel. In practice, the program splits the disk up into 16MiByte pages and processes one page on each available core. This means that 24-core machines process a disk roughly 24 times faster than a 1-core machine. bulk_extractor is also thorough. That’s because bulk_extractor automatically detects, decompresses, and recursively re-processes compressed data that is compressed with a variety of algorithms. Our testing has shown that there is a significant amount of compressed data in the unallocated regions of file systems that is missed by most forensic tools that are commonly in use today.
+
Apple Inc.'s Macintosh OS X (pronounced "'''OS Ten'''") is the operating system distributed with Apple computers. It includes heavily used several programs by default, including [[Apple Mail]], a web browser called [[Apple Safari | Safari]], and an [[Apple Address Book]], and [[iCal]].  
  
Another advantage of ignoring file systems is that bulk_extractor can be used to process any digital media. We have used the program to process hard drives, SSDs, optical media, camera cards, cell phones, network packet dumps, and other kinds of digital information.
+
== EFI boot ==
 +
* Configuration in nvram
  
==Output Feature Files==
+
The firmware looks for the boot partition in nvram.
  
bulk_extractor now creates an output directory that includes:
+
Mac OS X EFI boot process supports both MZ-PE/COFF and EFI fat binary type [[Executable|executables]].
* '''ccn.txt''' -- Credit card numbers
+
<pre>
* '''ccn_track2.txt''' -- Credit card “track 2″ information
+
/com.apple.recovery.boot/boot.efi
* '''domain.txt''' -- Internet domains found on the drive, including dotted-quad addresses found in text.
+
/System/Library/CoreServices/boot.efi
* '''email.txt''' -- Email addresses
+
/usr/standalone/i386/boot.efi
* '''ether.txt''' -- Ethernet MAC addresses found through IP packet carving of swap files and compressed system hibernation files and file fragments.
+
</pre>
* '''exif.txt''' -- EXIFs from JPEGs and video segments. This feature file contains all of the EXIF fields, expanded as XML records.
+
* '''find.txt''' -- The results of specific regular expression search requests.
+
* '''ip.txt''' -- IP addresses found through IP packet carving.
+
* '''telephone.txt''' --- US and international telephone numbers.
+
* '''url.txt''' --- URLs, typically found in browser caches, email messages, and pre-compiled into executables.
+
* '''url_searches.txt''' --- A histogram of terms used in Internet searches from services such as Google, Bing, Yahoo, and others.
+
* '''wordlist.txt''' --- :A list of all “words” extracted from the disk, useful for password cracking.
+
* '''wordlist_*.txt''' --- The wordlist with duplicates removed, formatted in a form that can be easily imported into a popular password-cracking program.
+
* '''zip.txt''' --- A file containing information regarding every ZIP file component found on the media. This is exceptionally useful as ZIP files contain internal structure and ZIP is increasingly the compound file format of choice for a variety of products such as Microsoft Office
+
  
For each of the above, two additional files may be created:
+
== Disk image types ==
* '''*_stopped.txt''' --- bulk_extractor supports a stop list, or a list of items that do not need to be brought to the user’s attention. However rather than simply suppressing this information, which might cause something critical to be hidden, stopped entries are stored in the stopped files.
+
* '''*_histogram.txt''' --- bulk_extractor can also create histograms of features. This is important, as experience has shown that email addresses, domain names, URLs, and other information that appear more frequently on a hard drive or in a cell phone’s memory can be used to rapidly create a pattern of life report.
+
  
Bulk extractor also creates a file that captures the provenance of the run:
+
Mac OS X has support for various disk image types build-in, some of which are:
;report.xml
+
* read-write disk image (.dmg) some of which use the [[Raw Image Format]]
:A Digital Forensics XML report that includes information about the source media, how the bulk_extractor program was compiled and run, the time to process the digital evidence, and a meta report of the information that was found.
+
* [[Sparse Image format|Sparse disk image (.spareimage)]]
 +
* [[Sparse Bundle Image format|Sparse bundle disk image (.sparsebundle)]]
  
==Post-Processing==
+
== Burn Folder ==
  
We have developed four programs for post-processing the bulk_extractor output:
+
Mac OS X Burn Folder:
;bulk_diff.py
+
<pre>
:This program reports the differences between two bulk_extractor runs. The intent is to image a computer, run bulk_extractor on a disk image, let the computer run for a period of time, re-image the computer, run bulk_extractor on the second image, and then report the differences. This can be used to infer the user’s activities within a time period.
+
$NAME.fpbf
;cda_tool.py
+
</pre>
:This tool, currently under development, reads multiple bulk_extractor reports from multiple runs against multiple drives and performs a multi-drive correlation using Garfinkel’s Cross Drive Analysis technique. This can be used to automatically identify new social networks or to identify new members of existing networks.
+
;identify_filenames.py
+
:In the bulk_extractor feature file, each feature is annotated with the byte offset from the beginning of the image in which it was found. The program takes as input a bulk_extractor feature file and a DFXML file containing the locations of each file on the drive (produced with Garfinkel’s fiwalk program) and produces an annotated feature file that contains the offset, feature, and the file in which the feature was found.
+
;make_context_stop_list.py
+
:Although forensic analysts frequently make “stop lists”—for example, a lsit of email addresses that appear in the operating system and should therefore be ignored—such lists have a significant problem. Because it is relatively easy to get an email address into the binary of an open source application, ignoring all of these email addresses may make it possible to cloak email addresses from forensic analysis. Our solution is to create context-sensitive stop lists, in which the feature to be stopped is presented with the context in which it occures. The make_context_stop_list.py program takes the results of multiple bulk_extractor runs and creates a single context-sensitive stop list that can then be used to suppress features when found in a specific context. One such stop list constructed from Windows and Linux operating systems is available on the bulk extractor website.
+
  
== Download ==
+
This folder normally contains [[Mac OS X Alias Files|alias files]] (similar to LNK files under Windows). Which should have the following signature.
The current version of '''bulk_extractor''' is 1.3.  
+
<pre>
 +
00000000  62 6f 6f 6b 00 00 00 00  6d 61 72 6b 00 00 00 00  |book....mark....|
 +
</pre>
  
* Downloads are available at: http://digitalcorpora.org/downloads/bulk_extractor/
+
These [[Mac OS X Alias Files|alias files]] contain additional date and time values.
* A WIndows installer with the GUI can be downloaded from: http://digitalcorpora.org/downloads/bulk_extractor/executables/be_installer-1.3.exe
+
  
== Bibliography ==
+
Also check the following files for references to deleted .fpbf paths:
=== Academic Publications ===
+
<pre>
# Garfinkel, Simson, [http://simson.net/clips/academic/2013.COSE.bulk_extractor.pdf Digital media triage with bulk data analysis and bulk_extractor]. Computers and Security 32: 56-72 (2013)
+
/Users/$USERNAME/Library/Preferences/com.apple.finder.plist
# Beverly, Robert, Simson Garfinkel and Greg Cardwell, [http://simson.net/clips/academic/2011.DFRWS.ipcarving.pdf "Forensic Carving of Network Packets and Associated Data Structures"], DFRWS 2011, Aug. 1-3, 2011, New Orleans, LA. BEST PAPER AWARD (Acceptance rate: 23%, 14/62)
+
/Users/$USERNAME/Library/Preferences/com.apple.sidebarlists.plist
#Garfinkel, S., [http://simson.net/clips/academic/2006.DFRWS.pdf Forensic Feature Extraction and Cross-Drive Analysis,]The 6th Annual Digital Forensic Research Workshop Lafayette, Indiana, August 14-16, 2006. (Acceptance rate: 43%, 16/37)
+
</pre>
  
===YouTube===
+
Actual burning of optical media is logged in:
'''[http://www.youtube.com/results?search_query=bulk_extractor search YouTube] for bulk_extractor videos'''
+
<pre>
* [http://www.youtube.com/watch?v=odvDTGA7rYI Simson Garfinkel speaking at CERIAS about bulk_extractor]
+
/var/log/system.log
* [http://www.youtube.com/watch?v=wTBHM9DeLq4 BackTrack 5 with bulk_extractor]
+
/Users/$USERNAME/Library/Logs/DiscRecording.log
* [http://www.youtube.com/watch?v=QVfYOvhrugg Ubuntu 12.04 forensics with bulk_extractor]
+
/private/var/.logs_exporter/cache/Users/$USERNAME/Library/Logs/DiscRecording.log
* [http://www.youtube.com/watch?v=57RWdYhNvq8 Social Network forensics with bulk_extractor]
+
</pre>
  
===Tutorials===
+
== HFS/HFS+ date and time values ==
# [http://simson.net/ref/2012/2012-08-08%20bulk_extractor%20Tutorial.pdf Using bulk_extractor for digital forensics triage and cross-drive analysis], DFRWS 2012
+
 
 +
In HFS+ date and time values are stored in an unsigned 32-bit integer containing the number of seconds since January 1, 1904 at 00:00:00 (midnight) UTC (GMT). This is slightly different from HFS where the date and time value are stored using the local time. The maximum representable date is February 6, 2040 at 06:28:15 UTC (GMT). The date values do not account for leap seconds. They do include a leap day in every year that is evenly divisible by four. This is sufficient given that the range of representable dates does not contain 1900 or 2100, neither of which have leap days. Also see: [http://web.archive.org/web/20090214212148/http://developer.apple.com/technotes/tn/tn1150.html Technical Note TN1150 - HFS Plus Volume Format]
 +
 
 +
Converting HFS/HFS+ date and time values with Python:
 +
<pre>
 +
import datetime
 +
 
 +
print datetime.datetime( 1904, 1, 1 ) + datetime.timedelta( seconds=0xCBDAF25B )
 +
</pre>
 +
 
 +
== Launch Agents ==
 +
System-wide:
 +
<pre>
 +
/Library/LaunchAgents
 +
/System/Library/LaunchAgents
 +
</pre>
 +
 
 +
Per user:
 +
<pre>
 +
/Users/$USERNAME/Library/LaunchAgents
 +
</pre>
 +
 
 +
These directories contain  [[Property list (plist)]] files.
 +
 
 +
== Launch Daemons ==
 +
System-wide:
 +
<pre>
 +
/Library/LaunchDaemons
 +
/System/Library/LaunchDaemons
 +
</pre>
 +
 
 +
These directories contain [[Property list (plist)]] files.
 +
 
 +
== Startup Items ==
 +
<pre>
 +
/Library/StartupItems/
 +
/System/Library/StartupItems/
 +
</pre>
 +
 
 +
== Crash Reporter ==
 +
<pre>
 +
/Library/Application Support/CrashReporter
 +
</pre>
 +
 
 +
Contains text files named .crash, .diag, .spin
 +
 
 +
== Diagnostic Reports ==
 +
<pre>
 +
/Library/Logs/DiagnosticReports
 +
</pre>
 +
 
 +
== Internet Plug-Ins ==
 +
System-wide:
 +
<pre>
 +
/Library/Internet Plug-Ins
 +
</pre>
 +
 
 +
Per user:
 +
<pre>
 +
/Users/$USERNAME/Library/Internet Plug-Ins
 +
</pre>
 +
 
 +
== Quarantine event database ==
 +
See [http://menial.co.uk/blog/2011/06/16/mac-quarantine-event-database/]
 +
 
 +
Snow Leopard and earlier
 +
<pre>
 +
/Users/$USER/Library/Preferences/com.apple.LaunchServices.QuarantineEvents
 +
</pre>
 +
 
 +
<pre>
 +
SELECT datetime(LSQuarantineTimeStamp + 978307200, "unixepoch") as LSQuarantineTimeStamp, LSQuarantineAgentName, LSQuarantineOriginURLString, LSQuarantineDataURLString from LSQuarantineEvent;
 +
</pre>
 +
 
 +
Lion and later
 +
<pre>
 +
/Users/$USER/Library/Preferences/com.apple.LaunchServices.QuarantineEventsV2
 +
</pre>
 +
 
 +
== sleepimage ==
 +
This file is similar to the hibernation file on Windows.
 +
<pre>
 +
/private/var/vm/sleepimage
 +
</pre>
 +
 
 +
Also see: [http://osxdaily.com/2010/10/11/sleepimage-mac/]
 +
 
 +
== Last shutdown logs ==
 +
<pre>
 +
/private/var/log/com.apple.launchd/launchd-shutdown.system.log
 +
/private/var/log/com.apple.launchd/launchd-shutdown.system.log.1
 +
</pre>
 +
 
 +
== Package Files (.PKG) ==
 +
Package Files (.PKG) are XAR archives [http://en.wikipedia.org/wiki/Xar_(archiver)] that contain a cpio archive and metadata [http://s.sudre.free.fr/Stuff/Ivanhoe/FLAT.html].
 +
 
 +
== Also see ==
 +
* [[MacOS Process Monitoring]]
 +
* [[Acquiring a MacOS System with Target Disk Mode]]
 +
* [[Converting Binary Plists]]
 +
* [[FileVault Disk Encryption]]
 +
* [[File Vault]]
 +
 
 +
=== Formats ===
 +
* [[Basic Security Module (BSM) file format]]
 +
* [[Property list (plist)]]
 +
 
 +
== External Links ==
 +
* [http://www.apple.com/macosx/ Official website]
 +
* [http://en.wikipedia.org/wiki/OS_X Wikipedia entry on OS X]
 +
* [http://menial.co.uk/blog/2011/06/16/mac-quarantine-event-database/ Quarantine event database]
 +
* [http://www2.tech.purdue.edu/cit/Courses/cit556/readings/MacForensicsCraiger.pdf Mac Forensics: Mac OS X and the HFS+ File System] by P. Craiger
 +
* [http://web.me.com/driley/iWeb/Previous_files/Directory_Services_Overview.pdf Mac OS X Directory Services Integration including Active Directory]
 +
* [http://digitalinvestigation.wordpress.com/2012/04/04/geek-post-nskeyedarchiver-files-what-are-they-and-how-can-i-use-them/ NSKeyedArchiver files – what are they, and how can I use them?]
 +
* [http://krypted.com/mac-os-x/command-line-alf-on-mac-os-x/ Command Line ALF on Mac OS X]
 +
* [http://newosxbook.com/DMG.html Demystifying the DMG File Format]
 +
* [https://code.google.com/p/mac-security-tips/wiki/ALL_THE_TIPS mac-security-tips]
 +
 
 +
=== Apple Examiner ===
 +
* [http://www.appleexaminer.com/ The Apple Examiner]
 +
* [http://www.appleexaminer.com/MacsAndOS/Analysis/USBOSX/USBOSX.html USB Entries on OS X]
 +
* [http://www.appleexaminer.com/Downloads/MacForensics.pdf Macintosh Forensics - A Guide for the Forensically Sound Examination of a Macintosh Computer] by Ryan R. Kubasiak
 +
 
 +
=== EFI ===
 +
* [http://refit.sourceforge.net/info/boot_process.html The Intel Mac boot process], by the [[rEFIt|rEFIt project]]
 +
* [http://ho.ax/posts/2012/02/carving-up-efi-fat-binaries/ Carving up EFI fat binaries], by snare, February 24, 2012
 +
 
 +
=== iCloud ===
 +
* [http://support.apple.com/kb/HT4865?viewlocale=en_US&locale=en_US iCloud: iCloud security and privacy overview]
 +
 
 +
[[Category:Mac OS X]]
 +
[[Category:Operating systems]]

Revision as of 06:04, 25 June 2014

Information icon.png

Please help to improve this article by expanding it.
Further information might be found on the discussion page.

Apple Inc.'s Macintosh OS X (pronounced "OS Ten") is the operating system distributed with Apple computers. It includes heavily used several programs by default, including Apple Mail, a web browser called Safari, and an Apple Address Book, and iCal.

EFI boot

  • Configuration in nvram

The firmware looks for the boot partition in nvram.

Mac OS X EFI boot process supports both MZ-PE/COFF and EFI fat binary type executables.

/com.apple.recovery.boot/boot.efi
/System/Library/CoreServices/boot.efi
/usr/standalone/i386/boot.efi

Disk image types

Mac OS X has support for various disk image types build-in, some of which are:

Burn Folder

Mac OS X Burn Folder:

$NAME.fpbf

This folder normally contains alias files (similar to LNK files under Windows). Which should have the following signature.

00000000  62 6f 6f 6b 00 00 00 00  6d 61 72 6b 00 00 00 00  |book....mark....|

These alias files contain additional date and time values.

Also check the following files for references to deleted .fpbf paths:

/Users/$USERNAME/Library/Preferences/com.apple.finder.plist
/Users/$USERNAME/Library/Preferences/com.apple.sidebarlists.plist

Actual burning of optical media is logged in:

/var/log/system.log
/Users/$USERNAME/Library/Logs/DiscRecording.log
/private/var/.logs_exporter/cache/Users/$USERNAME/Library/Logs/DiscRecording.log

HFS/HFS+ date and time values

In HFS+ date and time values are stored in an unsigned 32-bit integer containing the number of seconds since January 1, 1904 at 00:00:00 (midnight) UTC (GMT). This is slightly different from HFS where the date and time value are stored using the local time. The maximum representable date is February 6, 2040 at 06:28:15 UTC (GMT). The date values do not account for leap seconds. They do include a leap day in every year that is evenly divisible by four. This is sufficient given that the range of representable dates does not contain 1900 or 2100, neither of which have leap days. Also see: Technical Note TN1150 - HFS Plus Volume Format

Converting HFS/HFS+ date and time values with Python:

import datetime

print datetime.datetime( 1904, 1, 1 ) + datetime.timedelta( seconds=0xCBDAF25B )

Launch Agents

System-wide:

/Library/LaunchAgents
/System/Library/LaunchAgents

Per user:

/Users/$USERNAME/Library/LaunchAgents

These directories contain Property list (plist) files.

Launch Daemons

System-wide:

/Library/LaunchDaemons
/System/Library/LaunchDaemons

These directories contain Property list (plist) files.

Startup Items

/Library/StartupItems/
/System/Library/StartupItems/

Crash Reporter

/Library/Application Support/CrashReporter

Contains text files named .crash, .diag, .spin

Diagnostic Reports

/Library/Logs/DiagnosticReports

Internet Plug-Ins

System-wide:

/Library/Internet Plug-Ins

Per user:

/Users/$USERNAME/Library/Internet Plug-Ins

Quarantine event database

See [1]

Snow Leopard and earlier

/Users/$USER/Library/Preferences/com.apple.LaunchServices.QuarantineEvents
SELECT datetime(LSQuarantineTimeStamp + 978307200, "unixepoch") as LSQuarantineTimeStamp, LSQuarantineAgentName, LSQuarantineOriginURLString, LSQuarantineDataURLString from LSQuarantineEvent;

Lion and later

/Users/$USER/Library/Preferences/com.apple.LaunchServices.QuarantineEventsV2

sleepimage

This file is similar to the hibernation file on Windows.

/private/var/vm/sleepimage

Also see: [2]

Last shutdown logs

/private/var/log/com.apple.launchd/launchd-shutdown.system.log
/private/var/log/com.apple.launchd/launchd-shutdown.system.log.1

Package Files (.PKG)

Package Files (.PKG) are XAR archives [3] that contain a cpio archive and metadata [4].

Also see

Formats

External Links

Apple Examiner

EFI

iCloud