Difference between pages "File Carving:SmartCarving" and "Forensic Live CD issues"

From Forensics Wiki
(Difference between pages)
Jump to: navigation, search
 
m
 
Line 1: Line 1:
'''SmartCarving''' is a [[File Carving|file carving]] technique to recover fragmented files first proposed by [[User:PashaPal|A. Pal]], T. Sencar and [[User:NasirMemon|N. Memon]] in DFRWS 2008. The term '''Smart Carving''' was already used in [http://sandbox.dfrws.org/2006/mora/dfrws2006.pdf]
+
== The problem ==
  
SmartCarving utilizes a combination of structure based validation along with validation of each file's unique content. Results for the SmartCarving technique
+
[[Tools#Forensics_Live_CDs | Forensic Linux Live CD distributions]] are widely used during computer forensic investigations. Currently, many vendors of such Live CD distributions spread false claims that their distributions "do not touch anything", "write protect everything" and so on. Community-developed distributions are not exception here, unfortunately. Finally, it turns out that many forensic Linux Live CD distributions are not tested properly and there are no suitable test cases developed.
were demonstrated on fragmented jpegs in the DFRWS 2006 and DFRWS 2007 challenges. From these two challenges SmartCarving was able
+
to recover all but one fragmented jpeg file.  
+
  
==History==
+
== Another side of the problem ==
[[User:NasirMemon|Memon]] et al.[1] presented an efficient algorithm based on a greedy heuristic and alpha-beta pruning for reassembling fragmented images.
+
Building on this work, [[User:NasirMemon|Memon]] et al.[2] researched and introduced sequential hypothesis testing as a an effective mechanism for detecting fragmentation points of file. This paper won the best paper award for DFRWS 2008. The techniques presented in the paper were the foundation for the overall SmartCarving design.
+
  
==Details==
+
Another side of the problem of insufficient testing of forensic Live CD distributions is that many users do not know what happens "under the hood" of such distributions and cannot adequately test them.
After identifying a header block of a specific file type, for example, jpeg, a SmartCarver will analyze each subsequent block to determine if it
+
belongs or does not belong to the starting block. If a block is determined not to belong, then the file is assumed to be fragmented and the
+
SmartCarving algorithm looks for the next fragment by matching the data of other available blocks with the first fragment. This process can be
+
done in parallel for many files.
+
  
==Applications==
+
=== Example ===
There are currently two applications available that utilize SmartCarving, both produced by Digital Assembly:
+
* [[Adroit Photo Forensics]]
+
* Adroit Photo Recovery
+
  
== References ==
+
For example, [http://forensiccop.blogspot.com/2009/10/forensic-cop-journal-13-2009.html ''Forensic Cop Journal'' (Volume 1(3), Oct 2009)] describes a test case when an Ext3 file system was mounted using "-o ro" mount flag as a way to write protect the data. The article says that all tests were successful (i.e. no data modification was found after unmounting the file system), but it is known that damaged (i.e not properly unmounted) Ext3 file systems cannot be write protected using only "-o ro" mount flags (write access will be enabled during file system recovery).
* A. Pal and N. Memon, [http://digital-assembly.com/technology/research/pubs/ieee-trans-2006.pdf "Automated reassembly of file fragmented images using greedy algorithms"] in IEEE Transactions on Image processing, February 2006, pp 385­393
+
* A. Pal, T. Sencar and N. Memon, [http://digital-assembly.com/technology/research/pubs/dfrws2008.pdf "Detecting File Fragmentation Point Using Sequential Hypothesis Testing"], Digital Investigations, Fall 2008
+
  
==External links==
+
And the question is: will many users test damaged Ext3 file system (together with testing the clean one) when validating their favourite forensic Live CD distribution? My answer is "no", because many users are unaware of such traits.
* [http://digital-assembly.com/products/adroit-photo-recovery/ Adroit Photo Recovery]
+
 
* [http://digital-assembly.com/products/adroit-photo-forensics/ Adroit Photo Forensics]
+
== Problems ==
* [http://digital-assembly.com/technology/ Link to SmartCarving Technology and Research]
+
 
* [http://digital-assembly.com Digital Assembly]
+
Here is a list of common problems of forensic Linux Live CD distributions that can be used by developers and users for testing purposes. Each problem is followed by an up to date list of distributions affected.
 +
 
 +
=== Journaling file systems updates ===
 +
 
 +
When mounting (and unmounting) several journaling file systems with only "-o ro" mount flag a different number of data writes may occur. Here is a list of such file systems:
 +
 
 +
{| class="wikitable" border="1"
 +
|-
 +
!  File system
 +
!  When data writes happen
 +
!  Notes
 +
|-
 +
|  Ext3
 +
|  File system requires journal recovery
 +
|  To disable recovery: use "noload" flag, or use "ro,loop" flags, or use "ext2" file system type
 +
|-
 +
|  Ext4
 +
|  File system requires journal recovery
 +
|  To disable recovery: use "noload" flag, or use "ro,loop" flags, or use "ext2" file system type
 +
|-
 +
|  ReiserFS
 +
|  In most cases
 +
|  "nolog" flag does not work (see ''man mount''). To disable journal updates: use "ro,loop" flags
 +
|-
 +
|  XFS
 +
|  Always
 +
|  "norecovery" flag does not help. To disable data writes: use "ro,loop" flags. The bug was fixed in recent 2.6 kernels.
 +
|}
 +
 
 +
Incorrect mount flags can be used to mount a file system on evidentiary media during the boot process or during the file system preview process. As described above, this may result in modification of a file system's data. For example, several Ubuntu-based forensic Linux Live CD distributions mount Ext3/4 file systems on fixed media (e.g. hard drives) during execution of ''initrd'' scripts (these scripts mount every supported file system type on every supported media type using only "-o ro" flag in order to find a root file system image).
 +
 
 +
List of distributions that recover Ext3 (and sometimes Ext4) file systems during the boot:
 +
 
 +
{| class="wikitable" border="1"
 +
|-
 +
!  Distribution
 +
!  Version
 +
|-
 +
|  Helix3
 +
|  2009R1
 +
|-
 +
|  SMART Linux (Ubuntu)
 +
|  2010-01-20
 +
|-
 +
|  FCCU GNU/Linux Forensic Boot CD
 +
|  12.1
 +
|-
 +
|  SPADA
 +
|  4
 +
|}
 +
 
 +
=== Root file system spoofing ===
 +
 
 +
Lets look at Casper scripts (that are used by initrd). Here is a function (see ''scripts/casper'' in the initrd image of Ubuntu-based forensic Linux Live CD distributions) that is used to search for a root file system image during the early stage of boot process:
 +
 
 +
find_livefs() {
 +
    timeout="${1}"
 +
    # first look at the one specified in the command line
 +
    if [ ! -z "${LIVEMEDIA}" ]; then
 +
        if check_dev "null" "${LIVEMEDIA}" "skip_uuid_check"; then
 +
            return 0
 +
        fi
 +
    fi
 +
    # don't start autodetection before timeout has expired
 +
    if [ -n "${LIVEMEDIA_TIMEOUT}" ]; then
 +
        if [ "${timeout}" -lt "${LIVEMEDIA_TIMEOUT}" ]; then
 +
            return 1
 +
        fi
 +
    fi
 +
    # or do the scan of block devices
 +
    for sysblock in $(echo /sys/block/* | tr ' ' '\n' | grep -v loop | grep -v ram); do
 +
        devname=$(sys2dev "${sysblock}")
 +
        fstype=$(get_fstype "${devname}")
 +
        if /lib/udev/cdrom_id ${devname} > /dev/null; then
 +
            if check_dev "null" "${devname}" ; then
 +
                return 0
 +
            fi
 +
        elif is_nice_device "${sysblock}" ; then
 +
            for dev in $(subdevices "${sysblock}"); do
 +
                if check_dev "${dev}" ; then
 +
                    return 0
 +
                fi
 +
            done
 +
        elif [ "${fstype}" = "squashfs" -o \
 +
                "${fstype}" = "ext3" -o \
 +
                "${fstype}" = "ext2" ]; then
 +
            # This is an ugly hack situation, the block device has
 +
            # an image directly on it.  It's hopefully
 +
            # casper, so take it and run with it.
 +
            ln -s "${devname}" "${devname}.${fstype}"
 +
            echo "${devname}.${fstype}"
 +
            return 0
 +
        fi
 +
    done
 +
    return 1
 +
}
 +
 
 +
=== Swap space activation ===
 +
 
 +
=== Incorrect automount policy for removable media ===
 +
 
 +
=== Incorrect write-blocking approach ===
 +
 
 +
=== Software RAID (Linux RAID) activation ===

Revision as of 08:17, 3 February 2010

Contents

The problem

Forensic Linux Live CD distributions are widely used during computer forensic investigations. Currently, many vendors of such Live CD distributions spread false claims that their distributions "do not touch anything", "write protect everything" and so on. Community-developed distributions are not exception here, unfortunately. Finally, it turns out that many forensic Linux Live CD distributions are not tested properly and there are no suitable test cases developed.

Another side of the problem

Another side of the problem of insufficient testing of forensic Live CD distributions is that many users do not know what happens "under the hood" of such distributions and cannot adequately test them.

Example

For example, Forensic Cop Journal (Volume 1(3), Oct 2009) describes a test case when an Ext3 file system was mounted using "-o ro" mount flag as a way to write protect the data. The article says that all tests were successful (i.e. no data modification was found after unmounting the file system), but it is known that damaged (i.e not properly unmounted) Ext3 file systems cannot be write protected using only "-o ro" mount flags (write access will be enabled during file system recovery).

And the question is: will many users test damaged Ext3 file system (together with testing the clean one) when validating their favourite forensic Live CD distribution? My answer is "no", because many users are unaware of such traits.

Problems

Here is a list of common problems of forensic Linux Live CD distributions that can be used by developers and users for testing purposes. Each problem is followed by an up to date list of distributions affected.

Journaling file systems updates

When mounting (and unmounting) several journaling file systems with only "-o ro" mount flag a different number of data writes may occur. Here is a list of such file systems:

File system When data writes happen Notes
Ext3 File system requires journal recovery To disable recovery: use "noload" flag, or use "ro,loop" flags, or use "ext2" file system type
Ext4 File system requires journal recovery To disable recovery: use "noload" flag, or use "ro,loop" flags, or use "ext2" file system type
ReiserFS In most cases "nolog" flag does not work (see man mount). To disable journal updates: use "ro,loop" flags
XFS Always "norecovery" flag does not help. To disable data writes: use "ro,loop" flags. The bug was fixed in recent 2.6 kernels.

Incorrect mount flags can be used to mount a file system on evidentiary media during the boot process or during the file system preview process. As described above, this may result in modification of a file system's data. For example, several Ubuntu-based forensic Linux Live CD distributions mount Ext3/4 file systems on fixed media (e.g. hard drives) during execution of initrd scripts (these scripts mount every supported file system type on every supported media type using only "-o ro" flag in order to find a root file system image).

List of distributions that recover Ext3 (and sometimes Ext4) file systems during the boot:

Distribution Version
Helix3 2009R1
SMART Linux (Ubuntu) 2010-01-20
FCCU GNU/Linux Forensic Boot CD 12.1
SPADA 4

Root file system spoofing

Lets look at Casper scripts (that are used by initrd). Here is a function (see scripts/casper in the initrd image of Ubuntu-based forensic Linux Live CD distributions) that is used to search for a root file system image during the early stage of boot process:

find_livefs() {
    timeout="${1}"
    # first look at the one specified in the command line
    if [ ! -z "${LIVEMEDIA}" ]; then
        if check_dev "null" "${LIVEMEDIA}" "skip_uuid_check"; then
            return 0
        fi
    fi
    # don't start autodetection before timeout has expired
    if [ -n "${LIVEMEDIA_TIMEOUT}" ]; then
        if [ "${timeout}" -lt "${LIVEMEDIA_TIMEOUT}" ]; then
            return 1
        fi
    fi
    # or do the scan of block devices
    for sysblock in $(echo /sys/block/* | tr ' ' '\n' | grep -v loop | grep -v ram); do
        devname=$(sys2dev "${sysblock}")
        fstype=$(get_fstype "${devname}")
        if /lib/udev/cdrom_id ${devname} > /dev/null; then
            if check_dev "null" "${devname}" ; then
                return 0
            fi
        elif is_nice_device "${sysblock}" ; then
            for dev in $(subdevices "${sysblock}"); do
                if check_dev "${dev}" ; then
                    return 0
                fi
            done
        elif [ "${fstype}" = "squashfs" -o \
                "${fstype}" = "ext3" -o \
                "${fstype}" = "ext2" ]; then
            # This is an ugly hack situation, the block device has
            # an image directly on it.  It's hopefully
            # casper, so take it and run with it.
            ln -s "${devname}" "${devname}.${fstype}"
            echo "${devname}.${fstype}"
            return 0
        fi
    done
    return 1
}

Swap space activation

Incorrect automount policy for removable media

Incorrect write-blocking approach

Software RAID (Linux RAID) activation