Difference between pages "File Systems" and "Ddrescue"

From ForensicsWiki
(Difference between pages)
Jump to: navigation, search
m
 
(linked to my blog containing some useful scripts/oneliners -- not sure if linking is allowed this way... else lmk!)
 
Line 1: Line 1:
= Conventional File Systems =  
+
{{Infobox_Software |
 +
  name = ddrescure |
 +
  maintainer = [[Antonio Diaz Diaz]]|
 +
  os = {{Linux}}|
 +
  genre = {{Disk imaging}} |
 +
  license = {{GPL}} |
 +
  website = [http://www.gnu.org/software/ddrescue/ddrescue.html http://www.gnu.org/software/ddrescue/ddrescue.html] |
 +
}}
  
; [[ext2]], [[ext3]]
+
'''ddrescue''' is a raw disk imaging tool that "copies data from one file or block device to another, trying hard to rescue data in case of read errors."  The application is developed as part of the GNU project and has written with UNIX/Linux in mind.
: ext2 was introduced with [[Linux]]. ext3 is a journaled version of ext2 which allows for speedy disk recovery after a crash.
+
  
; [[FAT]]
+
'''ddrescue''' and '''[[dd_rescue]]''' are completely different programs which share no development between them. The two projects are not related in any way except that they both attempt to enhance the standard [[dd]] tool and coincidentally chose similar names for their new programs.
: Originally used by [[MS-DOS]]. Includes [[FAT12]] (for floppy disks), [[FAT16]] and [[FAT32]].
+
  
; [[ffs]]
+
From the [[ddrescue]] info pages:
: The '''Fast File System''' used by some BSD versions of [[UNIX]] and from which [[UFS]] was derived supporting faster disk access and [[symbolic link]]s like ffs.
+
<blockquote>
 +
GNU ddrescue is a data recovery tool. It copies data from one file or block device (hard disc, cdrom, etc) to another, trying hard to rescue data in case of read errors.<br><br>
  
; [[HFS]]
+
Ddrescue does not truncate the output file if not asked to. So, every time you run it on the same output file, it tries to fill in the gaps.<br><br>
: Used by [[Apple]] systems, it has been succeed by [[HFS Plus|HFS+]].
+
  
; [[JFS]]
+
The basic operation of ddrescue is fully automatic. That is, you don't have to wait for an error, stop the program, read the log, run it in reverse mode, etc.<br><br>
: IBM's Journaled File System introduced with their flavor of UNIX (AIX)
+
  
; [[NTFS]]
+
If you use the logfile feature of ddrescue, the data is rescued very efficiently (only the needed blocks are read). Also you can interrupt the rescue at any time and resume it later at the same point.<br><br>
: The '''New Technology File System''', introduced by [[Microsoft]] with [[Windows NT]] 4.0. Now used on [[Windows XP]].
+
  
; [[reiserfs]]
+
Automatic merging of backups: If you have two or more damaged copies of a file, cdrom, etc, and run ddrescue on all of them, one at a time, with the same output file, you will probably obtain a complete and error-free file. This is so because the probability of having damaged areas at the same places on different input files is very low. Using
: A journaling filesystem for Linux.
+
the logfile, only the needed blocks are read from the second and successive copies.
 +
</blockquote>
  
; [[UFS]]
+
== Installation ==
: The '''Unix File System''', introduced with [[UNIX]].
+
  
; [[XFS]]
+
=== Bootable CD ===
: [[SGI]]’s high performance journaling filesystem that originated on their [[IRIX]] (flavor of [[UNIX]]) platform. XFS supports variable blocking sizes, is extent based, and makes extensive use of [[Btree]]s to facilitate both performance and scalability. Additionally, support is also provided for real-time environments.
+
ddrescue is available on bootable rescue cds such as SystemRescueCd http://www.sysresccd.org/Main_Page.
 +
=== Debian and Ubuntu ===
 +
The package 'ddrescue' in Debian and Ubuntu is actually [[dd_rescue]], another dd-like program which does not maintain a recovery log. The correct package is gddrescue.
  
= Cryptographic File Systems =
+
Debian
 +
<blockquote>
 +
aptitude install gddrescue
 +
</blockquote>
 +
Ubuntu
 +
<blockquote>
 +
sudo apt-get install gddrescue
 +
</blockquote>
 +
=== Gentoo ===
 +
<blockquote>
 +
emerge ddrescue
 +
</blockquote>
 +
== Partition recovery ==
  
'''Cryptographic file systems,''' also known as encrypted file systems, encrypt information before it is stored on the media. Some of these file systems store encrypted files directly. Others are better thought of as device drivers, which are then used to store some of the file systems discussed above.
+
=== Kernel 2.6.3+ & ddrescue 1.4+ ===
 +
'ddrescue --direct' will open the input with the O_DIRECT option for uncached reads. 'raw devices' are not needed on newer kernels. For older kernels see below.
  
; [[File Vault]]
+
First you copy as much data as possible, without retrying or splitting sectors:
: A clever user interface to [[Apple]]'s encrypted disk images. Uses the ".sparseimage" extension on disk files.
+
<blockquote>
 +
ddrescue --no-split /dev/hda1 imagefile logfile
 +
</blockquote>
  
; [[CFS]]
+
Now let it retry previous errors 3 times, using uncached reads:
: Matt Blaze's '''Cryptographic File System''' for [[Unix]].
+
<blockquote>
: [http://www.crypto.com/papers/cfskey.pdf Key Management in an Encrypting File System], Matt Blaze, USENIX Summer 1994 Technical Conference, Boston, MA, June 1994.
+
ddrescue --direct --max-retries=3 /dev/hda1 imagefile logfile
: [http://www.crypto.com/papers/cfs.pdf A Cryptographic File System for Unix], Matt Blaze, Proceedings of the First ACM Conference on Computer and Communications Security, Fairfax, VA, November 1993.
+
</blockquote>
  
; [[Windows Encrypted File System |EFS]]
+
If that fails you can try again but retrimmed, so it tries to reread full sectors:
: EFS is the Encrypted File System built into versions of Microsoft Windows.
+
<blockquote>
 +
ddrescue --direct --retrim  --max-retries=3 /dev/hda1 imagefile logfile
 +
</blockquote>
  
; [[NCryptfs]]
+
You can now use ddrescue (or normal dd) to copy the imagefile to a new partition on a new disk. Use the appropriate filesystem checkers (fsck, CHKDSK) to try to fix errors caused by the bad blocks. Be sure to keep the imagefile around. Just in case the filesystem is severely broken, and datacarving tools like testdisk need to to be used on the original image.
: http://www.fsl.cs.sunysb.edu/docs/ncryptfs/ncryptfs.pdf NCryptfs: A Secure and Convenient Cryptographic File System], Charles P. Wright, Michael C. Martino, and Erez Zadok, Stony Brook University ,USENIX 2003 Annual Technical Conference.
+
  
; [[TCFS]]
+
=== Before linux kernel 2.6.3 / 2.4.x ===
: '''Transparent Cryptographic File System'''.
+
In 2.6.3 the 'raw device' has been marked obsolete. On later kernels ddrescue will use O_DIRECT on the input to do uncached reads.
: http://www.tcfs.it/
+
: http://www.tcfs.it/docs/tcfs.ps
+
  
; [[SFS]]
+
First you copy as much data as possible, without retrying or splitting sectors:
: '''Secure File System'''.
+
<blockquote>
: http://atrey.karlin.mff.cuni.cz/~rebel/sfs/
+
ddrescue --no-split /dev/hda1 imagefile logfile
 +
</blockquote>
  
See also [[Full Disk Encryption]], which are disk- or applicance-based cryptographic file systems.
+
Now change over to raw device access. Let it retry previous errors 3 times, don't read past last block in logfile:
 +
<blockquote>
 +
modprobe raw<br>
 +
raw /dev/raw/raw1 /dev/hda1<br>
 +
ddrescue --max-retries=3 --complete-only /dev/raw/raw1 imagefile logfile
 +
</blockquote>
  
= External Links =
+
If that fails you can try again (still using raw) but retrimmed, so it tries to reread full sectors:
 +
<blockquote>
 +
ddrescue --retrim --max-retries=3 --complete-only /dev/raw/raw1 imagefile logfile
 +
</blockquote>
  
* http://en.wikipedia.org/wiki/File_system
+
You can now use ddrescue (or normal dd) to copy the imagefile to a new partition on a new disk. Use the appropriate filesystem checkers (fsck, CHKDSK) to try to fix errors caused by the bad blocks. Be sure to keep the imagefile around. Just in case the filesystem is severely broken, and datacarving tools like testdisk need to to be used on the original image.
* http://en.wikipedia.org/wiki/List_of_file_systems
+
 
* http://en.wikipedia.org/wiki/Comparison_of_file_systems
+
At the end you may want to unbind the raw device:
 +
<blockquote>
 +
raw /dev/raw/raw1 0 0
 +
</blockquote>
 +
 
 +
== Examples ==
 +
 
 +
These two examples are taken directly from the [[ddrescue]] info pages.
 +
 
 +
Example 1: Rescue an ext2 partition in /dev/hda2 to /dev/hdb2
 +
 
 +
'''Please Note:''' This will overwrite ALL data on the partition you are copying to. If you do not want to do that, rather create an image of the partition to be rescued.
 +
<blockquote>
 +
ddrescue -r3 /dev/hda2 /dev/hdb2 logfile<br>
 +
e2fsck -v -f /dev/hdb2<br>
 +
mount -t ext2 -o ro /dev/hdb2 /mnt<br>
 +
</blockquote>
 +
 
 +
Example 2: Rescue a CD-ROM in /dev/cdrom
 +
<blockquote>
 +
ddrescue -b 2048 /dev/cdrom cdimage logfile
 +
</blockquote>
 +
write cdimage to a blank CD-ROM
 +
 
 +
 
 +
This example is derived from the ddrescue manual.
 +
 
 +
Example 3: Rescue an entire hard disk /dev/sda to another disk /dev/sdb
 +
 
 +
copy the error free areas first
 +
ddrescue -n /dev/sda /dev/sdb rescue.log
 +
attempt to recover any bad sectors
 +
ddrescue -r 1 /dev/sda /dev/sdb rescue.log
 +
 
 +
== Options ==
 +
 
 +
-h, --help
 +
    display this help and exit
 +
-V, --version
 +
    output version information and exit
 +
-b, --block-size=<bytes>
 +
    hardware block size of input device [512]
 +
-B, --binary-prefixes
 +
    show binary multipliers in numbers [default SI]
 +
-c, --cluster-size=<blocks>
 +
    hardware blocks to copy at a time [128]
 +
-C, --complete-only
 +
    do not read new data beyond logfile limits
 +
-d, --direct
 +
    use direct disc access for input file
 +
-D, --synchronous
 +
    use synchronous writes for output file
 +
-e, --max-errors=<n>
 +
    maximum number of error areas allowed
 +
-F, --fill=<types>
 +
    fill given type areas with infile data (?*/-+)
 +
-g, --generate-logfile
 +
    generate approximate logfile from partial copy
 +
-i, --input-position=<pos>
 +
    starting position in input file [0]
 +
-n, --no-split
 +
    do not try to split or retry error areas
 +
-o, --output-position=<pos>
 +
    starting position in output file [ipos]
 +
-q, --quiet
 +
    quiet operation
 +
-r, --max-retries=<n>
 +
    exit after given retries (-1=infinity) [0]
 +
-R, --retrim
 +
    mark all error areas as non-trimmed
 +
-s, --max-size=<bytes>
 +
    maximum size of data to be copied
 +
-S, --sparse
 +
    use sparse writes for output file
 +
-t, --truncate
 +
    truncate output file
 +
-v, --verbose
 +
    verbose operation
 +
 
 +
Numbers may be followed by a multiplier: b = blocks, k = kB = 10^3 = 1000, Ki = KiB = 2^10 = 1024, M = 10^6, Mi = 2^20, G = 10^9, Gi = 2^30, etc...
 +
 
 +
 
 +
== Cygwin ==
 +
 
 +
As of release 1.4-rc1, it can be compiled directly in [[Cygwin]] [http://en.wikipedia.org/wiki/Out_of_the_box Out of the Box]. Precompiled packages are available in the [http://cygwin.com/packages/ Cygwin distribution]. This makes it usable natively on [[Windows]] systems.
 +
 
 +
== See also ==
 +
 
 +
* [[aimage]]
 +
* [[Blackbag]]
 +
* [[dcfldd]]
 +
* [[dd]]
 +
* [[dd_rescue]]
 +
* [[sdd]]
 +
 
 +
== Other Resources ==
 +
[[http://pfuender.net/?p=80|Useful code-snippets for DDrescue]]

Revision as of 08:09, 25 June 2010

ddrescure
Maintainer: Antonio Diaz Diaz
OS: Linux
Genre: Disk imaging
License: GPL
Website: http://www.gnu.org/software/ddrescue/ddrescue.html

ddrescue is a raw disk imaging tool that "copies data from one file or block device to another, trying hard to rescue data in case of read errors." The application is developed as part of the GNU project and has written with UNIX/Linux in mind.

ddrescue and dd_rescue are completely different programs which share no development between them. The two projects are not related in any way except that they both attempt to enhance the standard dd tool and coincidentally chose similar names for their new programs.

From the ddrescue info pages:

GNU ddrescue is a data recovery tool. It copies data from one file or block device (hard disc, cdrom, etc) to another, trying hard to rescue data in case of read errors.

Ddrescue does not truncate the output file if not asked to. So, every time you run it on the same output file, it tries to fill in the gaps.

The basic operation of ddrescue is fully automatic. That is, you don't have to wait for an error, stop the program, read the log, run it in reverse mode, etc.

If you use the logfile feature of ddrescue, the data is rescued very efficiently (only the needed blocks are read). Also you can interrupt the rescue at any time and resume it later at the same point.

Automatic merging of backups: If you have two or more damaged copies of a file, cdrom, etc, and run ddrescue on all of them, one at a time, with the same output file, you will probably obtain a complete and error-free file. This is so because the probability of having damaged areas at the same places on different input files is very low. Using the logfile, only the needed blocks are read from the second and successive copies.

Installation

Bootable CD

ddrescue is available on bootable rescue cds such as SystemRescueCd http://www.sysresccd.org/Main_Page.

Debian and Ubuntu

The package 'ddrescue' in Debian and Ubuntu is actually dd_rescue, another dd-like program which does not maintain a recovery log. The correct package is gddrescue.

Debian

aptitude install gddrescue

Ubuntu

sudo apt-get install gddrescue

Gentoo

emerge ddrescue

Partition recovery

Kernel 2.6.3+ & ddrescue 1.4+

'ddrescue --direct' will open the input with the O_DIRECT option for uncached reads. 'raw devices' are not needed on newer kernels. For older kernels see below.

First you copy as much data as possible, without retrying or splitting sectors:

ddrescue --no-split /dev/hda1 imagefile logfile

Now let it retry previous errors 3 times, using uncached reads:

ddrescue --direct --max-retries=3 /dev/hda1 imagefile logfile

If that fails you can try again but retrimmed, so it tries to reread full sectors:

ddrescue --direct --retrim --max-retries=3 /dev/hda1 imagefile logfile

You can now use ddrescue (or normal dd) to copy the imagefile to a new partition on a new disk. Use the appropriate filesystem checkers (fsck, CHKDSK) to try to fix errors caused by the bad blocks. Be sure to keep the imagefile around. Just in case the filesystem is severely broken, and datacarving tools like testdisk need to to be used on the original image.

Before linux kernel 2.6.3 / 2.4.x

In 2.6.3 the 'raw device' has been marked obsolete. On later kernels ddrescue will use O_DIRECT on the input to do uncached reads.

First you copy as much data as possible, without retrying or splitting sectors:

ddrescue --no-split /dev/hda1 imagefile logfile

Now change over to raw device access. Let it retry previous errors 3 times, don't read past last block in logfile:

modprobe raw
raw /dev/raw/raw1 /dev/hda1
ddrescue --max-retries=3 --complete-only /dev/raw/raw1 imagefile logfile

If that fails you can try again (still using raw) but retrimmed, so it tries to reread full sectors:

ddrescue --retrim --max-retries=3 --complete-only /dev/raw/raw1 imagefile logfile

You can now use ddrescue (or normal dd) to copy the imagefile to a new partition on a new disk. Use the appropriate filesystem checkers (fsck, CHKDSK) to try to fix errors caused by the bad blocks. Be sure to keep the imagefile around. Just in case the filesystem is severely broken, and datacarving tools like testdisk need to to be used on the original image.

At the end you may want to unbind the raw device:

raw /dev/raw/raw1 0 0

Examples

These two examples are taken directly from the ddrescue info pages.

Example 1: Rescue an ext2 partition in /dev/hda2 to /dev/hdb2

Please Note: This will overwrite ALL data on the partition you are copying to. If you do not want to do that, rather create an image of the partition to be rescued.

ddrescue -r3 /dev/hda2 /dev/hdb2 logfile
e2fsck -v -f /dev/hdb2
mount -t ext2 -o ro /dev/hdb2 /mnt

Example 2: Rescue a CD-ROM in /dev/cdrom

ddrescue -b 2048 /dev/cdrom cdimage logfile

write cdimage to a blank CD-ROM


This example is derived from the ddrescue manual.

Example 3: Rescue an entire hard disk /dev/sda to another disk /dev/sdb

copy the error free areas first

ddrescue -n /dev/sda /dev/sdb rescue.log

attempt to recover any bad sectors

ddrescue -r 1 /dev/sda /dev/sdb rescue.log

Options

-h, --help
   display this help and exit 
-V, --version
   output version information and exit 
-b, --block-size=<bytes>
   hardware block size of input device [512] 
-B, --binary-prefixes
   show binary multipliers in numbers [default SI] 
-c, --cluster-size=<blocks>
   hardware blocks to copy at a time [128] 
-C, --complete-only
   do not read new data beyond logfile limits 
-d, --direct
   use direct disc access for input file 
-D, --synchronous
   use synchronous writes for output file 
-e, --max-errors=<n>
   maximum number of error areas allowed 
-F, --fill=<types>
   fill given type areas with infile data (?*/-+) 
-g, --generate-logfile
   generate approximate logfile from partial copy 
-i, --input-position=<pos>
   starting position in input file [0] 
-n, --no-split
   do not try to split or retry error areas 
-o, --output-position=<pos>
   starting position in output file [ipos] 
-q, --quiet
   quiet operation 
-r, --max-retries=<n>
   exit after given retries (-1=infinity) [0] 
-R, --retrim
   mark all error areas as non-trimmed 
-s, --max-size=<bytes>
   maximum size of data to be copied 
-S, --sparse
   use sparse writes for output file 
-t, --truncate
   truncate output file 
-v, --verbose
   verbose operation

Numbers may be followed by a multiplier: b = blocks, k = kB = 10^3 = 1000, Ki = KiB = 2^10 = 1024, M = 10^6, Mi = 2^20, G = 10^9, Gi = 2^30, etc...


Cygwin

As of release 1.4-rc1, it can be compiled directly in Cygwin Out of the Box. Precompiled packages are available in the Cygwin distribution. This makes it usable natively on Windows systems.

See also

Other Resources

[code-snippets for DDrescue]