Difference between pages "Tools:Visualization" and "Ddrescue"

From Forensics Wiki
(Difference between pages)
Jump to: navigation, search
m (Other Resources)
 
(linked to my blog containing some useful scripts/oneliners -- not sure if linking is allowed this way... else lmk!)
 
Line 1: Line 1:
Although not strictly for forensic purposes, '''visualization tools''' such as the ones discussed here can be very useful for visualizing large data sets. As forensic practitioners need to process more and more data, it is likely that some of the techniques implemented by these tools will need to be adopted.
+
{{Infobox_Software |
 +
  name = ddrescure |
 +
  maintainer = [[Antonio Diaz Diaz]]|
 +
  os = {{Linux}}|
 +
  genre = {{Disk imaging}} |
 +
  license = {{GPL}} |
 +
  website = [http://www.gnu.org/software/ddrescue/ddrescue.html http://www.gnu.org/software/ddrescue/ddrescue.html] |
 +
}}
  
== Open Source ==
+
'''ddrescue''' is a raw disk imaging tool that "copies data from one file or block device to another, trying hard to rescue data in case of read errors."  The application is developed as part of the GNU project and has written with UNIX/Linux in mind.
=== Visualization Toolkits and Libraries ===
+
* [http://csbi.sourceforge.net/index.html Graph Interface Library (GINY)] - Java
+
* [http://www.gravisto.org/ Gravisto: Graph Visualization Toolkit] - An editor and toolkit for developing graph visualization algorithms.
+
*  [http://hypergraph.sourceforge.net/ HyperGraph] - Hyperbolic trees, in Java. Check out the home page. Try clicking on the logo...
+
* [http://ivtk.sourceforge.net/ InfoViz Toolkit] - Java, originally developed at [[INRA]].
+
* [https://jdigraph.dev.java.net/ Jdigrah] - Java Directed Graphs.
+
* [http://jgrapht.sourceforge.net/ JGraphT] - A Java visualization kit designed to be simple and extensible.
+
* [http://www.softwaresecretweapons.com/jspwiki/Wiki.jsp?page=LinguineMaps Linguine Maps] - An open-source Java-based system for visualizing software call maps.
+
* [http://prefuse.sourceforge.net/ Perfuse] - A Java-based toolkit for building interactive information visualization applications
+
* [http://www.gnu.frb.br:8080/rox Rox Graph Theory Framework] - An open-source plug-in framework for graph theory visualization.
+
* [http://touchgraph.sourceforge.net/ TouchGraph] - Library for building graph-based interfaces.
+
* [http://www.ssec.wisc.edu/~billh/visad.html#intro VisAD] - A Java component library for interactive and collaborative visualization.
+
* [http://public.kitware.com/VTK/ The Visualization Toolkit] - C++ multi-platform with interfaces available for Tcl/Tk, Java and Python. Professional support provided by [http://www.kitware.com/ Kitware].
+
* [http://zvtm.sourceforge.net/index.html Zoomable Visual Transformation Machine] - Java. Originally started at Xerox Research Europe.
+
* [http://processing.org/ Processing.org] - A new language for doing graphics and visualization.
+
  
====Tree Maps====
+
'''ddrescue''' and '''[[dd_rescue]]''' are completely different programs which share no development between them. The two projects are not related in any way except that they both attempt to enhance the standard [[dd]] tool and coincidentally chose similar names for their new programs.
[http://www.cs.umd.edu/hcil/treemap-history/index.shtml Treemaps] are a visualization technique developed at the University of Maryland for visualizing large amounts of multi-dimensional data. There is a GPLed implementation of treemaps written in C. You can find a copy of in it:
+
* [http://www.derlien.com/ Disk Inventory X]  
+
* [http://kdirstat.sourceforge.net/ KDirStat]
+
  
===Graph Drawing Applications===
+
From the [[ddrescue]] info pages:
* [http://www.graphviz.org/ Graphviz] - Originally developed by the [http://public.research.att.com/areas/visualization/ AT&T Information Visualization Gorup], designed for drawing connected graphs of nodes and edges. Neato is a similar system but does layout based on a spring model. Can produce output as [[PostScript]], [[PNG]], [[GIF]], or as an annotated graph file with the locations of all of the objects — ideal for drawing in a GUI. Runs from the command line on [[Unix]], [[Windows]] and [[Mac]], although there is also a [http://www.pixelglow.com/graphviz/ MacOS GUI version].
+
<blockquote>
* [http://graphexploration.cond.org/ Guess: The Graph Exploration System] - Originally developed at HP, this is a large Jython/Java-based system that you can use for building your own applications. Distributed under GPL.
+
GNU ddrescue is a data recovery tool. It copies data from one file or block device (hard disc, cdrom, etc) to another, trying hard to rescue data in case of read errors.<br><br>
* [http://sourceforge.net/projects/ivc/ InfoVis Cyberinfrastructure] - Another graph drawing system written in Java.
+
* [http://jung.sourceforge.net/ Java Universal Network/Graph Framework (JUNG)] - Graphing, [[data mining]], [[social network]] analysis, and other stuff.
+
* [http://www.andrew.cmu.edu/user/krack/krackplot.shtml Krackplot] - "KrackPlot is a program for network visualization designed for social network analysts."
+
* [http://bioinformatics.icmb.utexas.edu/lgl/ Large Graph Layout (LGL)] - A bioinformatics system from University of Texas. They really mean Large.
+
* [http://www.sfu.ca/~richards/Multinet/Pages/multinet.htm MultiNet] - A data analysis package for drawing conventional data and graph data.
+
* [http://www.analytictech.com/netdraw.htm NetDraw] - "a free program written by Steve Borgatti for visualizing both 1-mode and 2-mode social network data."
+
* [http://web.mit.edu/bshi/Public/nv2d/ NetVis 2D] - Another graph visualization and layout tool written in Java.
+
* [http://www.opendx.org/ OpenDX] - Based on [[IBM]]'s Visualization Data Explorer, runs on [[Unix]]/X11/Motif.
+
* [http://vlado.fmf.uni-lj.si/pub/networks/pajek/ Pajek] - Windows program for drawing large networks.
+
* [http://sourceforge.net/projects/sonia/ Social Network Image Animator (SoNIA)] - Originally developed at Stanford. Written in Java. Makes movies.
+
* [http://www.informatik.uni-bremen.de/uDrawGraph/en/uDrawGraph/uDrawGraph.html uDrawGraph]
+
* [http://wilma.sourceforge.net/ WilmaScope] - Real-time animations of dynamic graph structures. Written in Java. Sophisticated force model with strings and attraction.
+
* [http://www.caida.org/tools/visualization/walrus/ Walrus] - A 3-d graph network exploration tool. Employs 3D hyperbolic displays and layout based on a user-supplied spanning tree.
+
  
== Geographical Drawing Programs ==
+
Ddrescue does not truncate the output file if not asked to. So, every time you run it on the same output file, it tries to fill in the gaps.<br><br>
  
* [http://openmap.bbn.com/ OpenMap] -- From [[BBN]].
+
The basic operation of ddrescue is fully automatic. That is, you don't have to wait for an error, stop the program, read the log, run it in reverse mode, etc.<br><br>
  
== Commercial Tools ==
+
If you use the logfile feature of ddrescue, the data is rescued very efficiently (only the needed blocks are read). Also you can interrupt the rescue at any time and resume it later at the same point.<br><br>
  
* [http://www.aisee.com/ aiSee Graph Layout Software] - Supports 15 layout algorithms, recursive graph nesting, and easy printing. Runs on [[Windows]], [[Linux]], [[Solaris]], [[NetBSD]], and [[MacOS]]. 30-day trial and free registered versions available. Academic pricing available.
+
Automatic merging of backups: If you have two or more damaged copies of a file, cdrom, etc, and run ddrescue on all of them, one at a time, with the same output file, you will probably obtain a complete and error-free file. This is so because the probability of having damaged areas at the same places on different input files is very low. Using
*  [http://www.geomantics.com/ Geomantics] - Geographical, Visualization and Graphics software. Runs on [[Windows]].
+
the logfile, only the needed blocks are read from the second and successive copies.
* [http://www.kylebank.com/ Graphis 2D and 3D graphing software] - Runs on [[Windows]]. Free 30-day evaluation copy available.
+
</blockquote>
* [http://www.openviz.com/ OpenViz] and  [http://www.powerviz.com/ PowerViz] - Both from Advanced Visual Systems, super high-end visualization toolkits. $$$$
+
* [http://www.tomsawyer.com/ Tom Sawyer Software] Analysis, Visualizaiton, and Layout programs. - Heavy support for drawing graphs. Beautiful gallery. ActiveX, Java, C++ and .NET editions.
+
* [http://www.netminer.com/ NetMiner] - "One of the most comprehensive and usable software tools for Social Network Analysis in the world." Runs on Winodws, with a Linux version under development. $35 for "Express" student version, $250 for "Professional" student version, $950 for "Normal" "Professional" version.
+
* [http://www.analytictech.com/ucinet.htm UCINET] - A comprehensive package for the analysis of social network data as well as other 1-mode and 2-mode data.
+
  
= Other Resources =
+
== Installation ==
==Journals and Conferences==
+
 
* [http://www.palgrave-journals.com/ivs/index.html Information Visualization Journal]
+
=== Bootable CD ===
* [http://rw4.cs.uni-sb.de/~diehl/softvis/seminar/index.php?goto=seminar ACM Symposium on Software Visualization]
+
ddrescue is available on bootable rescue cds such as SystemRescueCd http://www.sysresccd.org/Main_Page.
==Link Farms==
+
=== Debian and Ubuntu ===
* [http://www-static.cc.gatech.edu/gvu/ii/resources/infovis.html GVU's Information Visualization Resources link farm]
+
The package 'ddrescue' in Debian and Ubuntu is actually [[dd_rescue]], another dd-like program which does not maintain a recovery log.  The correct package is gddrescue.
* [http://directory.google.com/Top/Science/Math/Combinatorics/Software/Graph_Drawing/ Google Directory of Graph Drawing Software]
+
 
* [http://directory.fsf.org/science/visual/ GNU Free Software directory of scientific visualization software]
+
Debian
* [http://www.manageability.org/blog/stuff/open-source-graph-network-visualization-in-java/view Open Source Graph Network Visualization in Java]
+
<blockquote>
* [http://www.insna.org/INSNA/soft_inf.html INSNA's web page of Computer Programs for Social Network Analysis]
+
aptitude install gddrescue
==Research Groups==
+
</blockquote>
* [http://www.msi.umn.edu/user_support/scivis/scivis-list.html Scientific Visualization at the Supercomputing Institute]
+
Ubuntu
* [http://www.cs.brown.edu/people/rt/gd.html Roberto Tamassia's resources on Graph Drawing]
+
<blockquote>
* [http://bailando.sims.berkeley.edu/infovis.html Bailando Visualization at UC Berkeley]
+
sudo apt-get install gddrescue
 +
</blockquote>
 +
=== Gentoo ===
 +
<blockquote>
 +
emerge ddrescue
 +
</blockquote>
 +
== Partition recovery ==
 +
 
 +
=== Kernel 2.6.3+ & ddrescue 1.4+ ===
 +
'ddrescue --direct' will open the input with the O_DIRECT option for uncached reads. 'raw devices' are not needed on newer kernels. For older kernels see below.
 +
 
 +
First you copy as much data as possible, without retrying or splitting sectors:
 +
<blockquote>
 +
ddrescue --no-split /dev/hda1 imagefile logfile
 +
</blockquote>
 +
 
 +
Now let it retry previous errors 3 times, using uncached reads:
 +
<blockquote>
 +
ddrescue --direct --max-retries=3 /dev/hda1 imagefile logfile
 +
</blockquote>
 +
 
 +
If that fails you can try again but retrimmed, so it tries to reread full sectors:
 +
<blockquote>
 +
ddrescue --direct --retrim  --max-retries=3 /dev/hda1 imagefile logfile
 +
</blockquote>
 +
 
 +
You can now use ddrescue (or normal dd) to copy the imagefile to a new partition on a new disk. Use the appropriate filesystem checkers (fsck, CHKDSK) to try to fix errors caused by the bad blocks. Be sure to keep the imagefile around. Just in case the filesystem is severely broken, and datacarving tools like testdisk need to to be used on the original image.
 +
 
 +
=== Before linux kernel 2.6.3 / 2.4.x ===
 +
In 2.6.3 the 'raw device' has been marked obsolete. On later kernels ddrescue will use O_DIRECT on the input to do uncached reads.
 +
 
 +
First you copy as much data as possible, without retrying or splitting sectors:
 +
<blockquote>
 +
ddrescue --no-split /dev/hda1 imagefile logfile
 +
</blockquote>
 +
 
 +
Now change over to raw device access. Let it retry previous errors 3 times, don't read past last block in logfile:
 +
<blockquote>
 +
modprobe raw<br>
 +
raw /dev/raw/raw1 /dev/hda1<br>
 +
ddrescue --max-retries=3 --complete-only /dev/raw/raw1 imagefile logfile
 +
</blockquote>
 +
 
 +
If that fails you can try again (still using raw) but retrimmed, so it tries to reread full sectors:
 +
<blockquote>
 +
ddrescue --retrim --max-retries=3 --complete-only /dev/raw/raw1 imagefile logfile
 +
</blockquote>
 +
 
 +
You can now use ddrescue (or normal dd) to copy the imagefile to a new partition on a new disk. Use the appropriate filesystem checkers (fsck, CHKDSK) to try to fix errors caused by the bad blocks. Be sure to keep the imagefile around. Just in case the filesystem is severely broken, and datacarving tools like testdisk need to to be used on the original image.
 +
 
 +
At the end you may want to unbind the raw device:
 +
<blockquote>
 +
raw /dev/raw/raw1 0 0
 +
</blockquote>
 +
 
 +
== Examples ==
 +
 
 +
These two examples are taken directly from the [[ddrescue]] info pages.
 +
 
 +
Example 1: Rescue an ext2 partition in /dev/hda2 to /dev/hdb2
 +
 
 +
'''Please Note:''' This will overwrite ALL data on the partition you are copying to. If you do not want to do that, rather create an image of the partition to be rescued.
 +
<blockquote>
 +
ddrescue -r3 /dev/hda2 /dev/hdb2 logfile<br>
 +
e2fsck -v -f /dev/hdb2<br>
 +
mount -t ext2 -o ro /dev/hdb2 /mnt<br>
 +
</blockquote>
 +
 
 +
Example 2: Rescue a CD-ROM in /dev/cdrom
 +
<blockquote>
 +
ddrescue -b 2048 /dev/cdrom cdimage logfile
 +
</blockquote>
 +
write cdimage to a blank CD-ROM
 +
 
 +
 
 +
This example is derived from the ddrescue manual.
 +
 
 +
Example 3: Rescue an entire hard disk /dev/sda to another disk /dev/sdb
 +
 
 +
copy the error free areas first
 +
ddrescue -n /dev/sda /dev/sdb rescue.log
 +
attempt to recover any bad sectors
 +
ddrescue -r 1 /dev/sda /dev/sdb rescue.log
 +
 
 +
== Options ==
 +
 
 +
-h, --help
 +
    display this help and exit
 +
-V, --version
 +
    output version information and exit
 +
-b, --block-size=<bytes>
 +
    hardware block size of input device [512]
 +
-B, --binary-prefixes
 +
    show binary multipliers in numbers [default SI]
 +
-c, --cluster-size=<blocks>
 +
    hardware blocks to copy at a time [128]
 +
-C, --complete-only
 +
    do not read new data beyond logfile limits
 +
-d, --direct
 +
    use direct disc access for input file
 +
-D, --synchronous
 +
    use synchronous writes for output file
 +
-e, --max-errors=<n>
 +
    maximum number of error areas allowed
 +
-F, --fill=<types>
 +
    fill given type areas with infile data (?*/-+)
 +
-g, --generate-logfile
 +
    generate approximate logfile from partial copy
 +
-i, --input-position=<pos>
 +
    starting position in input file [0]  
 +
-n, --no-split
 +
    do not try to split or retry error areas
 +
-o, --output-position=<pos>
 +
    starting position in output file [ipos]
 +
-q, --quiet
 +
    quiet operation
 +
-r, --max-retries=<n>
 +
    exit after given retries (-1=infinity) [0]
 +
-R, --retrim
 +
    mark all error areas as non-trimmed
 +
-s, --max-size=<bytes>
 +
    maximum size of data to be copied
 +
-S, --sparse
 +
    use sparse writes for output file
 +
-t, --truncate
 +
    truncate output file
 +
-v, --verbose
 +
    verbose operation
 +
 
 +
Numbers may be followed by a multiplier: b = blocks, k = kB = 10^3 = 1000, Ki = KiB = 2^10 = 1024, M = 10^6, Mi = 2^20, G = 10^9, Gi = 2^30, etc...
 +
 
 +
 
 +
== Cygwin ==
 +
 
 +
As of release 1.4-rc1, it can be compiled directly in [[Cygwin]] [http://en.wikipedia.org/wiki/Out_of_the_box Out of the Box]. Precompiled packages are available in the [http://cygwin.com/packages/ Cygwin distribution]. This makes it usable natively on [[Windows]] systems.
 +
 
 +
== See also ==
 +
 
 +
* [[aimage]]
 +
* [[Blackbag]]
 +
* [[dcfldd]]
 +
* [[dd]]
 +
* [[dd_rescue]]
 +
* [[sdd]]
 +
 
 +
== Other Resources ==
 +
[[http://pfuender.net/?p=80|Useful code-snippets for DDrescue]]

Revision as of 07:09, 25 June 2010

ddrescure
Maintainer: Antonio Diaz Diaz
OS: Linux
Genre: Disk imaging
License: GPL
Website: http://www.gnu.org/software/ddrescue/ddrescue.html

ddrescue is a raw disk imaging tool that "copies data from one file or block device to another, trying hard to rescue data in case of read errors." The application is developed as part of the GNU project and has written with UNIX/Linux in mind.

ddrescue and dd_rescue are completely different programs which share no development between them. The two projects are not related in any way except that they both attempt to enhance the standard dd tool and coincidentally chose similar names for their new programs.

From the ddrescue info pages:

GNU ddrescue is a data recovery tool. It copies data from one file or block device (hard disc, cdrom, etc) to another, trying hard to rescue data in case of read errors.

Ddrescue does not truncate the output file if not asked to. So, every time you run it on the same output file, it tries to fill in the gaps.

The basic operation of ddrescue is fully automatic. That is, you don't have to wait for an error, stop the program, read the log, run it in reverse mode, etc.

If you use the logfile feature of ddrescue, the data is rescued very efficiently (only the needed blocks are read). Also you can interrupt the rescue at any time and resume it later at the same point.

Automatic merging of backups: If you have two or more damaged copies of a file, cdrom, etc, and run ddrescue on all of them, one at a time, with the same output file, you will probably obtain a complete and error-free file. This is so because the probability of having damaged areas at the same places on different input files is very low. Using the logfile, only the needed blocks are read from the second and successive copies.

Contents

Installation

Bootable CD

ddrescue is available on bootable rescue cds such as SystemRescueCd http://www.sysresccd.org/Main_Page.

Debian and Ubuntu

The package 'ddrescue' in Debian and Ubuntu is actually dd_rescue, another dd-like program which does not maintain a recovery log. The correct package is gddrescue.

Debian

aptitude install gddrescue

Ubuntu

sudo apt-get install gddrescue

Gentoo

emerge ddrescue

Partition recovery

Kernel 2.6.3+ & ddrescue 1.4+

'ddrescue --direct' will open the input with the O_DIRECT option for uncached reads. 'raw devices' are not needed on newer kernels. For older kernels see below.

First you copy as much data as possible, without retrying or splitting sectors:

ddrescue --no-split /dev/hda1 imagefile logfile

Now let it retry previous errors 3 times, using uncached reads:

ddrescue --direct --max-retries=3 /dev/hda1 imagefile logfile

If that fails you can try again but retrimmed, so it tries to reread full sectors:

ddrescue --direct --retrim --max-retries=3 /dev/hda1 imagefile logfile

You can now use ddrescue (or normal dd) to copy the imagefile to a new partition on a new disk. Use the appropriate filesystem checkers (fsck, CHKDSK) to try to fix errors caused by the bad blocks. Be sure to keep the imagefile around. Just in case the filesystem is severely broken, and datacarving tools like testdisk need to to be used on the original image.

Before linux kernel 2.6.3 / 2.4.x

In 2.6.3 the 'raw device' has been marked obsolete. On later kernels ddrescue will use O_DIRECT on the input to do uncached reads.

First you copy as much data as possible, without retrying or splitting sectors:

ddrescue --no-split /dev/hda1 imagefile logfile

Now change over to raw device access. Let it retry previous errors 3 times, don't read past last block in logfile:

modprobe raw
raw /dev/raw/raw1 /dev/hda1
ddrescue --max-retries=3 --complete-only /dev/raw/raw1 imagefile logfile

If that fails you can try again (still using raw) but retrimmed, so it tries to reread full sectors:

ddrescue --retrim --max-retries=3 --complete-only /dev/raw/raw1 imagefile logfile

You can now use ddrescue (or normal dd) to copy the imagefile to a new partition on a new disk. Use the appropriate filesystem checkers (fsck, CHKDSK) to try to fix errors caused by the bad blocks. Be sure to keep the imagefile around. Just in case the filesystem is severely broken, and datacarving tools like testdisk need to to be used on the original image.

At the end you may want to unbind the raw device:

raw /dev/raw/raw1 0 0

Examples

These two examples are taken directly from the ddrescue info pages.

Example 1: Rescue an ext2 partition in /dev/hda2 to /dev/hdb2

Please Note: This will overwrite ALL data on the partition you are copying to. If you do not want to do that, rather create an image of the partition to be rescued.

ddrescue -r3 /dev/hda2 /dev/hdb2 logfile
e2fsck -v -f /dev/hdb2
mount -t ext2 -o ro /dev/hdb2 /mnt

Example 2: Rescue a CD-ROM in /dev/cdrom

ddrescue -b 2048 /dev/cdrom cdimage logfile

write cdimage to a blank CD-ROM


This example is derived from the ddrescue manual.

Example 3: Rescue an entire hard disk /dev/sda to another disk /dev/sdb

copy the error free areas first

ddrescue -n /dev/sda /dev/sdb rescue.log

attempt to recover any bad sectors

ddrescue -r 1 /dev/sda /dev/sdb rescue.log

Options

-h, --help
   display this help and exit 
-V, --version
   output version information and exit 
-b, --block-size=<bytes>
   hardware block size of input device [512] 
-B, --binary-prefixes
   show binary multipliers in numbers [default SI] 
-c, --cluster-size=<blocks>
   hardware blocks to copy at a time [128] 
-C, --complete-only
   do not read new data beyond logfile limits 
-d, --direct
   use direct disc access for input file 
-D, --synchronous
   use synchronous writes for output file 
-e, --max-errors=<n>
   maximum number of error areas allowed 
-F, --fill=<types>
   fill given type areas with infile data (?*/-+) 
-g, --generate-logfile
   generate approximate logfile from partial copy 
-i, --input-position=<pos>
   starting position in input file [0] 
-n, --no-split
   do not try to split or retry error areas 
-o, --output-position=<pos>
   starting position in output file [ipos] 
-q, --quiet
   quiet operation 
-r, --max-retries=<n>
   exit after given retries (-1=infinity) [0] 
-R, --retrim
   mark all error areas as non-trimmed 
-s, --max-size=<bytes>
   maximum size of data to be copied 
-S, --sparse
   use sparse writes for output file 
-t, --truncate
   truncate output file 
-v, --verbose
   verbose operation

Numbers may be followed by a multiplier: b = blocks, k = kB = 10^3 = 1000, Ki = KiB = 2^10 = 1024, M = 10^6, Mi = 2^20, G = 10^9, Gi = 2^30, etc...


Cygwin

As of release 1.4-rc1, it can be compiled directly in Cygwin Out of the Box. Precompiled packages are available in the Cygwin distribution. This makes it usable natively on Windows systems.

See also

Other Resources

[code-snippets for DDrescue]