Difference between pages "Tools:Visualization" and "AFF"

From Forensics Wiki
(Difference between pages)
Jump to: navigation, search
m (Java)
 
m
 
Line 1: Line 1:
Although not strictly for forensic purposes, '''visualization tools''' such as the ones discussed here can be very useful for visualizing large data sets. As forensic practitioners need to process more and more data, it is likely that some of the techniques implemented by these tools will need to be adopted.
+
The '''Advanced Forensics Format''' ('''AFF''') is an extensible open format for the storage of [[disk image]]s and related forensic [[metadata]]. It was originally developed by [[Simson Garfinkel]] and [[Basis Technology]]. The last version of AFF is implemented in the [[AFFLIBv3]] library, which can be found on [https://github.com/simsong/AFFLIBv3 github].  [[AFF4]] builds upon many of the concepts developed in AFF.  AFF4 was developed by [[Michael Cohen]], Simson Garfinkel and Bradley Schatz. That version can be downloaded from [https://code.google.com/p/aff4/ Google Code].
  
==Programming Languages and Developer Toolkits==
+
[[Sleuthkit]], [[Autopsy]] , [[OSFMount]], [[Xmount]], [[FTK Imager]] and [[FTK]] support the AFFv3 image format.
If you are building forensic tools, you probably want to start with one of these:
+
===Java===
+
Advantage: Portable and lots of good documentation out there.
+
  
Disadvantage: Programs are a bit verbose, and only offers about 1/2 the performance of C
+
==AFF Background==
* [http://csbi.sourceforge.net/index.html Graph Interface Library (GINY)] - Java
+
AFF was created to be an open and extensible file format to store disk images and associated metadata. The goal was to create a disk imaging format that would not lock users into a proprietary format that may limit how he or she may analyze it. An open standard enables investigators to quickly and efficiently use their preferred tools to solve crimes, gather intelligence, and resolve security incidents. The format was implemented in AFFLIB which was distributed with an open source license.
* [http://hypergraph.sourceforge.net/ HyperGraph] - Hyperbolic trees, in Java. Check out the home page. Try clicking on the logo...
+
* [http://ivtk.sourceforge.net/ InfoViz Toolkit] - Java, originally developed at [[INRA]].
+
* [https://jdigraph.dev.java.net/ Jdigrah] - Java Directed Graphs.
+
* [http://jgrapht.sourceforge.net/ JGraphT] - A Java visualization kit designed to be simple and extensible.
+
* [http://prefuse.sourceforge.net/ Perfuse] - A Java-based toolkit for building interactive information visualization applications
+
* [http://www.ssec.wisc.edu/~billh/visad.html#intro VisAD] - A Java component library for interactive and collaborative visualization.
+
* [http://www.softwaresecretweapons.com/jspwiki/Wiki.jsp?page=LinguineMaps Linguine Maps] - An open-source Java-based system for visualizing software call maps.
+
* [http://zvtm.sourceforge.net/index.html Zoomable Visual Transformation Machine] - Java. Originally started at Xerox Research Europe.
+
* [http://openmap.bbn.com/ OpenMap] A Java-based Geographical Information System framework, from [[BBN]].
+
  
===Python===
+
After AFFLIB was published, [[Joachim Metz]] published [[libewf]], an open source implementation of the EnCase Expert Witness format. Later, Guidance Software modified its format to allow single disk volumes larger than 4GiB. Together these two changes significantly decreased the need for AFF and AFFLIB.
: Advantage: Portable
+
: Disadvantage: Python is one of the slowest modern languages around and doesn't work well with big data sets.
+
* Python with Tk
+
* Python with matplotlib
+
* Python with wxWidgets (not installed by default)
+
* [http://networkx.lanl.gov/ NetworkX], a pure Python network layout program which uses matplotlib to do the actual graphing.
+
  
===Processing.org===
+
In 2009 Cohen, Garfinkel and Schatz published an article on AFF4, a new file format that incorporated and expanded on the underlying AFF ideas. AFF4 provides for multiple data views within a single data archives and allows links between archives. As a result, AFF4 natively supports selective imaging, logical file volumes, hash-based imaging, and a variety of case-management scenarios.
: Advantage: Programming language specifically developed for visualization; compiles to java byte code
+
: Disadvantage: Very oddball
+
*  [http://processing.org processing.org] and  
+
  
 +
==AFFv3 Extensions==
  
===JavaScript===
+
The original AFF format is a single file that contains segments with drive data and metadata. Its contents can be compressed, but it can be quite large as the data on modern hard disks often reach 100GB in size.
Browsers are now very fast and there are a large number of data plotting systems built on JavaScript. Check out:
+
AFFv3 supported three file extensions --– AFF, AFD and AFM –-- and provided a tool to easily convert between the variations.
* [http://processingjs.org processingjs.org]
+
* [http://highcharts.com/ Highcharts JS]
+
* [http://mbostock.github.com/d3/ D3 - Data Driven Documents] - A visualization library for the Web.
+
* [http://key-lines.com KeyLines] - Commercially licensed visualization library for networks/graphs.
+
  
===C/C++===
+
For ease of transfer, large AFF files can be broken into multiple AFD format files. The smaller AFD files can be readily moved around a FAT32 file system which limits files to 2GB or stored on DVDs, which have similar size restrictions. The AFM format stores the metadata in an AFF file, and the disk data in a separate raw file. This format allows analysis tools that support the raw format to access the data, but without losing the metadata.
* [http://www.vtk.org/ The Visualization Toolkit] - C++ multi-platform with interfaces available for Tcl/Tk, Java and Python.  
+
  
== Applications ==
+
===Compression and Encryption===
Most of these applications can be used to do visualization
+
AFF supports two compression algorithms: zlib, which is fast and reasonably efficient, and LZMA, which is slower but dramatically more efficient. zlib is the same compression algorithm used by EnCase. As a result, AFF files compressed with zlib are roughly the same size as the equivalent EnCase file. AFF files can be recompressed using the LZMA algorithm. These files are anywhere from 1/2 to 1/10th the size of the original AFF/EnCase file.
  
; Crystal Space 3D
+
AFF2.0 supports encryption of disk images. Unlike the password implemented by EnCase, encrypted images cannot be accessed without the necessary encryption key. FTK Imager/FTK added support for this encryption  in version 3.0 and are able to create and access AFF encrypted images.
  
; Panda#D
+
=== AFFLIBv3 Tools ===
  
===Data Plotting Applications===
+
* [[aimage]]
* http://ploticus.sourceforge.net
+
* [[ident]]
* http://www.gnuplot.info/
+
* [[afcat]]
* http://tulip.labri.fr/TulipDrupal/
+
* [[afcompare]]
 +
* [[afconvert]]
 +
* [[affix]]
 +
* [[affuse]]
 +
* [[afinfo]]
 +
* [[afstats]]
 +
* [[afxml]]
 +
* [[afsegment]]
  
===Graph and (Social) Network Visualization===
+
= See Also =
* [http://www.cytoscape.org/ Cytoscape] - Cytoscape is an open source software platform for visualizing complex networks and integrating these with any type of attribute data.
+
* [http://www.graphviz.org/ Graphviz] - Originally developed by the [http://public.research.att.com/areas/visualization/ AT&T Information Visualization Gorup], designed for drawing connected graphs of nodes and edges. Neato is a similar system but does layout based on a spring model. Can produce output as [[PostScript]], [[PNG]], [[GIF]], or as an annotated graph file with the locations of all of the objects — ideal for drawing in a GUI. Runs from the command line on [[Unix]], [[Windows]] and [[Mac]], although there is also a [http://www.pixelglow.com/graphviz/ MacOS GUI version].
+
* [http://www.codeplex.com/nodexl NodeXL] - Free/open excel add-in extends the spreadsheet with network metrics and visualizations. (Only runs on Windows)
+
* [http://gephi.org/ Gephi] -Gephi is an interactive visualization and exploration platform for all kinds of networks and complex systems, dynamic and hierarchical graphs
+
  
* http://projects.skewed.de/graph-tool/ - graph-tool is an efficient python module for manipulation and statistical analysis of graphs (a.k.a. networks).
+
* [[AFF Developers Guide]] --- A guide for programmers on how to use the AFF
* http://igraph.sourceforge.net/ - Integrates with R.
+
* [[AFF Development Task List]] --- Want to help with AFF? Here is a list of things that need to be done.
* http://socnetv.sourceforge.net/ - "Social Networks Visualizer (SocNetV) is a flexible and user-friendly tool for the analysis and visualization of Social Networks."
+
* [https://sites.google.com/site/netdrawsoftware/home NetDraw] - "a free program written by Steve Borgatti for visualizing both 1-mode and 2-mode social network data."
+
* [http://pajek.imfm.si/doku.php Pajek] - Windows program for drawing large networks.
+
* [http://sourceforge.net/projects/sonia/ Social Network Image Animator (SoNIA)] - Originally developed at Stanford. Written in Java. Makes movies.
+
* [http://www.caida.org/tools/visualization/walrus/ Walrus] - A 3-d graph network exploration tool. Employs 3D hyperbolic displays and layout based on a user-supplied spanning tree.
+
* [http://afterglow.sf.net AfterGlow] - A tool to simplify the generation of network graphs in GraphViz, Gephi, etc.  Designed for security visualizations.
+
* http://tulip.labri.fr/TulipDrupal/ - Tulip is an information visualization framework dedicated to the analysis and visualization of relational data.  
+
  
See also: http://en.wikipedia.org/wiki/Social_network_analysis_software
+
== External Links ==
 +
* [http://www.basistech.com/digital-forensics/aff.html Basis Technology's AFF website]
 +
* [http://www.osforensics.com/tools/mount-disk-images.html OSFMount - 3rd party tool for mounting AFF disk images with a drive letter]
  
Reas govisual diagram editor
+
[[Category:Forensics File Formats]]
reas.com
+
[[Category:Open Source Tools]]
gdf.net
+
 
+
 
+
=== Commercial Graphic Applications and Tools===
+
 
+
* [http://www.aisee.com/ aiSee Graph Layout Software] - Supports 15 layout algorithms, recursive graph nesting, and easy printing. Runs on [[Windows]], [[Linux]], [[Solaris]], [[NetBSD]], and [[MacOS]]. 30-day trial and free registered versions available. Academic pricing available.
+
*  [http://www.geomantics.com/ Geomantics] - Geographical, Visualization and Graphics software. Runs on [[Windows]].
+
* [http://www.kylebank.com/ Graphis 2D and 3D graphing software] - Runs on [[Windows]]. Free 30-day evaluation copy available.
+
* [http://www.openviz.com/ OpenViz] and  [http://www.powerviz.com/ PowerViz] - Both from Advanced Visual Systems, super high-end visualization toolkits. $$$$
+
* [http://www.tomsawyer.com/ Tom Sawyer Software] Analysis, Visualizaiton, and Layout programs. - Heavy support for drawing graphs. Beautiful gallery. ActiveX, Java, C++ and .NET editions.
+
* [http://www.netminer.com/ NetMiner] - A comprehensive tool for Social Network Analysis. Runs on Windows, with a Linux version under development. $35 for "Express" student version, $250 for "Professional" student version, $950 for "Normal" "Professional" version.
+
* [http://www.analytictech.com/ucinet.htm UCINET] - A comprehensive package for the analysis of social network data as well as other 1-mode and 2-mode data.
+
* [http://www.clarifiednetworks.com/logster Logster] - an ultra-easy software tool to visualize Apache-style logs on a world map.
+
* [http://www.clarifiednetworks.com/Clarified%20Analyzer Clarified Analyzer] - Visualizes Network Traffic and allows to drill down from visualizations to the packet level.
+
 
+
===Unclassified===
+
* [http://gravisto.fim.uni-passau.de/ Gravisto: Graph Visualization Toolkit] - An editor and toolkit for developing graph visualization algorithms.
+
* [http://www.gnu.frb.br:8080/rox Rox Graph Theory Framework] - An open-source plug-in framework for graph theory visualization.
+
* [http://touchgraph.sourceforge.net/ TouchGraph] - Library for building graph-based interfaces.
+
 
+
==See Also==
+
* [[Network Data Visualizations]]
+
* [http://www-static.cc.gatech.edu/gvu/ii/resources/infovis.html GVU's Information Visualization Resources link farm]
+
* [http://directory.google.com/Top/Science/Math/Combinatorics/Software/Graph_Drawing/ Google Directory of Graph Drawing Software]
+
* [http://directory.fsf.org/science/visual/ GNU Free Software directory of scientific visualization software]
+
* [http://www.manageability.org/blog/stuff/open-source-graph-network-visualization-in-java/view Open Source Graph Network Visualization in Java]
+
* [http://www.insna.org/INSNA/soft_inf.html INSNA's web page of Computer Programs for Social Network Analysis]
+
* http://www.dmoz.org/Science/Math/Combinatorics/Software/Graph_Drawing/
+
* http://www.ontopia.net/omnigator/models/index.jsp
+
* http://visant.bu.edu/
+
* http://sourceforge.net/projects/touchgraph/
+
* [http://www.informatik.uni-rostock.de/~ct/ Christian Tominski]
+
 
+
CAIDA has 15+ years of work visualizing Internet topologies. You may find their tools to be useful:
+
* http://www.caida.org/tools/visualization/
+
* http://www.caida.org/publications/visualizations/
+
* http://www.caida.org/tools/visualization/walrus/gallery1/
+
* http://www.caida.org/research/topology/as_core_network/
+

Revision as of 07:04, 8 April 2013

The Advanced Forensics Format (AFF) is an extensible open format for the storage of disk images and related forensic metadata. It was originally developed by Simson Garfinkel and Basis Technology. The last version of AFF is implemented in the AFFLIBv3 library, which can be found on github. AFF4 builds upon many of the concepts developed in AFF. AFF4 was developed by Michael Cohen, Simson Garfinkel and Bradley Schatz. That version can be downloaded from Google Code.

Sleuthkit, Autopsy , OSFMount, Xmount, FTK Imager and FTK support the AFFv3 image format.

Contents

AFF Background

AFF was created to be an open and extensible file format to store disk images and associated metadata. The goal was to create a disk imaging format that would not lock users into a proprietary format that may limit how he or she may analyze it. An open standard enables investigators to quickly and efficiently use their preferred tools to solve crimes, gather intelligence, and resolve security incidents. The format was implemented in AFFLIB which was distributed with an open source license.

After AFFLIB was published, Joachim Metz published libewf, an open source implementation of the EnCase Expert Witness format. Later, Guidance Software modified its format to allow single disk volumes larger than 4GiB. Together these two changes significantly decreased the need for AFF and AFFLIB.

In 2009 Cohen, Garfinkel and Schatz published an article on AFF4, a new file format that incorporated and expanded on the underlying AFF ideas. AFF4 provides for multiple data views within a single data archives and allows links between archives. As a result, AFF4 natively supports selective imaging, logical file volumes, hash-based imaging, and a variety of case-management scenarios.

AFFv3 Extensions

The original AFF format is a single file that contains segments with drive data and metadata. Its contents can be compressed, but it can be quite large as the data on modern hard disks often reach 100GB in size. AFFv3 supported three file extensions --– AFF, AFD and AFM –-- and provided a tool to easily convert between the variations.

For ease of transfer, large AFF files can be broken into multiple AFD format files. The smaller AFD files can be readily moved around a FAT32 file system which limits files to 2GB or stored on DVDs, which have similar size restrictions. The AFM format stores the metadata in an AFF file, and the disk data in a separate raw file. This format allows analysis tools that support the raw format to access the data, but without losing the metadata.

Compression and Encryption

AFF supports two compression algorithms: zlib, which is fast and reasonably efficient, and LZMA, which is slower but dramatically more efficient. zlib is the same compression algorithm used by EnCase. As a result, AFF files compressed with zlib are roughly the same size as the equivalent EnCase file. AFF files can be recompressed using the LZMA algorithm. These files are anywhere from 1/2 to 1/10th the size of the original AFF/EnCase file.

AFF2.0 supports encryption of disk images. Unlike the password implemented by EnCase, encrypted images cannot be accessed without the necessary encryption key. FTK Imager/FTK added support for this encryption in version 3.0 and are able to create and access AFF encrypted images.

AFFLIBv3 Tools

See Also

External Links