Difference between pages "Tools:Visualization" and "New Technology File System (NTFS)"

From ForensicsWiki
(Difference between pages)
Jump to: navigation, search
m (Data Plotting)
 
m (spelling correction for "Windows NTFS Metadata Extactor Utility")
 
Line 1: Line 1:
Although not strictly for forensic purposes, '''visualization tools''' such as the ones discussed here can be very useful for visualizing large data sets. As forensic practitioners need to process more and more data, it is likely that some of the techniques implemented by these tools will need to be adopted.
+
The '''New Technology File System''' ('''NTFS''') is a [[file system]] developed and introduced by [[Microsoft]] in 1995 with [[Windows]] NT. As a replacement for the [[FAT]] file system, it quickly became the standard for [[Windows 2000]], [[Windows XP]] and [[Windows Server 2003]].
  
==Programming Languages and Developer Toolkits==
+
The features of NTFS include:
If you are building forensic tools, you probably want to start with one of these:
+
; Java and Swing
+
: Advantage: Portable and lots of good documentation out there.
+
: Disadvantage: Programs are a bit verbose, and only offers about 1/2 the performance of C
+
  
; Python with tkinter
+
* [[Hard-links]]
: Advantage: Portable
+
* Improved performance, reliability and disk space utilization
: Disadvantage: Python is one of the slowest modern languages around.
+
* Security [[access control lists]]
 +
* File system journaling
  
; Python with wxWidgets
+
== Time Stamps ==
: Advantage: Portable and a better development environment than tkiner
+
: Disadvantage: wxWidgets is not installed by default, so you'll need to get it installed. Not as well documented as Tkinter
+
  
; [http://www.pyside.org Python with Qt (PySide)]
+
NTFS keeps track of lots of time stamps. Each file has a time stamp for 'Create', 'Modify', 'Access', and 'Entry Modified'. The latter refers to the time when the MFT entry itself was modified. These four values are commonly abbreviated as the 'MACE' values. Note that other attributes in each MFT record may also contain timestamps that are of forensic value.
: Advantage: Portable and a better development environment than tkiner or wxWidgets
+
: Disadvantage: Though not difficult, PySide has to be installed separately
+
  
; [http://processing.org processing.org]
+
Additional information on how NTFS timestamps work when files are moved or copied is available here: [http://support.microsoft.com/kb/299648 Microsoft KB 299648]
: Advantage: Programming language specifically developed for visualization; compiles to java byte code
+
: Disadvantage: Very oddball
+
  
; JavaFX - Java's version of Flash
+
=== Changes in Windows Vista  ===
  
; Flash
+
In Windows Vista, NTFS no longer tracks the Last Access time of a file by default. This feature can be enabled by the user if desired via setting the registry key 'HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem\NtfsDisableLastAccessUpdate' to '0'.
  
Or you could try one of these higher-level toolkits:
+
== Alternate Data Streams ==
 +
The '''NTFS''' file system includes a feature referred to as Alternate Data Streams (ADSs).  This feature has also been referred to as "multiple data streams", "alternative data streams", etc.  ADSs were included in '''NTFS''' in order to support the resource forks employed by the Hierarchal File System ([[HFS]]) employed by Macintosh systems. 
  
; Crystal Space 3D
+
As of [[Windows XP]] SP2, files downloaded via Internet Explorer, Outlook, and Windows Messenger were automatically given specific "zoneid" ADSs.  The [[Windows]] Explorer shell would then display a warning when the user attempted to execute these files (by double-clicking them).
  
; Panda#D
+
Sysadmins should be aware that prior to Vista, there are no tools native to the [[Windows]] platform that would allow you to view the existence of arbitrary ADSs.  While ADSs can be created and their contents executed or viewed, it wasn't until the "/r" switch was introduced with the "dir" command on Vista that arbitrary ADSs would be visible.  Prior to this, tools such as [http://www.heysoft.de/Frames/f_sw_la_en.htm LADS] could be used to view the existence of these files.
  
== Applications ==
+
Microsoft FSRM (File System Resource Manager) also uses ADS as part of 'file classification'.
Most of these are scriptable.
+
===Open Source===
+
====Data Plotting====
+
* http://ploticus.sourceforge.net
+
* http://www.gnuplot.info/
+
* [http://www.opendx.org/ OpenDX] - Based on [[IBM]]'s Visualization Data Explorer for 3D, runs on [[Unix]]/X11/Motif. (2007)
+
* http://tulip.labri.fr/TulipDrupal/
+
* http://highcharts.com/
+
* matplotlib
+
  
====Graph and (Social) Network Visualization====
+
Examiners should be aware that most forensic analysis applications, including [[EnCase]] and ProDiscover, will display ADSs found in acquired images in red.
* [http://code.google.com/p/boomslang/ Boomslang] --- A thin layer of python over matplotlib that makes it easier to do common plots.
+
* [http://www.cytoscape.org/ Cytoscape] - Cytoscape is an open source software platform for visualizing complex networks and integrating these with any type of attribute data.
+
* [http://www.graphviz.org/ Graphviz] - Originally developed by the [http://public.research.att.com/areas/visualization/ AT&T Information Visualization Gorup], designed for drawing connected graphs of nodes and edges. Neato is a similar system but does layout based on a spring model. Can produce output as [[PostScript]], [[PNG]], [[GIF]], or as an annotated graph file with the locations of all of the objects — ideal for drawing in a GUI. Runs from the command line on [[Unix]], [[Windows]] and [[Mac]], although there is also a [http://www.pixelglow.com/graphviz/ MacOS GUI version].
+
* [http://www.codeplex.com/nodexl NodeXL] - Free/open excel add-in extends the spreadsheet with network metrics and visualizations. (Only runs on Windows)
+
* [http://gephi.org/ Gephi] -Gephi is an interactive visualization and exploration platform for all kinds of networks and complex systems, dynamic and hierarchical graphs
+
* http://graphstream-project.org/
+
* http://projects.skewed.de/graph-tool/ - graph-tool is an efficient python module for manipulation and statistical analysis of graphs (a.k.a. networks).
+
* http://igraph.sourceforge.net/
+
* http://nwb.cns.iu.edu/ - NetworkWorkbench, a workbench for network scientists.
+
* http://socnetv.sourceforge.net/ - "Social Networks Visualizer (SocNetV) is a flexible and user-friendly tool for the analysis and visualization of Social Networks."
+
* [http://jung.sourceforge.net/ Java Universal Network/Graph Framework (JUNG)] - Graphing, [[data mining]], [[social network]] analysis, and other stuff.
+
* [http://bioinformatics.icmb.utexas.edu/lgl/ Large Graph Layout (LGL)] - A bioinformatics system from University of Texas. They really mean Large.
+
* [http://www.analytictech.com/netdraw/netdraw.htm NetDraw] - "a free program written by Steve Borgatti for visualizing both 1-mode and 2-mode social network data."
+
* [http://pajek.imfm.si/doku.php Pajek] - Windows program for drawing large networks.
+
* [http://sourceforge.net/projects/sonia/ Social Network Image Animator (SoNIA)] - Originally developed at Stanford. Written in Java. Makes movies.
+
* [http://wilma.sourceforge.net/ WilmaScope] - Real-time animations of dynamic graph structures. Written in Java. Sophisticated force model with strings and attraction.
+
* [http://www.caida.org/tools/visualization/walrus/ Walrus] - A 3-d graph network exploration tool. Employs 3D hyperbolic displays and layout based on a user-supplied spanning tree.
+
* [http://afterglow.sf.ne AfterGlowt] - A tool to convert CSV data into DOT descriptions with highly configurable instructions.
+
* http://tulip.labri.fr/TulipDrupal/ - Tulip is an information visualization framework dedicated to the analysis and visualization of relational data.  
+
  
See also: http://en.wikipedia.org/wiki/Social_network_analysis_software
+
== Advanced Format (4KB Sector) Hard Drives ==
 +
NTFS does not natively handle drives that use the new standard of 4KB sectors. For information on this, see [[Advanced Format]].
  
====Abandoned Social network Graphers====
+
== Transactional NTFS (TxF) ==
(Abandoned means that it hasn't been updated since 2009)
+
* [http://graphexploration.cond.org/ Guess: The Graph Exploration System] - Originally developed at HP, this is a large Jython/Java-based system that you can use for building your own applications. Distributed under GPL. (2007)
+
* [http://sourceforge.net/projects/ivc/ InfoVis Cyberinfrastructure] - Another graph drawing system written in Java. (2009)
+
* [http://www.andrew.cmu.edu/user/krack/krackplot.shtml Krackplot] - "KrackPlot is a program for network visualization designed for social network analysts." (2006; Windows only; no source, apparently)
+
* [http://www.sfu.ca/~richards/Multinet/Pages/multinet.htm MultiNet] - A data analysis package for drawing conventional data and graph data.
+
(2007)
+
* [http://developer.berlios.de/projects/nv2d/ NetVis 2D] - Another graph visualization and layout tool written in Java. (2005)
+
* [http://www.informatik.uni-bremen.de/uDrawGraph/en/uDrawGraph/uDrawGraph.html uDrawGraph]
+
* [http://ubietylab.net/ubigraph/ Ubigraph] - a tool for visualizing dynamic graphs. The basic version is free, and talks to Python, Ruby, PHP, Java, C, C++, C#, Haskell, and OCaml.  (2008)
+
  
====Computer Network Visualization====
+
According to MSDN Transactional NTFS (TxF) allows file operations on an NTFS file system volume to be performed in a transaction.  
* [http://tnv.sourceforge.net/ The Network Visualizer]
+
  
=== Commercial Graphic Applications and Tools===
+
Several TxF related file-system-metadata files can be found in the file-system-metadata directory: \$Extend\$RmMetadata\. TxF also uses the MFT attribute $LOGGING_UTILITY_STREAM with the name $TXF_DATA.
  
* [http://www.aisee.com/ aiSee Graph Layout Software] - Supports 15 layout algorithms, recursive graph nesting, and easy printing. Runs on [[Windows]], [[Linux]], [[Solaris]], [[NetBSD]], and [[MacOS]]. 30-day trial and free registered versions available. Academic pricing available.
+
TxF uses the [[Common Log File System (CLFS)]]
*  [http://www.geomantics.com/ Geomantics] - Geographical, Visualization and Graphics software. Runs on [[Windows]].
+
* [http://www.kylebank.com/ Graphis 2D and 3D graphing software] - Runs on [[Windows]]. Free 30-day evaluation copy available.
+
* [http://www.openviz.com/ OpenViz] and  [http://www.powerviz.com/ PowerViz] - Both from Advanced Visual Systems, super high-end visualization toolkits. $$$$
+
* [http://www.tomsawyer.com/ Tom Sawyer Software] Analysis, Visualizaiton, and Layout programs. - Heavy support for drawing graphs. Beautiful gallery. ActiveX, Java, C++ and .NET editions.
+
* [http://www.netminer.com/ NetMiner] - A comprehensive tool for Social Network Analysis. Runs on Windows, with a Linux version under development. $35 for "Express" student version, $250 for "Professional" student version, $950 for "Normal" "Professional" version.
+
* [http://www.analytictech.com/ucinet.htm UCINET] - A comprehensive package for the analysis of social network data as well as other 1-mode and 2-mode data.
+
* [http://www.clarifiednetworks.com/logster Logster] - an ultra-easy software tool to visualize Apache-style logs on a world map.
+
* [http://www.clarifiednetworks.com/Clarified%20Analyzer Clarified Analyzer] - Visualizes Network Traffic and allows to drill down from visualizations to the packet level.
+
  
== Visualization Toolkits and Libraries ==
+
== External links ==
===C/C++===
+
* [http://en.wikipedia.org/wiki/NTFS Wikipedia: NTFS]
* [http://public.kitware.com/VTK/ The Visualization Toolkit] - C++ multi-platform with interfaces available for Tcl/Tk, Java and Python. Professional support provided by [http://www.kitware.com/ Kitware].
+
* [http://msdn.microsoft.com/en-us/library/bb968806%28v=VS.85%29.aspx MSDN on Transactional NTFS]  
* [http://kdirstat.sourceforge.net/ KDirStat], an open source implementation of [http://www.cs.umd.edu/hcil/treemap-history/index.shtml Treemaps] written in C. (Treemaps are a visualization technique developed at the University of Maryland for visualizing large amounts of multi-dimensional data.)  You can find a copy of it in [http://www.derlien.com/ Disk Inventory X] and
+
* [http://en.wikipedia.org/wiki/Transactional_NTFS Wikipedia on Transactional NTFS]  
===Java===
+
* [http://www.tzworks.net/prototype_page.php?proto_id=12  Windows NTFS Metadata Extractor Utility] Free tool that can be run on Windows, Linux or Mac OS-X
* [http://csbi.sourceforge.net/index.html Graph Interface Library (GINY)] - Java
+
* [http://sourceforge.net/projects/linux-ntfs/files/NTFS%20Documentation/ Linux-ntfs Documentation] Detailed documentation of the NTFS format by the Linux-NTFS driver creators.
* [http://hypergraph.sourceforge.net/ HyperGraph] - Hyperbolic trees, in Java. Check out the home page. Try clicking on the logo...
+
* [http://ivtk.sourceforge.net/ InfoViz Toolkit] - Java, originally developed at [[INRA]].
+
* [https://jdigraph.dev.java.net/ Jdigrah] - Java Directed Graphs.
+
* [http://jgrapht.sourceforge.net/ JGraphT] - A Java visualization kit designed to be simple and extensible.
+
* [http://prefuse.sourceforge.net/ Perfuse] - A Java-based toolkit for building interactive information visualization applications
+
* [http://www.ssec.wisc.edu/~billh/visad.html#intro VisAD] - A Java component library for interactive and collaborative visualization.
+
* [http://www.softwaresecretweapons.com/jspwiki/Wiki.jsp?page=LinguineMaps Linguine Maps] - An open-source Java-based system for visualizing software call maps.
+
* [http://zvtm.sourceforge.net/index.html Zoomable Visual Transformation Machine] - Java. Originally started at Xerox Research Europe.
+
* [http://openmap.bbn.com/ OpenMap] A Java-based Geographical Information System framework, from [[BBN]].
+
===JavaScript===
+
* [http://mbostock.github.com/d3/ D3 - Data Driven Documents] - A visualization library for the Web.
+
===Unclassified===
+
* [http://gravisto.fim.uni-passau.de/ Gravisto: Graph Visualization Toolkit] - An editor and toolkit for developing graph visualization algorithms.
+
* [http://www.gnu.frb.br:8080/rox Rox Graph Theory Framework] - An open-source plug-in framework for graph theory visualization.
+
* [http://touchgraph.sourceforge.net/ TouchGraph] - Library for building graph-based interfaces.
+
  
==Journals and Conferences==
+
[[Category:Disk file systems]]
* [http://www.palgrave-journals.com/ivs/index.html Information Visualization Journal]
+
* [http://rw4.cs.uni-sb.de/~diehl/softvis/seminar/index.php?goto=seminar ACM Symposium on Software Visualization]
+
==Research Groups==
+
===Berkeley===
+
* [http://bailando.sims.berkeley.edu/infovis.html Bailando Visualization]
+
* [http://vis.berkeley.edu/ Berkeley Visualization Lab]
+
===Brown===
+
* [http://www.cs.brown.edu/people/rt/gd.html Roberto Tamassia's resources on Graph Drawing]
+
===Stanford===
+
* [http://window.stanford.edu/projects/rivet/ Rivet Project] (Visualization complex systems)
+
===UNM===
+
* [http://www.msi.umn.edu/user_support/scivis/scivis-list.html Scientific Visualization at the Supercomputing Institute]
+
===Wattenberg===
+
* [http://www.bewitched.com/ Bewitched], a one-man research group.
+
==See Also==
+
* [http://www-static.cc.gatech.edu/gvu/ii/resources/infovis.html GVU's Information Visualization Resources link farm]
+
* [http://directory.google.com/Top/Science/Math/Combinatorics/Software/Graph_Drawing/ Google Directory of Graph Drawing Software]
+
* [http://directory.fsf.org/science/visual/ GNU Free Software directory of scientific visualization software]
+
* [http://www.manageability.org/blog/stuff/open-source-graph-network-visualization-in-java/view Open Source Graph Network Visualization in Java]
+
* [http://www.insna.org/INSNA/soft_inf.html INSNA's web page of Computer Programs for Social Network Analysis]
+
* http://www.dmoz.org/Science/Math/Combinatorics/Software/Graph_Drawing/
+
* http://www.ontopia.net/omnigator/models/index.jsp
+
* http://visant.bu.edu/
+
* http://sourceforge.net/projects/touchgraph/
+
 
+
CAIDA has 15+ years of work visualizing Internet topologies. You may find their tools to be useful:
+
* http://www.caida.org/tools/visualization/
+
* http://www.caida.org/publications/visualizations/
+
* http://www.caida.org/tools/visualization/walrus/gallery1/
+
* http://www.caida.org/research/topology/as_core_network/
+

Revision as of 02:09, 16 March 2012

The New Technology File System (NTFS) is a file system developed and introduced by Microsoft in 1995 with Windows NT. As a replacement for the FAT file system, it quickly became the standard for Windows 2000, Windows XP and Windows Server 2003.

The features of NTFS include:

Time Stamps

NTFS keeps track of lots of time stamps. Each file has a time stamp for 'Create', 'Modify', 'Access', and 'Entry Modified'. The latter refers to the time when the MFT entry itself was modified. These four values are commonly abbreviated as the 'MACE' values. Note that other attributes in each MFT record may also contain timestamps that are of forensic value.

Additional information on how NTFS timestamps work when files are moved or copied is available here: Microsoft KB 299648

Changes in Windows Vista

In Windows Vista, NTFS no longer tracks the Last Access time of a file by default. This feature can be enabled by the user if desired via setting the registry key 'HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem\NtfsDisableLastAccessUpdate' to '0'.

Alternate Data Streams

The NTFS file system includes a feature referred to as Alternate Data Streams (ADSs). This feature has also been referred to as "multiple data streams", "alternative data streams", etc. ADSs were included in NTFS in order to support the resource forks employed by the Hierarchal File System (HFS) employed by Macintosh systems.

As of Windows XP SP2, files downloaded via Internet Explorer, Outlook, and Windows Messenger were automatically given specific "zoneid" ADSs. The Windows Explorer shell would then display a warning when the user attempted to execute these files (by double-clicking them).

Sysadmins should be aware that prior to Vista, there are no tools native to the Windows platform that would allow you to view the existence of arbitrary ADSs. While ADSs can be created and their contents executed or viewed, it wasn't until the "/r" switch was introduced with the "dir" command on Vista that arbitrary ADSs would be visible. Prior to this, tools such as LADS could be used to view the existence of these files.

Microsoft FSRM (File System Resource Manager) also uses ADS as part of 'file classification'.

Examiners should be aware that most forensic analysis applications, including EnCase and ProDiscover, will display ADSs found in acquired images in red.

Advanced Format (4KB Sector) Hard Drives

NTFS does not natively handle drives that use the new standard of 4KB sectors. For information on this, see Advanced Format.

Transactional NTFS (TxF)

According to MSDN Transactional NTFS (TxF) allows file operations on an NTFS file system volume to be performed in a transaction.

Several TxF related file-system-metadata files can be found in the file-system-metadata directory: \$Extend\$RmMetadata\. TxF also uses the MFT attribute $LOGGING_UTILITY_STREAM with the name $TXF_DATA.

TxF uses the Common Log File System (CLFS)

External links