Difference between pages "File Format Identification" and "SANS Investigative Forensic Toolkit Workstation"

From ForensicsWiki
(Difference between pages)
Jump to: navigation, search
 
m
 
Line 1: Line 1:
File Format Identification is the process of figuring out the format of a sequence of bytes. Operating systems typically do this by file extension or by embedded MIME information. Forensic applications need to identify file types by content.
+
'''The SANS SIFT Workstation''' is a [[VMware]] Appliance that is preconfigured with all the necessary tools to perform a forensic examination. It is compatible with [[Encase | Expert Witness Format]] (E01), Advanced Forensic Format ([[AFF]]), and raw (dd) evidence formats.
  
 +
== Overview ==
  
=Tools=
+
SIFT Workstation is based on Ubuntu.
==libmagic==
+
* Written in C.  
+
* Rules in /usr/share/file/magic and compiled at runtime.
+
* Powers the Unix “file” command, but you can also call the library directly from a C program.
+
* http://sourceforge.net/projects/libmagic
+
  
==Digital Preservation Efforts==
+
Software Includes:
PRONOM is  a project of the National Archives of the United Kingdom to develop a registry of file types. A similar project was started by JSTOR and Harvard as the JSTOR/Harvard Object Validation Environment. Attempts are now underway to merge these two efforts in the Global Digital Format Registry and the Universal Digital Format Registry.
+
  
The UK National Archives developed the Digital Record Object Identification (DROID) tool, an "automatic file format identification tool." This tool is written in Java and can be downloaded from SourgeForge.
+
# [[The Sleuth Kit]]
 +
# [[ssdeep]] & [[md5deep]]
 +
# [[Foremost]]/[[Scalpel]]
 +
# [[Wireshark]]
 +
# HexEditor
 +
# [[Vinetto]] ([[thumbs.db]] examination)
 +
# Pasco
 +
# Rifiuti
 +
# [[Volatility Framework]]
 +
# DFLabs PTK (GUI Front-End for [[Sleuthkit]])
 +
# [[Autopsy]] (GUI Front-End for [[Sleuthkit]])
  
See:
+
The SIFT Workstation will allow evidence to be viewed from a Windows workstation. The /images directory and the evidence mount point, the /mnt/hack directory, can be viewed from the local Windows operating system.
* [http://www.nationalarchives.gov.uk/PRONOM/Default.aspx  PRONOM]
+
* [http://hul.harvard.edu/jhove/ JHOVE]
+
* [https://wiki.ucop.edu/display/JHOVE2Info/Home JHOVE2]
+
* [http://www.gdfr.info/  GDFR]
+
* [http://www.udfr.org/  UDFR]
+
* [http://droid.sourceforge.net DROID download]
+
  
==TrID==
+
== Links ==
* XML config file
+
* Closed source; free for non-commercial use
+
* http://mark0.net/soft-trid-e.html
+
  
==Forensic Innovations File Investigator TOOLS==
+
* [http://forensics.sans.org/community/downloads/ Computer Forensics and e-Discovery downloads]
* Proprietary, but free trial available.
+
[[Category:VMWare Appliances]]
* Available as consumer applications and OEM API.
+
* Identifies 3,000+ file types, using multiple methods to maintain high accuracy.
+
* Extracts metadata for many of the supported file types.
+
* http://www.forensicinnovations.com/fitools.html
+
 
+
==Stellent/Oracle Outside-In==
+
* Proprietary but free demo.
+
* http://www.oracle.com/technology/products/content-management/oit/oit_all.html
+
 
+
==[[Forensic Assistant]]==
+
* Proprietary.
+
* Provides detection of password protected archives, some files of cryptographic programs, Pinch/Zeus binary reports, etc.
+
* http://nhtcu.ru/0xFA_eng.html
+
[[Category:Tools]]
+
 
+
=Data Sets=
+
If you are working in the field of file format identification, please consider reporting the results of your algorithm with one of these publicly available data sets:
+
* NPS govdocs1m - a corpus of 1 million files that can be redistributed without concern of copyright or PII. Download from http://domex.nps.edu/corp/files/govdocs1/
+
* The NPS Disk Corpus - a corpus of realistic disk images that contain no PII. Information is at: http://digitalcorpora.org/?s=nps
+
 
+
=Bibliography=
+
Current research papers on the file format identification problem. Most of these papers concern themselves with identifying file format of a few file sectors, rather than an entire file.  '''Please note that this bibliography is in chronological order!'''
+
 
+
 
+
;2001
+
 
+
* Mason McDaniel, [[Media:Mcdaniel01.pdf|Automatic File Type Detection Algorithm]], Masters Thesis, James Madison University,2001
+
 
+
; 2003
+
 
+
* [http://www2.computer.org/portal/web/csdl/abs/proceedings/hicss/2003/1874/09/187490332a.pdf Content Based File Type Detection Algorithms], Mason McDaniel and M. Hossain Heydari, 36th Annual Hawaii International Conference on System Sciences (HICSS'03) - Track 9, 2003.
+
 
+
; 2005
+
 
+
* Fileprints: identifying file types by n-gram analysis, LiWei-Jen, Wang Ke, Stolfo SJ, Herzog B..,  IProceeding of the 2005 IEEE workshop on information assurance, 2005. ([http://www.itoc.usma.edu/workshop/2005/Papers/Follow%20ups/FilePrintPresentation-final.pdf Presentation Slides])  ([http://www1.cs.columbia.edu/ids/publications/FilePrintPaper-revised.pdf PDF])
+
 
+
* Douglas J. Hickok, Daine Richard Lesniak, Michael C. Rowe, File Type Detection Technology,  2005 Midwest Instruction and Computing Symposium.([http://www.micsymposium.org/mics_2005/papers/paper7.pdf PDF])
+
 
+
; 2006
+
 
+
* Karresand Martin, Shahmehri Nahid [http://ieeexplore.ieee.org/iel5/10992/34632/01652088.pdf  File type identification of data fragments by their binary structure. ], Proceedings of the IEEE workshop on information assurance, pp.140–147, 2006.([http://www.itoc.usma.edu/workshop/2006/Program/Presentations/IAW2006-07-3.pdf Presentation Slides])
+
 
+
* Gregory A. Hall, Sliding Window Measurement for File Type Identification, Computer Forensics and Intrusion Analysis Group, ManTech Security and Mission Assurance, 2006. ([http://www.mantechcfia.com/SlidingWindowMeasurementforFileTypeIdentification.pdf PDF])
+
 
+
* FORSIGS; Forensic Signature Analysis of the Hard Drive for Multimedia File Fingerprints, John Haggerty and Mark Taylor, IFIP TC11 International Information Security Conference, 2006, Sandton, South Africa.
+
 
+
* Martin Karresand , Nahid Shahmehri, "Oscar -- Using Byte Pairs to Find File Type and Camera Make of Data Fragments," Annual Workshop on Digital Forensics and Incident Analysis, Pontypridd, Wales, UK, pp.85-94, Springer-Verlag, 2006.
+
 
+
; 2007
+
 
+
* Karresand M., Shahmehri N., [http://dx.doi.org/10.1007/0-387-33406-8_35 Oscar: File Type Identification of Binary Data in Disk Clusters and RAM Pages], Proceedings of IFIP International Information Security Conference: Security and Privacy in Dynamic Environments (SEC2006), Springer, ISBN 0-387-33405-x, pp.413-424, Karlstad, Sweden, May 2006.
+
 
+
* Robert F. Erbacher and John Mulholland, "Identification and Localization of Data Types within Large-Scale File Systems," Proceedings of the 2nd International Workshop on Systematic Approaches to Digital Forensic Engineering, Seattle, WA, April 2007.
+
 
+
* Ryan M. Harris, "Using Artificial Neural Networks for Forensic File Type Identification," Master's Thesis, Purdue University, May 2007. ([https://www.cerias.purdue.edu/tools_and_resources/bibtex_archive/archive/2007-19.pdf PDF])
+
 
+
* Predicting the Types of File Fragments, William Calhoun, Drue Coles, DFRWS 2008. ([http://www.dfrws.org/2008/proceedings/p14-calhoun_pres.pdf Presentation Slides])  ([http://www.dfrws.org/2008/proceedings/p14-calhoun.pdf PDF])
+
 
+
* Sarah J. Moody and Robert F. Erbacher, [http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=04545366 SÁDI – Statistical Analysis for Data type Identification], 3rd International Workshop on Systematic Approaches to Digital Forensic Engineering, 2008.
+
 
+
; 2008
+
 
+
* Mehdi Chehel Amirani, Mohsen Toorani, and Ali Asghar Beheshti Shirazi, [http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4625611 A New Approach to Content-based File Type Detection], Proceedings of the 13th IEEE Symposium on Computers and Communications (ISCC'08), pp.1103-1108, July 2008.  ([http://arxiv.org/ftp/arxiv/papers/1002/1002.3174.pdf PDF])
+
 
+
; 2009
+
* Roussev, Vassil, and Garfinkel, Simson, "File Classification Fragment-The Case for Specialized Approaches," Systematic Approaches to Digital Forensics Engineering (IEEE/SADFE 2009), Oakland, California. ([http://simson.net/clips/academic/2009.SADFE.Fragments.pdf PDF])
+
 
+
* Irfan Ahmed, Kyung-suk Lhee, Hyunjung Shin and ManPyo Hong, [http://www.springerlink.com/content/g2655k2044615q75/ On Improving the Accuracy and Performance of Content-based File Type Identification], Proceedings of the 14th Australasian Conference on Information Security and Privacy (ACISP 2009), pp.44-59, LNCS (Springer), Brisbane, Australia, July 2009.
+
 
+
; 2010
+
*Irfan Ahmed, Kyung-suk Lhee, Hyunjung Shin and ManPyo Hong, [http://www.alphaminers.net/sub05/sub05_03.php?swf_pn=5&swf_sn=3&swf_pn2=3    Fast File-type Identification], Proceedings of the 25th ACM Symposium on Applied Computing (ACM SAC 2010), ACM, Sierre, Switzerland, March 2010.
+
 
+
;2011
+
*Irfan Ahmed, Kyung-Suk Lhee, Hyun-Jung Shin, Man-Pyo Hong, [http://link.springer.com/chapter/10.1007/978-3-642-24212-0_5 Fast Content-Based File Type Identification], Proceedings of the 7th Annual IFIP WG 11.9 International Conference on Digital Forensics, Orlando, FL, USA, February, 2011
+
[[Category:Bibliographies]]
+

Latest revision as of 17:55, 15 June 2014

The SANS SIFT Workstation is a VMware Appliance that is preconfigured with all the necessary tools to perform a forensic examination. It is compatible with Expert Witness Format (E01), Advanced Forensic Format (AFF), and raw (dd) evidence formats.

Overview

SIFT Workstation is based on Ubuntu.

Software Includes:

  1. The Sleuth Kit
  2. ssdeep & md5deep
  3. Foremost/Scalpel
  4. Wireshark
  5. HexEditor
  6. Vinetto (thumbs.db examination)
  7. Pasco
  8. Rifiuti
  9. Volatility Framework
  10. DFLabs PTK (GUI Front-End for Sleuthkit)
  11. Autopsy (GUI Front-End for Sleuthkit)

The SIFT Workstation will allow evidence to be viewed from a Windows workstation. The /images directory and the evidence mount point, the /mnt/hack directory, can be viewed from the local Windows operating system.

Links