File Format Identification

File Format Identification is the process of figuring out the format of a sequence of bytes. Operating systems typically do this by file extension or by embedded MIME information. Forensic applications need to identify file types by content.

Tools

libmagic

• Written in C.
• Rules in /usr/share/file/magic and compiled at runtime.
• Powers the Unix “file” command, but you can also call the library directly from a C program.
• http://sourceforge.net/projects/libmagic

DROID

• Writen in Java
• Developed by PRONOM at the National Archives of the United Kingdom.
• http://droid.sourceforge.net

JHOVE

JSTOR/Harvard Object Validation Environment

• JHOVE
• JHOVE2

TrID

• XML config file
• Closed source; free for non-commercial use
• http://mark0.net/soft-trid-e.html

Forensic Innovations File Investigator TOOLS

• Proprietary, but free trial available.
• Available as consumer applications and OEM API.
• Identifies 3,000+ file types, using multiple methods to maintain high accuracy.
• Extracts metadata for many of the supported file types.
• http://www.forensicinnovations.com/fitools.html

Stellent/Oracle Outside-In

• Proprietary but free demo.
• http://www.oracle.com/technology/products/content-management/oit/oit_all.html

Forensic Assistant

• Proprietary.
• Provides detection of password protected archives, some files of cryptographic programs, Pinch/Zeus binary reports, etc.
• http://nhtcu.ru/0xFA_eng.html

Data Sets

If you are working in the field of file format identification, please consider reporting the results of your algorithm with one of these publicly available data sets:

• NPS govdocs1m - a corpus of 1 million files that can be redistributed without concern of copyright or PII. Download from http://domex.nps.edu/corp/files/govdocs1/
• The NPS Disk Corpus - a corpus of realistic disk images that contain no PII. Information is at: http://digitalcorpora.org/?s=nps

Bibliography

Current research papers on the file format identification problem. Most of these papers concern themselves with identifying file format of a few file sectors, rather than an entire file. Please note that this bibliography is in chronological order!

2001

• Mason McDaniel, Automatic File Type Detection Algorithm, Masters Thesis, James Madison University,2001

2003

• Content Based File Type Detection Algorithms, Mason McDaniel and M. Hossain Heydari, 36th Annual Hawaii International Conference on System Sciences (HICSS'03) - Track 9, 2003.

2005

• Fileprints: identifying file types by n-gram analysis, LiWei-Jen, Wang Ke, Stolfo SJ, Herzog B.., IProceeding of the 2005 IEEE workshop on information assurance, 2005. (Presentation Slides) (PDF)

• Douglas J. Hickok, Daine Richard Lesniak, Michael C. Rowe, File Type Detection Technology, 2005 Midwest Instruction and Computing Symposium.(PDF)

2006

• Karresand Martin, Shahmehri Nahid File type identification of data fragments by their binary structure. , Proceedings of the IEEE workshop on information assurance, pp.140–147, 2006.(Presentation Slides)

• Gregory A. Hall, Sliding Window Measurement for File Type Identification, Computer Forensics and Intrusion Analysis Group, ManTech Security and Mission Assurance, 2006. (PDF)

• FORSIGS; Forensic Signature Analysis of the Hard Drive for Multimedia File Fingerprints, John Haggerty and Mark Taylor, IFIP TC11 International Information Security Conference, 2006, Sandton, South Africa.

• Martin Karresand , Nahid Shahmehri, "Oscar -- Using Byte Pairs to Find File Type and Camera Make of Data Fragments," Annual Workshop on Digital Forensics and Incident Analysis, Pontypridd, Wales, UK, pp.85-94, Springer-Verlag, 2006.

2007

• Karresand M., Shahmehri N., Oscar: File Type Identification of Binary Data in Disk Clusters and RAM Pages, Proceedings of IFIP International Information Security Conference: Security and Privacy in Dynamic Environments (SEC2006), Springer, ISBN 0-387-33405-x, pp.413-424, Karlstad, Sweden, May 2006.

• Robert F. Erbacher and John Mulholland, "Identification and Localization of Data Types within Large-Scale File Systems," Proceedings of the 2nd International Workshop on Systematic Approaches to Digital Forensic Engineering, Seattle, WA, April 2007.

• Ryan M. Harris, "Using Artificial Neural Networks for Forensic File Type Identification," Master's Thesis, Purdue University, May 2007. (PDF)

• Predicting the Types of File Fragments, William Calhoun, Drue Coles, DFRWS 2008. (Presentation Slides) (PDF)

• Sarah J. Moody and Robert F. Erbacher, SÁDI – Statistical Analysis for Data type Identification, 3rd International Workshop on Systematic Approaches to Digital Forensic Engineering, 2008.

2008

• Mehdi Chehel Amirani, Mohsen Toorani, and Ali Asghar Beheshti Shirazi, A New Approach to Content-based File Type Detection, Proceedings of the 13th IEEE Symposium on Computers and Communications (ISCC'08), pp.1103-1108, July 2008. (PDF)

2009

• Roussev, Vassil, and Garfinkel, Simson, "File Classification Fragment-The Case for Specialized Approaches," Systematic Approaches to Digital Forensics Engineering (IEEE/SADFE 2009), Oakland, California. (PDF)

• Irfan Ahmed, Kyung-suk Lhee, Hyunjung Shin and ManPyo Hong, On Improving the Accuracy and Performance of Content-based File Type Identification, Proceedings of the 14th Australasian Conference on Information Security and Privacy (ACISP 2009), pp.44-59, LNCS (Springer), Brisbane, Australia, July 2009.

2010

• Irfan Ahmed, Kyung-suk Lhee, Hyunjung Shin and ManPyo Hong, Fast File-type Identification, Proceedings of the 25th ACM Symposium on Applied Computing (ACM SAC 2010), ACM, Sierre, Switzerland, March 2010.