Difference between pages "Imager NG Ideas" and "Barnyard2"

From ForensicsWiki
(Difference between pages)
Jump to: navigation, search
(Supportive tooling)
 
 
Line 1: Line 1:
This page is for discussing ideas regarding next-generation (NG) imaging tools.
+
==About ==
 +
Barnyard2 is an open source interpreter for Snort unified2 binary output files.
 +
Its primary use is allowing Snort to write to disk in an efficient manner and
 +
leaving the task of parsing binary data into various formats to a separate
 +
process that will not cause Snort to miss network traffic.
  
Note that some of the ideas mentioned can be already used by imaging tools, but the idea of this page is to determine how useful these features could be for next-generation of imaging tools.
+
Barnyard2 has 3 modes of operation:
The scope is mainly a software-based imaging tools, but not limited to. Some features might not be doable, because of limitations of certain image file formats.
+
  1. batch (or one-shot),
 +
  2. continual, and
 +
  3. continual w/ bookmark.
  
Please, do not delete text (ideas) here. Use something like this:
+
In batch (or one-shot) mode, barnyard2 will process the explicitly specified
 +
file(s) and exit.
  
<pre>
+
In continual mode, barnyard2 will start with a location to look and a specified
<s>bad idea</s>
+
file pattern and continue to process new data (and new spool files) as they
good idea
+
appear.
</pre>
+
  
This will look like:
+
Continual mode w/ bookmarking will also use a checkpoint file (or waldo file in
 +
the snort world) to track where it is. In the event the barnyard2 process ends
 +
while a waldo file is in use, barnyard2 will resume processing at the last
 +
entry as listed in the waldo file.
  
<s>bad idea</s>
+
The "-f", "-w", and "-o" options are used to determine which mode barnyard2
 +
will run in.  It is legal for both the "-f" and "-w" options to be used on the
 +
command line at the same time, however any data that exists in the waldo file
 +
will override the command line data from the "-f" and "-d" options. See the
 +
command directives section below for more detail.
  
good idea
+
Barnyard2 processing is controlled by two main types of directives: input
 +
processors and output plugins. The input processors read information in from a
 +
specific format ( currently the spo_unified2 output module of Snort ) and
 +
output them in one of several ways.[https://github.com/firnsy/barnyard2/]
  
= License =
+
==History ==
 +
Barnyard is a critical tool for the parsing of Snort's unified binary files, processing and on-forwarding to a variety of output plugins. Unfortunately it has not seen an updated in over 4 years and is not going to be maintained by the original developers. With the new version of the unified format (ie. unified2) arriving we need something to bridge this gap.
 +
To quote directly from the Snort FAQ:
 +
* "Barnyard is an output system for Snort. Snort creates a special binary output format called unified. Barnyard reads this file, and then resends the data to a database backend. Unlike the database output plug-in, Barnyard is aware of a failure to send the alert to the database, and it stops sending alerts. It is also aware when the database can accept connections again and will start sending the alerts again."
  
= Features =
+
The SXL team love barnyard. So much so that we want it to stay and have been tinkering around with the code to give it a breath of new life. Here is what we have achieved to far for this reinvigorated code base:
* Compression
+
Parsing of the new unified2 log files.
* Integrity checks
+
*Maintaining majority of the command syntax of barnyard.
* Encryption
+
*Addressed all associated bug reports and feature requests arising since barnyard-0.2.0.
* Error correction (parity)
+
*Completely rewritten code based on the GPLv2 Snort making it entirely GPLv2.
* Pre-processing during imaging
+
* User suspend/resume, resume after failure
+
* Remote imaging
+
* Error resistance in reading storage media, e.d. disks
+
** maybe have different techniques, e.g. to use for heavily damaged storage media
+
* Support different types of storage media
+
** disk
+
** volume
+
** optical discs
+
** memory
+
** files and directories
+
* Store relevant data about the storage media and the imaging process
+
** read errors
+
* Support multiple image format
+
** not all image formats have support for all the features
+
  
== Compression ==
+
== References ==
* Reduces the amount of data that needs to be written; improved the overall imaging speed.
+
All information on this page is referenced to [http://www.securixlive.com/barnyard2/about.php securixlive.com] where further information about Barnyard can be found.
** hash-based imaging
+
<references />
** detection of easy (emtpy-block) and hard (encrypted block) to compress data
+
** multi-threaded compression
+
** sparse ranges
+
** de-duplication
+
 
+
=== de-duplication ===
+
* hash-based imaging
+
* sparse or repeated ranges
+
* pattern-fill
+
 
+
== Integrity checks ==
+
* Integrity hash (MD5, SHA1, SHA256)
+
* piecewise hashing
+
 
+
= Supportive tooling =
+
== Image verification ==
+
* modes:
+
** full verification and print a report at the end
+
** stop on error (useful for automation?)
+
 
+
= Image format =
+
Implied features for an image format
+
* High-speed imaging
+
* Compact storage
+
* Error-resistant storage (over a longer time)
+
* Minimal overhead on read
+
* Evidence bag
+
** multiple images in one image format
+
** support for additional information e.g. case data
+
 
+
[[Category:Research]]
+

Revision as of 16:19, 13 March 2013

About

Barnyard2 is an open source interpreter for Snort unified2 binary output files. Its primary use is allowing Snort to write to disk in an efficient manner and leaving the task of parsing binary data into various formats to a separate process that will not cause Snort to miss network traffic.

Barnyard2 has 3 modes of operation:

 1. batch (or one-shot),
 2. continual, and
 3. continual w/ bookmark.

In batch (or one-shot) mode, barnyard2 will process the explicitly specified file(s) and exit.

In continual mode, barnyard2 will start with a location to look and a specified file pattern and continue to process new data (and new spool files) as they appear.

Continual mode w/ bookmarking will also use a checkpoint file (or waldo file in the snort world) to track where it is. In the event the barnyard2 process ends while a waldo file is in use, barnyard2 will resume processing at the last entry as listed in the waldo file.

The "-f", "-w", and "-o" options are used to determine which mode barnyard2 will run in. It is legal for both the "-f" and "-w" options to be used on the command line at the same time, however any data that exists in the waldo file will override the command line data from the "-f" and "-d" options. See the command directives section below for more detail.

Barnyard2 processing is controlled by two main types of directives: input processors and output plugins. The input processors read information in from a specific format ( currently the spo_unified2 output module of Snort ) and output them in one of several ways.[1]

History

Barnyard is a critical tool for the parsing of Snort's unified binary files, processing and on-forwarding to a variety of output plugins. Unfortunately it has not seen an updated in over 4 years and is not going to be maintained by the original developers. With the new version of the unified format (ie. unified2) arriving we need something to bridge this gap. To quote directly from the Snort FAQ:

  • "Barnyard is an output system for Snort. Snort creates a special binary output format called unified. Barnyard reads this file, and then resends the data to a database backend. Unlike the database output plug-in, Barnyard is aware of a failure to send the alert to the database, and it stops sending alerts. It is also aware when the database can accept connections again and will start sending the alerts again."

The SXL team love barnyard. So much so that we want it to stay and have been tinkering around with the code to give it a breath of new life. Here is what we have achieved to far for this reinvigorated code base: Parsing of the new unified2 log files.

  • Maintaining majority of the command syntax of barnyard.
  • Addressed all associated bug reports and feature requests arising since barnyard-0.2.0.
  • Completely rewritten code based on the GPLv2 Snort making it entirely GPLv2.

References

All information on this page is referenced to securixlive.com where further information about Barnyard can be found. <references />