LibAFF4 - Revision history

Joachim Metz: /* Obtaining LibAFF4 */

2010-08-20T13:07:17Z

‎Obtaining LibAFF4

Joachim Metz: /* Obtaining LibAFF4 */

2010-08-20T13:02:28Z

‎Obtaining LibAFF4

Joachim Metz: /* Obtaining LibAFF4 */

2010-08-20T12:53:42Z

‎Obtaining LibAFF4

Scudette at 10:16, 26 August 2009

2009-08-26T10:16:42Z

Scudette at 23:42, 25 April 2009

2009-04-25T23:42:11Z

Scudette at 12:50, 25 April 2009

2009-04-25T12:50:09Z

Scudette at 12:12, 25 April 2009

2009-04-25T12:12:52Z

Show changes

Scudette: New page: LibAFF4 is an implementation of the AFF4 standard. It is still in a very early stage of development but it allows people to have a play with the new standard. This page documents it. == O...

2009-04-25T11:44:36Z

New page: LibAFF4 is an implementation of the AFF4 standard. It is still in a very early stage of development but it allows people to have a play with the new standard. This page documents it. == O...

New page

LibAFF4 is an implementation of the AFF4 standard. It is still in a very early stage of development but it allows people to have a play with the new standard. This page documents it.

== Obtaining LibAFF4 ==

The source code is currently kept under version control on the main PyFlag server. It can be fetched using:

<code>
darcs get http://www.pyflag.net/AFF4/
</code>

Once the AFF4 directory is downloaded, a configure file needs to be created:

<code>
sh autogen.sh
./configure
make install
</code>

The following libraries are mandatory:

* openssl-dev
* uuid-dev

The following are optional but recommended libraries:

* libcurl4-openssl-dev
* libfuse-dev

== Taking an image ==

Probably the first step is to actually acquire the image. Right now there is no fancy interface or GUI - its just a simple commandline:

<pre>
AFF4/tools$ sudo aff4imager -i -o /tmp/test.zip /dev/sda1
Imaging Mode selected
Dumping segment urn:aff4:dd3102c1-370d-4b15-83e7-5a1991b5a9e1/00000000 (55425084 bytes)
Dumping segment urn:aff4:dd3102c1-370d-4b15-83e7-5a1991b5a9e1/00000001 (33314018 bytes)
Dumping segment urn:aff4:dd3102c1-370d-4b15-83e7-5a1991b5a9e1/00000002 (3011613 bytes)
Dumping segment urn:aff4:dd3102c1-370d-4b15-83e7-5a1991b5a9e1/00000003 (51756091 bytes)
Creating a link object 'default' for stream 'urn:aff4:dd3102c1-370d-4b15-83e7-5a1991b5a9e1'

</pre>

We can see the imager dumping a new zip file with the default settings. By default bevies are 64Mb in size. The aff4imager tool reports each new bevy which is saved to disk and the size it ended up taking (after compression). Finally we see a link object is created to the new stream to make it easier to access it (so we dont need to remember the long URN for it).

== Applying the Sleuthkit patch ==
There is a patch to sleuthkit for adding support to AFF4. The patch is against sleuthkit-3.0.1.

<pre>
tar -xvzf sleuthkit-3.0.1.tgz
cd sleuthkit-3.0.1
patch -p1 < /path/to/AFF4/patches/sleuthkit-3.0.1.patch

## The following is needed to get the autoconf system to pick up the changes
libtoolize --force
aclocal -I config
autoheader
automake
autoconf

## Now build it as normal
./configure
make install
</pre>

Currently the Sluethkit patch does not work with autodetection - hence it must be specified explicitely using <pre>-i aff4</pre>.

AFF4 is a bit different since it refers not only to a volume, but to a stream within the volume. The convension is to specify the stream as the last parameter after the volumes. So for example, to open the stream created in the example above:
<pre>
$ fls -i aff4 /tmp/test.zip default
zip.c:51 - FileBackedObject_Con: Can't open file:///tmp/test.zip/__URN__ (Not a directory)
libaff4.c:27 - open_volume: Loaded zip volume file:///tmp/test.zip
zip.c:51 - FileBackedObject_Con: Can't open file://default/__URN__ (No such file or directory)
zip.c:51 - FileBackedObject_Con: Can't open file://default (No such file or directory)
libaff4.c:100 - aff4_open: Trying to open stream default relative to all volumes
libaff4.c:107 - aff4_open: Trying urn:aff4:aeb9ed57-8dbe-4506-9fb8-e4106bf0767d/default
libaff4.c:122 - aff4_open: Using urn:aff4:dd3102c1-370d-4b15-83e7-5a1991b5a9e1 as the stream name
d/d 11: lost+found
r/r 10043: System.map-2.6.22-14-generic
r/r 6025: vmlinuz-2.6.22-14-generic
r/r 10041: config-2.6.22-14-generic
r/r 10042: abi-2.6.22-14-generic
r/r 13: initrd.img-2.6.22-14-generic
d/c 44177: grub
r/r 6029: initrd.img-2.6.24-21-generic
r/r 12: initrd.img-2.6.22-14-generic.bak
.....
</pre>

The debugging messages above are instructive to explain the log of how libaff4 opens the volumes. Since you can specify any number of volumes of the command line you can see that libaff4 first tries to open the volume as a directory volume, if that fails, it tries to open it as a zip file. The last element is assumed to be a volume name and if no such volume is found, its assumed to be a stream name. Then the stream name is searched in all the volumes specified. Finally the stream name is resolved (since its a link it returns the URN of the original stream).

@@ Line 1: / Line 1: @@
 LibAFF4 is an implementation of the AFF4 standard. It is still in a very early stage of development but it allows people to have a play with the new standard. This page documents it.
-== Obtaining LibAFF4 ==
+== Installing LibAFF4 ==
 The source code is currently kept under version control on Google code ( http://code.google.com/p/aff4/ ).
-It can be fetched using the Mercurial Source Control Management (SCM) tools:
+It can be obtained using the Mercurial Source Control Management (SCM) tools. To create a new copy of the AFF4 source tree.
 <code>
@@ Line 10: / Line 10: @@
 </code>
-Once the AFF4 directory (source tree) is downloaded it can be build and installed as following:
+Once the AFF4 source tree is downloaded in the aff4 directory, it can be build and installed as following:
 <code>
 ./scon install
 </code>
 AFF4 requires the following packages:

@@ Line 3: / Line 3: @@
 == Obtaining LibAFF4 ==
-The source code is currently kept under version control on Google code ( http://code.google.com/p/aff4/ ). It can be fetched using:
+The source code is currently kept under version control on Google code ( http://code.google.com/p/aff4/ ).
 <code>
@@ Line 9: / Line 10: @@
 </code>
-Once the AFF4 directory is downloaded it can be build and installed as following:
+Once the AFF4 directory (source tree) is downloaded it can be build and installed as following:
 <code>
@@ Line 15: / Line 16: @@
 </code>
-The following libraries are mandatory:
+AFF4 requires the following packages:
-* openssl-dev
+* OpenSSL development package (Debian: openssl-dev, Fedora: openssl-devel)
-* uuid-dev
+* UUID development package (Debian: uuid-dev, Fedora (recent version): libuuid-devel)
-The following are optional but recommended libraries:
+The following are optional but recommended packages:
 * libcurl4-openssl-dev

← Older revision		Revision as of 23:42, 25 April 2009
Line 286:		Line 286:

	As can be seen the fsbuilder tool creates new map streams for each new file found in the stream. The map streams are pushed to a new volume. Its possible now to fuse mount everything.		As can be seen the fsbuilder tool creates new map streams for each new file found in the stream. The map streams are pushed to a new volume. Its possible now to fuse mount everything.
		+
		+
		+	== Using HTTP for evidence management ==
		+	In practice, most large practitioners will need to manage and archive vast quantity of evidence. Its generally impossible to keep all the evidence you will ever need on the drive of your analysis system, and even if you get a vary large SAN its challanging to keep everything handy.
		+
		+	By allowing evidence to be managed by HTTP its possible to distribute the evidence around any number of HTTP servers, which can be deployed at any location throughout the corporate WAN. This allows very cheap storage to be used and the flexibility to move images around the network at will.
		+
		+	Similarly, current technologies for remote imaging are complex and proprietary. In a corporate LAN environment, for example, its beneficial to push the image directly to an imaging server from the target machine booted from a boot CD for example. Its nice to be able to do so with no prior preparation.
		+
		+	Since AFF4 uses URNs to describes all objects in the AFF4 universe, and a URL is just a special kind of URN, it is possible to describe everything in terms of URLs too. For example, its possible to specify certificates, keys, and other file names as a URL as well (which means they can reside on a HTTP server). For example, its possible to encrypt the private key and place it on a HTTP server to allow an investigator to sign an acquisition from anywhere on the network without having to manage their own keys and certificates.
		+
		+	Similarly its possible for investigators to create the image directly on the HTTP server using webdav. This section will discuss how to do this with an Apache2 server, but clearly since webdav is a standard its possible to use any other webserver too.
		+
		+	Note: Windows XP webdav support is widely known to be very buggy. If it works it may be possible to "open a web folder" to mount the remote webdav folder.
		+
		+	=== Apache configuration ===
		+	Our goal with this configuration is to allow anyone read access to the images with no authentication, but to require authentication for writing on the webdav directory:
		+
		+	<pre>
		+	<VirtualHost *:80>
		+	<Directory /var/www/webdav>
		+	DAV On
		+	AuthType Basic
		+	AuthName "test"
		+	AuthUserFile /etc/apache2/passwd.dav
		+
		+	<Limit PUT POST DELETE PROPFIND PROPPATCH MKCOL COPY MOVE LOCK UNLOCK>
		+	Require valid-user
		+	</Limit>
		+	</Directory>
		+	</VirtualHost>
		+	</pre>
		+
		+	You can add usernames to the password file /etc/apache2/passwd.dav using the htpasswd utility.
		+
		+	Now its possible to image directly there:
		+	<pre>
		+	aff2imager -i -o http://test:password@localhost/webdav/test.zip /dev/sda1
		+	</pre>

← Older revision		Revision as of 12:50, 25 April 2009
Line 227:		Line 227:

	Finally we see all the attributes of all the objects stored in the volumes specified. If any of these fact are also verified in an identity statement, we see a message indicating who signed this segment (The identity name is the certificate fingerprint).		Finally we see all the attributes of all the objects stored in the volumes specified. If any of these fact are also verified in an identity statement, we see a message indicating who signed this segment (The identity name is the certificate fingerprint).
		+
		+	== Fuse support ==
		+	Fuse is a library to create a filesystem in userspace. Its very easy to use and very powerful since it allows userspace programs to create abstracted filesystem like views into data. LibAFF4 has a fuse interface to allow access to the streams within a set of volumes. This feature is useful in order to allow other software to have access to the streams without needing specific AFF4 support.
		+
		+	<pre>
		+	$ affuse -d -- /tmp/test.zip /tmp/fuse
		+	zip.c:51 - FileBackedObject_Con: Can't open file:///tmp/test.zip/__URN__ (Not a directory)
		+	libaff4.c:27 - open_volume: Loaded zip volume file:///tmp/test.zip
		+	unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
		+	...
		+	</pre>
		+
		+	The trick here is that the aff4use program uses the <code>--</code> option to separate options specific to fuse (before the --) to options specific to libaff4 (after the --). The <code>-d</code> option requests fuse not fork into the background, but instead remain at the terminal and provide debugging messages. The <code>/tmp/fuse</code> directory is just somewhere to mount the fuse filesystem over.
		+
		+	Note that aff4fuse provides access to all objects with a stream interface. This will include all segments, but does not include identities or volumes. All relative URNs (like links) are placed at the root of the filesystem, and the full complicated URNs are placed in __URN__/ subdirectory (just in case we need to access a stream without a link).
		+	<pre>
		+	$ ls -l /tmp/fuse/
		+	total 0
		+	drwxr-xr-x 2 root root 1 2009-04-25 21:54 default
		+	-r--r--r-- 1 root root 254951424 1970-01-01 10:00 default.dd
		+	drwxr-xr-x 2 root root 1 2009-04-25 21:54 __URN__
		+
		+	$ fls /tmp/fuse/default.dd
		+	d/d 11: lost+found
		+	r/r 10043: System.map-2.6.22-14-generic
		+	r/r 6025: vmlinuz-2.6.22-14-generic
		+	r/r 10041: config-2.6.22-14-generic
		+	r/r 10042: abi-2.6.22-14-generic
		+	....
		+	</pre>
		+
		+	When you are finished and wish to unmount the filesystem:
		+	<pre>
		+	$ fusermount -u /tmp/fuse
		+	</pre>
		+
		+	== Creating map streams for filesystems ==
		+	Map streams are a great idea since they allow different mutations of the same data to be stored with zero cost. One application is to extract the block allocation table of a filesystem using sleuthkit and create maps for each file in the filesystem. This way applications dont know or care how to parse filesystems, since the underlying forensic format can present the data to them.
		+
		+	The map streams can be put into a new volume which simply refer to the old volume:
		+	<pre>
		+	$ fsbuilder -o /tmp/test2.zip /tmp/test.zip default
		+	zip.c:51 - FileBackedObject_Con: Can't open file:///tmp/test.zip/__URN__ (Not a directory)
		+	libaff4.c:27 - open_volume: Loaded zip volume file:///tmp/test.zip
		+	zip.c:51 - FileBackedObject_Con: Can't open file://default/__URN__ (No such file or directory)
		+	zip.c:51 - FileBackedObject_Con: Can't open file://default (No such file or directory)
		+	zip.c:51 - FileBackedObject_Con: Can't open file://default/__URN__ (No such file or directory)
		+	zip.c:51 - FileBackedObject_Con: Can't open file://default (No such file or directory)
		+	libaff4.c:100 - aff4_open: Trying to open stream default relative to all volumes
		+	libaff4.c:107 - aff4_open: Trying urn:aff4:89f8ec58-13f0-4ad4-b1f2-74da7f3aed40/default
		+	libaff4.c:122 - aff4_open: Using urn:aff4:fda4d4fa-c378-4a06-acac-06ff54f49d5f as the stream name
		+	File name urn:aff4:fafe0c11-0544-4eab-ae8a-3ee568f53f7c/System.map-2.6.22-14-generic
		+	File name urn:aff4:fafe0c11-0544-4eab-ae8a-3ee568f53f7c/vmlinuz-2.6.22-14-generic
		+	File name urn:aff4:fafe0c11-0544-4eab-ae8a-3ee568f53f7c/config-2.6.22-14-generic
		+	File name urn:aff4:fafe0c11-0544-4eab-ae8a-3ee568f53f7c/abi-2.6.22-14-generic
		+	....
		+	</pre>
		+
		+	As can be seen the fsbuilder tool creates new map streams for each new file found in the stream. The map streams are pushed to a new volume. Its possible now to fuse mount everything.

@@ Line 9: / Line 9: @@
 </code>
-Once the AFF4 directory is downloaded, a configure file needs to be created:
+Once the AFF4 directory is downloaded it can be build and installed as following:
 <code>
-sh autogen.sh
+./scon install
-./configure
-make install
 </code>