Difference between pages "Tcpflow" and "Gzip"

From Forensics Wiki
(Difference between pages)
Jump to: navigation, search
m
 
(See Also)
 
Line 1: Line 1:
{{Infobox_Software |
+
{{expand}}
  name = tcpflow |
+
  maintainer = Simson Garfinkel |
+
  os = {{Linux}} |
+
  genre = Network forensics |
+
  license = {{GPL}} |
+
  website = [http://afflib.org/software/tcpflow/] |
+
}}
+
  
'''tcpflow''' is a program that captures data transmitted as part of TCP connections (flows), and stores the data in a way that is convenient for protocol analysis and debugging. Each TCP flow is stored in its own file. Thus, the typical TCP flow will be stored in two files, one for each direction. tcpflow can also process stored ‘tcpdump’ packet flows.
+
== File format ==
 +
The gzip file (.gz) format consists of:
 +
* a file header
 +
* optional headers
 +
** extra fields
 +
** original file name
 +
** comment
 +
** header checksum
 +
* compressed data (commonly used compression method DEFLATE, without zlib header)
 +
* a file footer
  
tcpflow is similar to ‘tcpdump’, in that both process packets from the wire or from a stored file. But it’s different in that it reconstructs the actual data streams and stores each flow in a separate file for later analysis.
+
{| class="wikitable"
 +
! align="left"| Characteristics
 +
! Description
 +
|-
 +
| Byte order
 +
| little-endian
 +
|-
 +
| Date and time values
 +
| Filetime in UTC
 +
|-
 +
| Character string
 +
| ISO 8859-1 (LATIN-1)
 +
|}
  
tcpflow understands sequence numbers and will correctly reconstruct data streams regardless of retransmissions or out-of-order delivery.  
+
=== File header ===
 +
The file header is 10 bytes in size and contains:
 +
{| class="wikitable"
 +
! align="left"| Offset
 +
! Size
 +
! Value
 +
! Description
 +
|-
 +
| 0
 +
| 2
 +
| 0x1f 0x8b
 +
| Signature (or identification byte 1 and 2)
 +
|-
 +
| 2
 +
| 1
 +
|
 +
| Compression Method
 +
|-
 +
| 3
 +
| 1
 +
|
 +
| Flags
 +
|-
 +
| 4
 +
| 4
 +
|
 +
| Last modification time <br> Contains a POSIX timestamp.
 +
|-
 +
| 8
 +
| 1
 +
|
 +
| Compression flags (or extra flags)
 +
|-
 +
| 9
 +
| 1
 +
|
 +
| Operating system <br> Value that indicates on which operating system the gzip file was created.
 +
|}
  
Jeremy Elson developed the first version of tcpflow in 1999 but stopped maintaining it in 2003. In 2006 Simson Garfinkel took over maintenance of the program and added:
+
==== Compression method ====
  
* support for VLANs
+
{| class="wikitable"
* support for IPv6
+
! align="left"| Value
* [[DFXML]] output of the connections in a '''report.xml''' file.
+
! Identifier
* Improved performance through the use of the C++ STL classes.
+
! Description
* Support for continuous operation (tcpflow now purges out old flows).
+
|-
* Variable Filename specifications.
+
| 0 - 7
* A plug-in architecture.
+
|
 +
| Reserved
 +
|-
 +
| 8
 +
| deflate
 +
| deflate compressed data
 +
|}
  
tcpflow is based on the LBL Packet Capture Library (available from LBL) and therefore supports the same rich filtering expressions that programs like ‘tcpdump’ support. It should compile under most popular versions of UNIX; see the INSTALL file for details.
+
==== Flags ====
tcpflow stores all captured data in files that have names of the form:
+
    128.129.130.131.02345-010.011.012.013.45103[VLAN]
+
where the contents of the above file would be data transmitted from host 128.129.131.131 port 2345, to host 10.11.12.13 port 45103. VLAN information, if provided is stored in brackets.
+
  
== Overview ==
+
{| class="wikitable"
 +
! align="left"| Value
 +
! Identifier
 +
! Description
 +
|-
 +
| 0x01
 +
| FTEXT
 +
| If set the uncompressed data needs to be treated as text instead of binary data. <br> This flag hints end-of-line conversion for cross-platform text files but does not enforce it.
 +
|-
 +
| 0x02
 +
| FHCRC
 +
| The file contains a header checksum (CRC-16)
 +
|-
 +
| 0x04
 +
| FEXTRA
 +
| The file contains extra fields
 +
|-
 +
| 0x08
 +
| FNAME
 +
| The file contains an original file name string
 +
|-
 +
| 0x10
 +
| FCOMMENT
 +
| The file contains comment
 +
|-
 +
| 0x20
 +
|
 +
| Reserved
 +
|-
 +
| 0x40
 +
|
 +
| Reserved
 +
|-
 +
| 0x80
 +
|
 +
| Reserved
 +
|}
  
tcpflow stores all captured data in files that have names of the form
+
<b>Notes:</b>
: 128.129.130.131.02345-010.011.012.013.45103
+
* Reserved flags bits must be zero.
where the contents of the above file would be data transmitted from host ''128.129.131.131'' port ''2345'', to host ''10.11.12.13'' port ''45103''.
+
* The FHCRC bit was never set by versions of gzip up to 1.2.4, even though it was documented with a different meaning in gzip 1.2.4.
  
== Limitations ==
+
==== Compression flags ====
 +
This value contains flags specific to the compression method.
  
* tcpflow does not understand IP fragments;
+
===== Compression flags - deflate =====
* tcpflow does not understand 802.11 headers.
+
If compression method value is 8 (deflate) the following compression flags can be used:
 +
{| class="wikitable"
 +
! align="left"| Value
 +
! Identifier
 +
! Description
 +
|-
 +
| 0x02
 +
|
 +
| compressor used maximum compression, slowest algorithm
 +
|-
 +
| 0x04
 +
|
 +
| compressor used fastest algorithm
 +
|}
  
 +
==== Operating System ====
 +
{| class="wikitable"
 +
! align="left"| Value
 +
! Identifier
 +
! Description
 +
|-
 +
| 0
 +
|
 +
| FAT filesystem (MS-DOS, OS/2, NT/Win32)
 +
|-
 +
| 1
 +
|
 +
| Amiga
 +
|-
 +
| 2
 +
|
 +
| VMS (or OpenVMS)
 +
|-
 +
| 3
 +
|
 +
| Unix
 +
|-
 +
| 4
 +
|
 +
| VM/CMS
 +
|-
 +
| 5
 +
|
 +
| Atari TOS
 +
|-
 +
| 6
 +
|
 +
| HPFS filesystem (OS/2, NT)
 +
|-
 +
| 7
 +
|
 +
| Macintosh
 +
|-
 +
| 8
 +
|
 +
| Z-System
 +
|-
 +
| 9
 +
|
 +
| CP/M
 +
|-
 +
| 10
 +
|
 +
| TOPS-20
 +
|-
 +
| 11
 +
|
 +
| NTFS filesystem (NT)
 +
|-
 +
| 12
 +
|
 +
| QDOS
 +
|-
 +
| 13
 +
|
 +
| Acorn RISCOS
 +
|-
 +
| 255
 +
|
 +
| unknown
 +
|}
  
[[Category:Network Forensics]]
+
=== Optional headers ===
 +
==== Extra fields ====
 +
This value is present in the file if the FEXTRA flag is set in the file header flags.
 +
 
 +
The extra field are variable of size and contains:
 +
{| class="wikitable"
 +
! align="left"| Offset
 +
! Size
 +
! Value
 +
! Description
 +
|-
 +
| 0
 +
| 2
 +
|
 +
| Extra field data size <br> Value in bytes.
 +
|-
 +
| 2
 +
| ...
 +
|
 +
| Extra field data
 +
|}
 +
 
 +
==== Original file name ====
 +
This value is present in the file if the FNAME flag is set in the file header flags.
 +
 
 +
This is the original name of the file being compressed, with any directory components removed, and, if the file being compressed is on a file system with case insensitive names, forced to lower case.
 +
 
 +
Contains an ISO 8859-1 (LATIN-1) string with end-of-string character.
 +
 
 +
==== Comment ====
 +
This value is present in the file if the FCOMMENT flag is set in the file header flags.
 +
 
 +
Contains an ISO 8859-1 (LATIN-1) string with end-of-string character. Line breaks should be denoted by a single line feed character.
 +
 
 +
==== Header checksum ====
 +
The header checksum contain a CRC-16 that consists of the two least significant bytes of the CRC-32 for all bytes of the gzip header up to and not including the CRC-16.
 +
 
 +
=== File footer ===
 +
The file footer is 8 bytes in size and contains:
 +
{| class="wikitable"
 +
! align="left"| Offset
 +
! Size
 +
! Value
 +
! Description
 +
|-
 +
| 0
 +
| 4
 +
|
 +
| Checksum (CRC-32)
 +
|-
 +
| 4
 +
| 4
 +
|
 +
| Uncompressed data size <br> Value in bytes.
 +
|}
 +
 
 +
== See Also ==
 +
* [[bzip2]]
 +
* [[tar]]
 +
 
 +
== External Links ==
 +
 
 +
* [http://www.gzip.org/format.txt The gzip file format], by the [http://www.gzip.org/ gzip project]
 +
* [http://www.gzip.org/algorithm.txt The gzip compression algorithm], by the [http://www.gzip.org/ gzip project]
 +
* [http://tools.ietf.org/html/rfc1952 RFC1952: GZIP file format specification version 4.3], by [[IETF]]
 +
* [http://en.wikipedia.org/wiki/Gzip Wikipedia: gzip]
 +
 
 +
[[Category:File Formats]]

Revision as of 08:01, 30 November 2013

Information icon.png

Please help to improve this article by expanding it.
Further information might be found on the discussion page.

Contents

File format

The gzip file (.gz) format consists of:

  • a file header
  • optional headers
    • extra fields
    • original file name
    • comment
    • header checksum
  • compressed data (commonly used compression method DEFLATE, without zlib header)
  • a file footer
Characteristics Description
Byte order little-endian
Date and time values Filetime in UTC
Character string ISO 8859-1 (LATIN-1)

File header

The file header is 10 bytes in size and contains:

Offset Size Value Description
0 2 0x1f 0x8b Signature (or identification byte 1 and 2)
2 1 Compression Method
3 1 Flags
4 4 Last modification time
Contains a POSIX timestamp.
8 1 Compression flags (or extra flags)
9 1 Operating system
Value that indicates on which operating system the gzip file was created.

Compression method

Value Identifier Description
0 - 7 Reserved
8 deflate deflate compressed data

Flags

Value Identifier Description
0x01 FTEXT If set the uncompressed data needs to be treated as text instead of binary data.
This flag hints end-of-line conversion for cross-platform text files but does not enforce it.
0x02 FHCRC The file contains a header checksum (CRC-16)
0x04 FEXTRA The file contains extra fields
0x08 FNAME The file contains an original file name string
0x10 FCOMMENT The file contains comment
0x20 Reserved
0x40 Reserved
0x80 Reserved

Notes:

  • Reserved flags bits must be zero.
  • The FHCRC bit was never set by versions of gzip up to 1.2.4, even though it was documented with a different meaning in gzip 1.2.4.

Compression flags

This value contains flags specific to the compression method.

Compression flags - deflate

If compression method value is 8 (deflate) the following compression flags can be used:

Value Identifier Description
0x02 compressor used maximum compression, slowest algorithm
0x04 compressor used fastest algorithm

Operating System

Value Identifier Description
0 FAT filesystem (MS-DOS, OS/2, NT/Win32)
1 Amiga
2 VMS (or OpenVMS)
3 Unix
4 VM/CMS
5 Atari TOS
6 HPFS filesystem (OS/2, NT)
7 Macintosh
8 Z-System
9 CP/M
10 TOPS-20
11 NTFS filesystem (NT)
12 QDOS
13 Acorn RISCOS
255 unknown

Optional headers

Extra fields

This value is present in the file if the FEXTRA flag is set in the file header flags.

The extra field are variable of size and contains:

Offset Size Value Description
0 2 Extra field data size
Value in bytes.
2 ... Extra field data

Original file name

This value is present in the file if the FNAME flag is set in the file header flags.

This is the original name of the file being compressed, with any directory components removed, and, if the file being compressed is on a file system with case insensitive names, forced to lower case.

Contains an ISO 8859-1 (LATIN-1) string with end-of-string character.

Comment

This value is present in the file if the FCOMMENT flag is set in the file header flags.

Contains an ISO 8859-1 (LATIN-1) string with end-of-string character. Line breaks should be denoted by a single line feed character.

Header checksum

The header checksum contain a CRC-16 that consists of the two least significant bytes of the CRC-32 for all bytes of the gzip header up to and not including the CRC-16.

File footer

The file footer is 8 bytes in size and contains:

Offset Size Value Description
0 4 Checksum (CRC-32)
4 4 Uncompressed data size
Value in bytes.

See Also

External Links