Cyber Security Expo
 
NMAP Grepable Output by MadHat on 24/12/03

One of the often overlooked and underused output methods of nmap is the grepable or "machine" output. This output places all results for a single host on a single line, making it easier to use with other command line tools, like grep and awk. It also makes it easier to use when scripting.

One problem with this format is that it is not well documented and therefore not well understood. I hope to address this issue.

The man pages state:

-oG
This logs the results of your scans in a grepable form into the file you specify as an argument. This simple format provides all the information on one line (so you can easily grep for port or OS information and see all the IPs. This used to be the pre- ferred mechanism for programs to interact with Nmap, but now we recommend XML output (-oX instead). This simple format may not contain as much information as the other formats. You can give the argument "-" (without quotes) to shoot output into stdout (for shell pipelines, etc). In this case normal output will be suppressed. Watch out for error messages if you use this (they will still go to stderr). Also note that "-v" will cause some extra information to be printed.

Unfortunately, the documentation stops there.

Most of the information returned by the normal output or XML output is included with the grepable output, and as of version 3.49 of nmap, Version scanning is included in the grepable output.

The first example is a basic scan using all the defaults, except output.

$ nmap -oG - insecure.org
# nmap 3.48 scan initiated Thu Dec 11 10:51:45 2003 as: nmap -oG - insecure.org
Host: 205.217.153.53 () Ports: 22/open/tcp//ssh///, 25/open/tcp//smtp///, 53/open/tcp//domain///, 80/open/tcp//http///, 113/closed/tcp//auth/// Ignored State: filtered (1652)
# Nmap run completed at Thu Dec 11 10:52:51 2003 -- 1 IP address (1 host up) scanned in 66.544 seconds

The command line option used here was the -oG for grepable output, but I sent it to STDOUT by using the - option, rather than to a file.

I stated that there was one line per host, but on a simple run such as this, there are also 2 comment style lines. The first one notes what command was used to launch nmap, and the version, date and time started. The second one is at the end and tells the date and time that the scan finished, how long the scan took, and the number of hosts up. This reflects the basic information included in the standard output as well.

Looking at the line that contains the data we want to examine, we have:

Host: 205.217.153.53 () Ports: 22/open/tcp//ssh///, 25/open/tcp//smtp///, 53/open/tcp//domain///, 80/open/tcp//http///, 113/closed/tcp//auth/// Ignored State: filtered (1652)

We have Section Defining Tags, in this case Host, Ports, and Ignored State. These sections are separated with tabs (\t). We can use a split function, such as the one in Perl, to be able to work with each one of these fields. An example in Perl might look like:

@fields = split("\t", $nmap_output);

We now have three items in an array. These items would be:

Host: 205.217.153.53 () Ports: 22/open/tcp//ssh///, 25/open/tcp//smtp///, 53/open/tcp//domain///, 80/open/tcp//http///, 113/closed/tcp//auth/// Ignored State: filtered (1652)

Using this same concept, we can split on the colon (:) to generate a key-value pair consisting of the field name and value assigned to that field.

The Host entry only has 2 fields, space separated. This is the IP and the DNS entry for that IP address. In this case there is no reverse entry for DNS.

The second field in a basic scan is Ports. Ports is a comma separated list of ports. Each entry within this comma separated list is divided into 7 sections, but not all are used for a basic scan. These sections are separated by a forward slash (/). The fields are:

port number / state / protocol / owner / service / rpc info / version info

- port number:
The port number this entry is related to

- state:
open, closed or filtered. Depending on the scan type, the exact meaning and determination can change slightly for each of these, but the basic idea is that open means that the target machine will accept connetions on that port. closed means that as far as nmap can tell there is nothing blocking the port, but the device is not accepting connections. filtered means that some method was used to keep nmap from being able to determie whether or not the port is open. This is often done with firewalls or network filters.

An exaple of of these, looking at a SYN scan, would be that for open to be the state a SYN/ACK would be returned for our SYN sent. closed would be reported by nmap when a RST (reset) packet would be returned and filtered would show up as the state when either nothing was returned, and and we assume that a firewall dropped the request, or an ICMP port unreachable message was returned.

More details can be found in the man pages or at http:///www.insecure.org/nmap/

- protocol:
The protocol associated with the port, usually tcp or udp.

- owner:
This field will have the user running an app if a few requirements are fulfilled. First, you must specify to include a reverse ident scan (-I) and identd must be running on the remote host and be available. Keep in mind that identd can be set to return any info the administrator wants, so don't trust the data.

- service:
By default the service is going to be the name related to the port via the nmap-services file. If version scanning is used (-sV), this field will be populated with the service detected by the probe. In some cases the entry may be compound such as "ssl|http" or you may see a common service name with a trailing question mark.

Details on the Nmap version detection are available at http://www.insecure.org/nmap/versionscan.html

- rpc info:
RPC info is similar to the owner field in that it is only filled in with two requirements are met. Once again the scan has to be enabled via the RCP scan (-sR) that will gain the info the same way 'rpcinfo -p' does. This also means that the rpcbind, portmapper has to be running on the host being scanned.

- version info:
This field will also be empty unless the specific scan type is requested. Including the version scan (-sV) will fill this field if nmap is able to determine what is running on the port. This is also detailed at http://www.insecure.org/nmap/versionscan.html. We will look at this in more detail below.

Using the same concept as above we can show that with split we could access these fields easily.

($port, $state, $protocol, $owner, $service, $rpc_info, $version) = split('/', $ports);

The last field in the basic scan is the "Ignored State" which tells which state was ignored and how many entries (ports) were associated with that state. In this example there are 1652 ports filtered that were scanned.

An example of how to use the grepable output on the command line is to pipe the output to something like awk to find all hosts that have a particular port open. For example, if we want to know all the hosts with port 80 open on a subnet but all we want to see if the IP, we can use something like this:

$ nmap -p80 -PS80 -oG - 10.1.1.0/24 | awk '/open/{print $2}'
10.1.1.72
10.1.1.73
10.1.1.75
10.1.1.149
10.1.1.152
10.1.1.160
10.1.1.161
10.1.1.201
10.1.1.254

The -p80 states to scan for port 80. -PS80 means to use a SYN Ping method to see if the host is up. This will speed up the scan greatly because it will not scan twice, as it normally would, with the first being to see if the host is up and then scanning to see if the port is open. Nmap is smart enough to know that it has already tested that port with the SYN Ping. We also send the grepable output to STDOUT with -oG - and then pipe that to awk. The awk command is running this simple script, where /open/ matches on the test open (same as going a 'grep open') and the prints the second field breaking the output on spaces. If we wanted to include the domain name, we could use this:

$ nmap -p80 -PS80 -oG - 10.1.1.0/24 | awk '/open/{print $2 " " $3}'
10.1.1.72 (userA.corp.foocompany.biz)
10.1.1.73 (userB.corp.foocompany.biz)
10.1.1.75 (userC.corp.foocompany.biz)
10.1.1.149 (admin.corp.foocompany.biz)
10.1.1.152 (printer.corp.foocompany.biz)
10.1.1.160 (10-1-1-160.foocompany.biz)
10.1.1.161 (10-1-1-161.foocompany.biz)
10.1.1.201 (10-1-1-201.foocompany.biz)
10.1.1.254 (10-1-1-254.foocompany.biz)

This is where we grab the second and third field from each line.

--------------------

Now if we look at some other examples and other scanning options we can see the basic formatting does not change, and makes it fairly easy to parse with scripts or other applications.

If we turn on verbose logging we have more "comment" lines added.

$ sudo nmap -v -oG - 10.1.1.100
# nmap 3.48BETA1 scan initiated Thu Dec 11 15:03:01 2003 as: nmap -v -oG - 10.1.1.100
# Ports scanned: TCP(1657;1-1027,1029-1033,...,61439-61441,65301) UDP(0;) PROTOCOLS(0;)
Host: 10.1.1.100 (devbox.corp.foocompany.biz) Ports: 80/open/tcp//http///, 135/open/tcp//msrpc///, 139/open/tcp//netbios-ssn///, 443/open/tcp//https///, 445/open/tcp//microsoft-ds///, 1025/open/tcp//NFS-or-IIS///, 2105/open/tcp//eklogin///, 3389/open/tcp//ms-term-serv/// Ignored State: closed (1638)
# Nmap run completed at Thu Dec 11 15:03:09 2003 -- 1 IP address (1 host up) scanned in 7.668 seconds

We can see that we now have a line (that I shortened) that has information about what ports were scanned. Specifically it tell us that 1657 TCP ports were scanned, and then a semicolon (;), then a list of the ports. It also tells us that we did not scan any UDP ports and no extra protocols.

At this time adding more verbose tags on the command line (-vvv) does not further change the grepable output.

If we add the OS Detection to the scan, we see results like this:

$ sudo nmap -O -oG - 10.1.1.100
# nmap 3.48BETA1 scan initiated Thu Dec 11 15:15:00 2003 as: nmap -O -oG - 10.1.1.100
Host: 10.1.1.100 (devbox.corp.foocorp.biz) Ports: 80/open/tcp//http///, 135/open/tcp//msrpc///, 139/open/tcp//netbios-ssn///, 443/open/tcp//https///, 445/open/tcp//microsoft-ds///, 1025/open/tcp//NFS-or-IIS///, 2105/open/tcp//eklogin///, 3389/open/tcp//ms-term-serv/// Ignored State: closed (1638) OS: Microsoft Windows Millennium Edition (Me), Windows 2000 Professional or Advanced Server, or Windows XP|Microsoft Windows XP SP1 Seq Index: 22972 IPID Seq: Incremental
# Nmap run completed at Thu Dec 11 15:15:48 2003 -- 1 IP address (1 host up) scanned in 47.921 seconds

Once again we have our main sections of the output tab delimited (\t). Adding the OS Fingerprinting added a few new fields. Specifically we see:

OS: Microsoft Windows Millennium Edition (Me), Windows 2000 Professional or Advanced Server, or Windows XP|Microsoft Windows XP SP1
Seq Index: 22972
IPID Seq: Incremental

Each of these fields is in the same key/value format as before, being separated by a colon (:). As of this writing nmap grepable output does not support the "Device Type" or "Running" fields as returned from the standard or XML output.

The IPID and Seq Index only show up on the standard output with verbose (-v) turned on with OS detection.

The OS field is equivalent to the "OS details" from the standard and XML output which gives you the best guesses for the OS from the fingerprint when possible.

Specifically the IPID Seq field is the IP Sequence Generation field from standard output which will tell how the IPIDs are generated by the host. Examples are Incremental, Randomized, Random positive increments, etc. This value will help determine how vulnerable the host might be to certain types of attacks and information gathering tactics.

The Seq Index field is the difficulty rating to predict the next TCP sequence number. In the normal output we have some key terms to help read this number, but the basic idea is that the higher the number the harder it is to guess the next TCP sequence number in the TCP header.

  
  Seq < 10          - Trivial Joke
   11 < Seq < 80    - Easy
   81 < Seq < 3000  - Medium Difficulty
 3001 < Seq < 5000  - Formidable
 5001 < Seq < 10000 - Worthy challenge
10001 < Seq         - 'Good Luck!'

-------------------

Taking a step back and looking at simple scans, such as a ping scan, we can see that the basic format is the same, still tab (\t) delimited entries:

$ sudo nmap -oG - -sP 10.1.1.172/29
# nmap 3.48BETA1 scan initiated Thu Dec 11 15:49:17 2003 as: nmap -oG - -sP 10.1.1.168/29
Host: 10.1.1.168 (alice.corp.foocorp.biz) Status: Up
Host: 10.1.1.169 (madhat-sun.corp.foocorp.biz) Status: Up
Host: 10.1.1.170 (madhat.corp.foocorp.biz) Status: Up
Host: 10.1.1.171 (madhat-laptop.corp.foocorp.biz) Status: Up
Host: 10.1.1.172 (iss-scanner.dal.foocorp.biz) Status: Up
Host: 10.1.1.173 (hatta.corp.foocorp.biz) Status: Up
# Nmap run completed at Thu Dec 11 15:49:19 2003 -- 8 IP addresses (6 hosts up) scanned in 1.242 seconds

Once again we can split on the tab (\t) to get the 2 fields returned here. Each one returned is in the key/value pair format, using the colon (:) delimiter.

Adding a -v to this scan will show both up and down hosts, with only the Status field changing. The same output is used for the List scan (-sL) that only does a lookup on the IPs. The only difference is that it does not try to contact the hosts at all and only does a DNS lookup for each IP and the Status is reported as Unknown.

An example can be found here:

$ sudo nmap -oG - -sL 10.1.1.172/29
# nmap 3.48BETA1 scan initiated Thu Dec 11 15:49:17 2003 as: nmap -oG - -sP 10.1.1.168/29
Host: 10.1.1.168 (alice.corp.foocorp.biz) Status: Unknown
Host: 10.1.1.169 (madhat-sun.corp.foocorp.biz) Status: Unknown
Host: 10.1.1.170 (madhat.corp.foocorp.biz) Status: Unknown
Host: 10.1.1.171 (madhat-laptop.corp.foocorp.biz) Status: Unknown
Host: 10.1.1.172 (iss-scanner.dal.foocorp.biz) Status: Unknown
Host: 10.1.1.173 (hatta.corp.foocorp.biz) Status: Unknown
Host: 10.1.1.174 () Status: Unknown
Host: 10.1.1.175 () Status: Unknown
# Nmap run completed at Thu Dec 11 15:49:19 2003 -- 8 IP addresses (6 hosts up) scanned in 1.242 seconds

--------------------

Looking at the RPC scan (-sR) we can fill in another one of the fields available in each of the port indicators. Here is an example of using the RPC scan with RPC running on the box:

$ sudo nmap -oG - -p21-25,80,111,443,4045,32774 -sR -T4 box.foocorp.biz
# nmap 3.48BETA1 scan initiated Tue Dec 16 16:37:18 2003 as: nmap -oG - -p21-25,80,111,443,4045,32774 -sR -T4 box.foocorp.biz
Host: 10.1.1.146 (box.foocorp.biz) Ports: 21/open/tcp//ftp/N//, 22/open/tcp//ssh/N//, 23/closed/tcp//telnet///, 24/closed/tcp//priv-mail///, 25/open/tcp//smtp/N//, 80/closed/tcp//http///, 111/open/tcp//rpcbind/(rpcbind:100000*2-4)/2-4 (rpc #100000)/, 443/closed/tcp//https///, 4045/open/tcp//nlockmgr/(nlockmgr:100021*1-4)/1-4 (rpc #100021)/, 32774/open/tcp//sometimes-rpc11/N//
# Nmap run completed at Tue Dec 16 16:37:18 2003 -- 1 IP address (1 host up) scanned in 0.490 seconds

For this example we are only looking at the ports section of the output:

Ports: 21/open/tcp//ftp/N//, 22/open/tcp//ssh/N//, 23/closed/tcp//telnet///, 24/closed/tcp//priv-mail///, 25/open/tcp//smtp/N//, 80/closed/tcp//http///, 111/open/tcp//rpcbind/(rpcbind:100000*2-4)/2-4 (rpc #100000)/, 443/closed/tcp//https///, 4045/open/tcp//nlockmgr/(nlockmgr:100021*1-4)/1-4 (rpc #100021)/, 32774/open/tcp//sometimes-rpc11/N//

Each port's information is separated by a comma (,), and each field within the port's data is separated by a forward slash (/). Looking at the fields again, we have port number, state, protocol, and user, which will be discussed later, and RPC information and version information.

The RPC information is filled in by running queries on the remote host equivalent to rpcinfo -p . In this case we can see that many of ports that are open are reporting 'N' as its RPC information, which means that it is not an RPC process listening. Looking at the example of port 111, rpcbind:

111/open/tcp//rpcbind/(rpcbind:100000*2-4)/2-4 (rpc #100000)/

In this example we have TCP port 111 open. TCP port 111 is labeled as rpcbind in the nmap-services file and is reported as such here. The sixth field denotes the name, in this case again rpcbind, the RPC program number, the low version number of the RPC program found on that port and the high version of the RPC program found on that port. The data returned is always returned with parenthesis and in the format:

: * -

The last field is the version information field and prints the same information, but in a different format. It only includes the low and high versions found and the RPC program number. The format is:

- (rpc #)

Looking at rpcinfo output from the same host:

$ rpcinfo -p box.foocorp.biz
   program vers proto   port
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper

We can combine all this information to match what nmap has reported, minus the name mapping in the last field. We have the program number 100000. Lowest version shown is 2 and the highest is 4.

--------------------

Another option in nmap is to do a protocol scan, to see what protocols are supported on a host. This scan can not be combined with other scan types at this time.

Here is our example of this scan type:

$ sudo nmap -oG - -sO madhat-laptop.corp
# nmap 3.47 scan initiated Wed Dec 17 10:35:00 2003 as: nmap -oG - -sO madhat-laptop.corp
Host: 10.1.1.171 (madhat-laptop.corp.foocorp.biz) Protocols: 1/open/icmp/, 2/open/igmp/, 6/open/tcp/, 17/open/udp/, 255/open// Ignored State: closed (251)
# Nmap run completed at Wed Dec 17 10:35:04 2003 -- 1 IP address (1 host up) scanned in 3.750 seconds

Looking at the output we see it is very similar to other scan type outputs, with the sections still tab (\t) separate, except that here instead of a "Ports" section, we have a "Protocols" section. Within this section the format is similar to that of the Ports, but shorter. We have 3 fields, protocol ID, state and name. In our example protocol 1 is open and known as ICMP. The names are started in /etc/protocols or nmap-protocols. There are 256 protocols, so we can see that we had 251 in the Ignored State of closed and 5 open.

--------------------

Reverse ident scanning (-I), using the ident protocol, only works with TCP connect scans (-sT). You will actually get an error that it is being ignored when writing to a file or using standard output. The fourth field in the ports was mentioned earlier as being the "owner" as noted by an Ident scan. Here is an example:

$ nmap -oG - -T4 -sT -I madhat.corp
# nmap 3.47 scan initiated Wed Dec 17 11:02:59 2003 as: nmap -oG - -T4 -sT -I madhat.corp
Host: 10.1.1.170 (madhat.corp.foocorp.biz) Ports: 22/open/tcp/root/ssh///, 111/open/tcp/rpc/rpcbind///, 113/open/tcp/ident/auth///, 3306/open/tcp/mysql/mysql///, 6000/open/tcp/root/X11/// Ignored State: closed (1652)
# Nmap run completed at Wed Dec 17 11:03:00 2003 -- 1 IP address (1 host up) scanned in 0.701 seconds

Also mentioned before is the fact we can not truly trust this information, as many people do not run an identd server anymore or use it to report specific user names for IRC or other apps. The format has not changed; a new field was filled in. In this example we can see that the app that is listening on 22 is running as root, the app on 3306 is running as mysql and the app running on 113 is running as ident.

--------------------

As of version 3.49 of nmap Version Scanning is included in the grepable output. If we look at an example:

$ nmap -oG - -T4 -sTV madhat.corp
# nmap 3.48BETA1 scan initiated Wed Dec 17 11:43:09 2003 as: nmap -oG - -T4 -sTV madhat.corp
Host: 10.1.1.170 (madhat.corp.foocorp.biz) Ports: 22/open/tcp//ssh//OpenSSH 3.1p1 (protocol 1.99)/, 80/open/tcp//http//Apache httpd 1.3.27 ((Unix) (Red-Hat|Linux) mod_ssl|2.8.12 OpenSSL|0.9.6 PHP|4.1.2 mod_perl|1.24_01)/, 111/open/tcp//rpcbind//2 (rpc #100000)/, 113/open/tcp//ident//OpenBSD identd/, 443/open/tcp//ssl|http//Apache httpd 1.3.27 ((Unix) (Red-Hat|Linux) mod_ssl|2.8.12 OpenSSL|0.9.6 PHP|4.1.2 mod_perl|1.24_01)/, 3306/open/tcp//mysql//MySQL 3.23.55/, 6000/open/tcp//X11//(access denied)/ Ignored State: closed (1650)
# Nmap run completed at Wed Dec 17 11:43:19 2003 -- 1 IP address (1 host up) scanned in 10.689 seconds

Once again the output is basically the same as we have seen before, but we add the last, seventh, field in the Ports section. This field, as noted before, contains the version information. Details on how the version of an app is determined can be found at: http://www.insecure.org/nmap/versionscan.html

We first listen for a response. Based on what we receive or don't receive, we send predefined queries to elicit a response that will help us determine what is running on that port.

If we look at this example closer we can see that we have port 22 open, and it is running OpenSSH 3.1p1 (protocol 1.99). Looking at port 80, we have Apache httpd 1.3.27 ((Unix) (Red-Hat|Linux) mod_ssl|2.8.12 OpenSSL|0.9.6 PHP|4.1.2 mod_perl|1.24_01), which points out an important change in the grepable output from the standard or XML output. Because the grepable output uses the forward slash (/) as a delimiter the forward slash in the version information had to be escaped or changed. The final decision was to change it to a pipe (|) for ease of scripting or piping to another application. In the same way that standard output puts ssl/ in front of the service field when SSL is detected by a version scan, the grepable output also denotes SSLified ports but again uses the | as the delimiter, so as to not interfere with the existing formatting.

443/open/tcp//ssl|http//Apache httpd 1.3.27 ((Unix) (Red-Hat|Linux) mod_ssl|2.8.12 OpenSSL|0.9.6 PHP|4.1.2 mod_perl|1.24_01)/,

In our example from above we can see an example of both of these substitutions in the HTTPS port 443. The service name is ssl|http, denoting that the HTTP protocol is being used behind SSL. When the version information is returned in the format, everything outside of the parenthesis is the primary information consisting of the server and the version, when possible, and the data within the parenthesis is considered extra information gathered from the host. In the information in this example, we can see that the extra information returned by nmap about the version includes RedHat|Linux, meaning that what was return or normally displayed would be RedHat/Linux.

Once again more details on Version scanning with nmap can be found at: http://www.insecure.org/nmap/versionscan.html

Keep in mind that the default for -oG grepable output is to specify a file to write to, and when doing this, the standard output will be shown on STDOUT and the data show above will be written to the file.

Some examples on how to use the grepable output can be found at: http://www.unspecific.com/nmap/

Rate this article

All images, content & text (unless other ownership applies) are © copyrighted 2000 -  , Infosecwriters.com. All rights reserved. Comments are property of the respective posters.