Skip to content

nmap: a practical introduction

Published by Marius on 2019-04-20

Last updated on 2019-05-03

Contents

Good habits

Specify the source interface

Every time you use nmap, just get used to typing nmap -e followed by the local network interface from where all packets will be sent; e.g., nmap -e eth1.

Don’t be lazy – being explicit about the source interface every time will help you avoid confusion when working in machines with multiple interfaces connected to different networks, or in more complex scenarios involving SSH tunnels and such.

Consider setting variables for interface names when they’re longer and slightly annoying to type (as is the case with systemd’s predictable network interface names); e.g., i1="enp2s0", so then you may issue nmap -e $i1.

To view a list of existing interfaces, use ip link show, which may be shortened to ip l sh (apologies for triggering those CCNA flashbacks).

No DNS resolution

Remember how I said you should get used to typing nmap -e? Forget that. The real 1337 $h!7 is nmap -ne.

The -n switch tells nmap to never perform DNS resolution, so if you type a command wrongly, you won’t have to worry about nmap trying to figure out if herpderp is a real host, at which point you’d have to furiously hit Ctrl+C.

While you could use nmap -e $i1 -n… why? nmap -ne $i1, practice that.

On the flip side, if you want nmap to always resolve hostnames, use -R.

Output to file

Saving results to files is always a good idea, allowing you to visualize your progress, compare scans, use results in scripts and – most importantly – stop scrolling up on the terminal.

The most flexible option is -oX <file>, which outputs XML results to the specified file. Best when feeding the output into scripts.

A more readable option is -oN <file>, which simply redirects nmap’s normal output to the specified file.

Manual mode

nmap has an extremely powerful manual mode, achieved by the godlike command man nmap. You’ll need it. We all do.

Host discovery

For starters, let’s sweep an IP range for active hosts. For that, use the -sn switch – sweep network may be a useful mnemonic.

The default host discovery done with -sn consists of an ICMP echo request, TCP SYN to port 443, TCP ACK to port 80, and an ICMP timestamp request by default. When executed by an unprivileged user, only SYN packets are sent (using a connect call) to ports 80 and 443 on the target. When a privileged user tries to scan targets on a local ethernet network, ARP requests are used unless –send-ip was specified. The -sn option can be combined with any of the discovery probe types (the -P* options, excluding -Pn) for greater flexibility. If any of those probe type and port number options are used, the default probes are overridden. When strict firewalls are in place between the source host running Nmap and the target network, using those advanced techniques is recommended. Otherwise hosts could be missed when the firewall drops probes or their responses.

Source: man nmap

Add a network range, and we’ve got:

Remember that $i1 is just a variable for your chosen source interface, and you can use -oN for normal output to file instead of -oX for XML.

As the manual states, using the -P* options may help discover hosts when -sn alone won’t do. While those options are interesting and worth learning, they often aren’t needed when targeting VMs in a practice environment, for example.

Lastly, remember that using -sn as a privileged user on the local network will trigger ARP requests (and nothing else), which are wonderful for host discovery.

Port scanning

Having acquired a list of active targets, it’s time for a port scan. We have a few options here, the most common ones being -sT and -sS.

The -sT option executes a full TCP handshake against each chosen port (send TCP, why not).

On the other hand, the -sS option is faster and probably just as effective, sending only SYN packets without performing the full handshake. However, it does require privileged execution.

A handy way to specify target ports is the --top-ports option, which uses the /usr/share/nmap/nmap-services file to select the most popular ports.

You can perform host discovery and port scanning with a single command:

Note that there’s no -sn in there, as that option only performs host discovery, and cannot be used in conjunction with port scanning.

If you’d like to run only a port scan, without host discovery, -Pn (Ping none?) is for you:

Scripting

When all that 1337 $h!7 becomes too much to remember, you can of course use bash scripting to build a scanning platform exactly to your specifications, as I’ve done with nmap-probe.sh.

When I need something more powerful than bash alone, from parsing nmap’s XML output to conditionally kicking off different scans, my go-to tool is python-nmap.

Below, you’ll find nmap-recon.py as an example. While each of these scripts may be used alone, nmap-probe.sh has a flag to trigger nmap-recon.py automatically.

If you’re going to play around with these scripts, note that they require a specific directory structure:
.
├── output
└── scripts
   ├── nmap-probe.sh
   └── nmap-recon.py

Appendix

nmap-probe.sh

Usage: ./nmap-probe.sh <src_iface> <host|host_list> [X] [Y] [--top-N] [--recon] [--queue]
Scans the top N ports of IPv4 <host> or of each host listed in <host_list>.
<host_list> should be a file containing one full IPv4 address per line.
Each host is scanned in parallel as a background process.
If [X] is specified, limits the scan to the Xth host in <host_list>.
If [X] and [Y] are specified, limits the scan to the Xth to Yth hosts in <host_list>.
If --top-N is specified, scans the top N ports instead of the default 100. E.g., --top-300.
If --recon is specified, invokes nmap-recon.py for each target host.
If --queue is specified, nmap-recon.py will run serialized, rather than paralleled.
Without --recon, the --queue option will be ignored.
#! /bin/bash

# Define a cleanup function to be executed any time the script is terminated.
function on_exit {
    [[ -f ${queue} ]] && rm ${queue}
}

# When EXIT is received, call the on_exit function.
trap on_exit EXIT

RECON="--recon"
QUEUE="--queue"
TOP_N="--top-"

# If the number of arguments received is less than 2...
if [[ $# -lt 2 ]]
then
    echo "Usage: ${0} <src_iface> <host|host_list> [X] [Y] [--top-N] [${RECON}] [${QUEUE}]"
    echo "Scans the top N ports of IPv4 <host> or of each host listed in <host_list>."
    echo "<host_list> should be a file containing one full IPv4 address per line."
    echo "Each host is scanned in parallel as a background process."
    echo "If [X] is specified, limits the scan to the Xth host in <host_list>."
    echo "If [X] and [Y] are specified, limits the scan to the Xth to Yth hosts in <host_list>."
    echo "If --top-N is specified, scans the top N ports instead of the default 100. E.g., --top-300."
    echo "If --recon is specified, invokes nmap_recon.py for each target host."
    echo "If --queue is specified, nmap_recon.py will run serialized, rather than paralleled."
    echo "Without --recon, the --queue option will be ignored."
    exit 1
fi

iface="${1}"
# Check whether the specified interface exists.
[[ ! $(ip link show ${iface}) ]] && exit 1

host_arg="${2}"

range=""
top_ports="100"
# While the current optional argument is not empty...
while [[ -n ${3} ]]
do
    # If ${range} is not empty and current optional argument is numeric;
    # that is, if [X] and [Y] were both specified, append a comma and the argument to ${range}.
    [[ -n ${range} && ${3} =~ ^[0-9]+$ ]] && range="${range},${3}"
    # If ${range} is empty and the current optional argument is numeric;
    # that is, if [X] was specified, assign the argument to ${range}.
    [[ -z ${range} && ${3} =~ ^[0-9]+$ ]] && range="${3}"
    # If the current optional argument starts with "--top-", grab the specified number.
    [[ ${3} =~ ^${TOP_N}([0-9]{1,5})$ ]] && top_ports=${BASH_REMATCH[1]}
    # If the current optional argument is "--recon", set ${recon}.
    [[ ${3} == ${RECON} ]] && recon=true
    # If the current optional argument is "--queue", create a temporary file.
    # This file will hold commands to be executed in series later on.
    [[ ${3} == ${QUEUE} ]] && queue=$(mktemp -q)
    # Shift arguments left so that ${4} becomes the new ${3}.
    shift
done

# If ${host_arg} is a file...
if [[ -f ${host_arg} ]]
then
    # Read the file's lines while using "sed" to acquire only the desired lines,
    # which were optionally given via [X] and [Y] and are held in ${range}.
    # The outer parentheses make ${hosts} into an array.
    hosts=($(sed -n ${range}p ${host_arg}))
else
    hosts=(${host_arg})
fi

# Grab the directory where this script resides.
script_dir=$(dirname $(realpath ${0}))
# Erase all characters from the beginning of ${0} up to, and including, the last "/" found,
# so that we end up only with the actual file name of this script, rather than its full path.
script_name=${0##*/}

# ${hosts[@]} returns each item in the array individually quoted.
for host in ${hosts[@]}
do
    # ${script_name%.*} erases all characters from the end of ${script_name}
    # back to, and including, the first "." found, effectively removing the extension.
    output_file="${script_dir}/../output/${host}_${script_name%.*}.xml"

    nmap_cmd="nmap -e ${iface} -n -Pn --open -sT --top-ports ${top_ports} ${host} -oX ${output_file}"
    python_cmd="python3 ${script_dir}/nmap_recon.py ${iface} ${output_file}"

    if [[ ${recon} == true ]]
    then
        # If the queue file exists...
        if [[ -f ${queue} ]]
        then
            # Issue ${nmap_cmd} and add ${python_cmd} to the queue.
            # The trailing "&" spawns a background process for this command,
            # so the loop will continue without waiting for nmap/python to return.
            ${nmap_cmd} > /dev/null && echo ${python_cmd} >> ${queue} &
        else
            # Issue ${nmap_cmd} and call ${python_cmd} afterwards.
            ${nmap_cmd} > /dev/null && ${python_cmd} &
            # ${nmap_cmd} && ${python_cmd}
        fi
    else
        # If --recon was not specified, just run ${nmap_cmd} in the background.
        ${nmap_cmd} > /dev/null &
    fi
done

# If there's no queue file, we're done.
[[ ! -f ${queue} ]] && exit 0

# The following code forms a queue so only one instance of nmap_recon.py is active at any given time.

# ${re_hosts} will contain all hosts, separated by a space.
re_hosts=${hosts[@]}
# Each space in ${re_hosts} will be replaced with a "|".
re_hosts=${re_hosts// /|}

# While an nmap process exists for any chosen host, we need to keep working on ${queue}.
while [[ -n $(pgrep -af "top-ports ${top_ports} (${re_hosts})") ]]
do
    sleep 5
    # ${cmd} will be the current line being read from ${queue}.
    while read -r cmd
    do
        # Wait until the current nmap_recon.py process is finished.
        while [[ -n $(pgrep -af "nmap_recon.py.+(${re_hosts})") ]]; do sleep 5; done
        ${cmd}
    done < ${queue}
done

nmap-recon.py

Usage: nmap_recon.py <source interface> <nmap XML file>
Uses the given nmap XML file to carry out additional nmap scans from the specified interface.

Sample output:

Script:

#! /usr/bin/python3

from collections  import defaultdict as ddict, namedtuple
from nmap         import PortScanner as ps
from os           import path
from subprocess   import check_output, CalledProcessError
from sys          import argv, exit


PortData   = namedtuple('PortData'  , 'number,type,name,product,version')
ScriptData = namedtuple('ScriptData', 'name,output')


def iter_data(data_set, data_type):
    for data in data_set:
        if isinstance(data, data_type):
            yield data

def iter_port_data(data_set):
    return iter_data(data_set, PortData)

def iter_script_data(data_set):
    return iter_data(data_set, ScriptData)

def remove_outdated(port_number, port_type, data_set):
    # If additional information was found on a port that was previously scanned,
    # remove the current entry from data_set so that we may add the new one.
    outdated_ports = set()
    for existing_port in iter_port_data(data_set):
        if port_number == existing_port.number and port_type == existing_port.type:
            outdated_ports.add(existing_port)
    for outdated_port in outdated_ports:
        data_set.remove(outdated_port)

def add_port_data(data_set, port_type, port_dict):
    for port_number, port_data in port_dict.items():
        remove_outdated(port_number, port_type, data_set)
        data_set.add(PortData(
            port_number,
            port_type,
            port_data['name'],
            port_data['product'],
            port_data['version']
        ))
        if 'script' not in port_data.keys():
            continue
        for script, script_output in port_data['script'].items():
            data_set.add(ScriptData(script, script_output))

def compile_results():
    global results, host_data
    for host, data in results['scan'].items():
        # Scripts against each port
        if UDP in data:
            add_port_data(host_data[host], UDP, data[UDP])
        if TCP in data:
            add_port_data(host_data[host], TCP, data[TCP])
        # Scripts against the host
        if 'hostscript' not in data:
            continue
        for script in data['hostscript']:
            script_output = script['output']
            host_data[host].add(ScriptData(script['id'], script['output']))

def write_report():
    global host_data
    for host, data in host_data.items():
        with open(F'{mypath}/../output/{host}_{myname}.md', 'w') as f:
            f.write(F'## {host}\n\n')
            f.write('### Reconnaissance\n\n|Num/Type|Name|Product|Version|')
            f.write('\n|---:|:---|:---|:---|')
            for port in iter_port_data(data):
                output = '\n|{}/{}|{}|{}|{}|'.format(
                    port.number,
                    port.type,
                    port.name,
                    port.product,
                    port.version
                )
                f.write(output)
            f.write('\n')
            f.write('\n### Vulnerability Assessment\n\n```')
            for script in iter_script_data(data):
                f.write('\n{}\n'.format(script.name.rstrip()))
                f.write('{}\n'.format(script.output.strip('\r\n')))
            f.write('```\n')

def sweep_tcp(scanner, host, results):
    # Check for extra ports while excluding already scanned (not necessarily open) ports.
    scanned_ports = results['nmap']['scaninfo']['tcp']['services']
    for top_ports in range(100, 9001, 100):
        extra_args = F' --top-ports {top_ports} --exclude-ports {scanned_ports}'
        new_results = scanner.scan(
            hosts=host,
            arguments=base_args + extra_args
        )

        try:
            last_ports = new_results['nmap']['scaninfo']['tcp']['services']
            if last_ports:
                scanned_ports += ',' + last_ports
            if new_results['scan']:
                if not results['scan']:
                    results = dict(new_results)
                else:
                    results['scan'][host]['tcp'].update(new_results['scan'][host]['tcp'])
                compile_results()
                write_report()
        except KeyError:
            continue

if __name__ == '__main__':
    mypath = path.dirname(path.abspath(__file__))
    myname = path.basename(__file__)

    UDP = 'udp'
    TCP = 'tcp'

    if len(argv) != 3:
        print(F'Usage: {myname} <source interface> <nmap XML file>')
        print('Uses the given nmap XML file to carry out additional nmap scans from the specified interface.')
        exit(1)

    iface = argv[1]
    try:
        check_output(F'ip l sh {iface}'.split(' '))
    except CalledProcessError:
        print(F'Unknown interface: {iface}')
        exit(1)

    xml_file = argv[2]
    if not path.isfile(xml_file):
        print(F'File not found: {xml_file}')
        exit(1)

    with open(xml_file) as f:
        results = ps().analyse_nmap_xml_scan(f.read())

    if not results['scan']:
        print(F'Empty scan results in {xml_file}! Exiting.')
        exit(1)
    else:
        host_data = ddict(set)
        base_args = F'-e {iface} -n -Pn --open -sT'
        compile_results()
        write_report()

    for host, data in host_data.items():
        scanner = ps()
        sweep_tcp(scanner, host, results)

        # Enumerate services on open ports only.
        new_results = scanner.scan(
            hosts=host,
            ports=','.join(str(port.number) for port in iter_port_data(data)),
            arguments=base_args + 'V'
        )

        results.update(new_results)
        compile_results()
        write_report()

        # Now that we have (hopefully accurate) services, add corresponding scripts.
        scripts = set()
        for port in iter_port_data(data):
            if 'ftp' in port.name:
                scripts.add('ftp-anon')
                scripts.add('ftp-vuln*')
            if 'http' in port.name:
                scripts.add('http-vuln*')
            if 'smtp' in port.name:
                scripts.add('smtp-vuln*')
            if 'nfs' in port.name:
                scripts.add('nfs-showmount')
            if 'netbios-ssn' in port.name or 'microsoft-ds' in port.name:
                scripts.add('smb-ls')
                scripts.add('smb-os-discovery')
                scripts.add('smb-vuln*')
                if 'microsoft-ds' in port.name:
                    scripts.add('snmp-win*')

        extra_args = 'V'
        if scripts:
            extra_args += ' --script {}'.format(','.join(scripts))
        new_results = scanner.scan(
            hosts=host,
            ports=','.join(str(port.number) for port in iter_port_data(data)),
            arguments=base_args + extra_args
        )

        results.update(new_results)
        compile_results()
        write_report()
Published inUncategorized