ntopng – ntop Next Gen

ntopng is the next generation version of the original ntop, a network traffic probe that shows the network usage, similar to what the popular top Unix command does. ntopng is based on libpcap and it has been written in a portable way in order to virtually run on every Unix platform, MacOSX and on Win32 as well.

ntopng users can use a a web browser to navigate through ntop (that acts as a web server) traffic information and get a dump of the network status. In the latter case, ntopng can be seen as a simple RMON-like agent with an embedded web interface. The use of:

  • a web interface.
  • limited configuration and administration via the web interface.
  • reduced CPU and memory usage (they vary according to network size and traffic).
  • Sort network traffic according to many protocols
  • Show network traffic and IPv4/v6 active hosts
  • Store on disk persistent traffic statistics in RRD format
  • Geolocate hosts
  • Discover application protocols by leveraging on nDPI, ntop’s DPI framework.
  • Characterise HTTP traffic by leveraging on characterisation services provided by block.si. ntopng comes with a demo characterisation key, but if you need a permanent one, please mail [email protected]
  • Show IP traffic distribution among the various protocols
  • Analyse IP traffic and sort it according to the source/destination
  • Display IP Traffic Subnet matrix (who’s talking to who?)
  • Report IP protocol usage sorted by protocol type
  • Act as a NetFlow/sFlow collector for flows generated by routers (e.g. Cisco and Juniper) or switches (e.g. Foundry Networks) when used together with nProbe.
  • Produce HTML5/AJAX network traffic statistics

Installation (Ubuntu Server example):

At the time of writing ntopng does not appear in any of the repos, so you can install it from source or they do have binary packages available as nightly builds. However, there is a ppa for Ubuntu which is what I’ve used here to keeps things quick and easy to get a feel for the application. For binary downloads and source code see http://www.ntop.org/get-started/download/

PPA Install:
PPA from https://launchpad.net/~cavedon/+archive/ntop

sudo add-apt-repository ppa:cavedon/ntop
sudo apt-get update
sudo apt-get install ntopng
sudo service ntopng start 

ntopng should be running now, a quick check with a netstat verifies this as it runs on port 3000

sudo netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      857/sshd
tcp        0      0 0.0.0.0:3000            0.0.0.0:*               LISTEN      2306/ntopng
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      2163/redis-server 1
tcp6       0      0 :::22                   :::*                    LISTEN      857/sshd      

Open up a browser and point to the server with port 3000 e.g. http://yourservername:3000 as shown here:
ntopng

PSSH/CSSH/DSH – Tools for managing multiple Linux servers

If you only manage a few Linux servers, perhaps 3-5 in a small network or your home network and don’t really want to setup configuration automation such as Puppet then some of these tools will be very handy. Even so, these tools can still be used together with Puppet or to manage your servers and make things easier and quicker!

Note: with these tools you should first ensure that SSH key based logins have been setup and are working.

PSSH – Parallel SSH http://code.google.com/p/parallel-ssh/

PSSH is a program for executing ssh in parallel on a number of hosts. It provides features such as sending input to all of the processes, passing a  password  to  ssh,  saving output to files, and timing out. The  PSSH_NODENUM  and  PSSH_HOST  environment variables are sent to the remote host. The PSSH_NODENUM variable is assigned a unique number for each ssh connection, starting with 0 and  counting up.  The PSSH_HOST variable is assigned the name of the host as specified in the hosts list.  Note that sshd drops environment variables by default, so sshd_config on the remote host must include the line: AcceptEnv PSSH_NODENUM PSSH_HOST

Example:
Copy a file to several servers, in this case I’m copying a file named scripts.tar.gz to the home folder of keith on 2 servers named gummy and snapper:

 pscp.pssh -v -H "[email protected] [email protected]" scripts.tar.gz /home/keith 

CSSH – ClusterSSH http://sourceforge.net/projects/clusterssh/

ClusterSSH is a tool for making the same change on multiple servers at the same time. The ‘cssh’ command opens an xterm to all specified hosts and an administration console. Any text typed into the console is replicated to all windows. All windows may also be typed into directly.

This tool is intended for (but not limited to) cluster administration where the same configuration or commands must be run on each node within the cluster. Performing these commands all at once via this tool ensures all nodes are kept in sync.

Example:
Create cluster config file named clusters in /etc which defines my two Ubuntu 12.04 servers of Gummy and Snapper:

 vim /etc/clusters
clusters = 1204servers
1204servers = gummy snapper 

Now execute the following:

 cssh [email protected] [email protected] 

You will now have two terminal windows appear and the same command will be issued to both terminals as you type it! Just Be Careful with what commands you issue.

DSH – Distributed Shell http://www.netfort.gr.jp/~dancer/software/dsh.html.en

DSH is short for “Distributed Shell” or “Dancer’s Shell” it is freely available on most major distributions of Linux, but can easily be built from source if your distribution does not offer it in its package repository. On Debian & Ubuntu “apt-get install dsh” and on Redhat/CentOS “yum install pdsh”.

Example:

Modify the configuration file to change the protocol to SSH in /etc/dsh/dsh.conf so that the line:
“remoteshell =rsh” is “remoteshell =ssh”
Now add your servers into the machines.list file:

 vim /etc/dsh/machines.list
gummy
snapper 

Let’s see the uptime and disk space details on servers gummy and snapper:

 dsh -aM -c uptime 
 dsh -aM -c df -h 

fping utility

fping is a program to send ICMP echo probes to network hosts, similar to ping, but much better performing when pinging multiple hosts. fping differs from ping in that you can specify any number of targets on the command line, or specify a file containing the lists of targets to ping. Instead of sending to one target until it times out or replies, fping will send out a ping packet and move on to the next target in a round-robin fashion. In the default mode, if a target replies, it is noted and removed from the list of targets to check; if a target does not respond within a certain time limit and/or retry limit it is designated as unreachable. fping also supports sending a specified number of pings to a target, or looping indefinitely (as in ping ). Unlike ping, fping is meant to be used in scripts, so its output is designed to be easy to parse.

To quickly check a network:

 fping -g 192.168.1.0/24 

To install fping:
Debian/Ubuntu

 apt-get install fping 

CentOS/Redhat

 yum install fping 

Options:

Usage: fping [options] [targets…]
-a         show targets that are alive
-A         show targets by address
-b n       amount of ping data to send, in bytes (default 56)
-B f       set exponential backoff factor to f
-c n       count of pings to send to each target (default 1)
-C n       same as -c, report results in verbose format
-D         print timestamp before each output line
-e         show elapsed time on return packets
-f file    read list of targets from a file ( – means stdin) (only if no -g specified)
-g         generate target list (only if no -f specified)
(specify the start and end IP in the target list, or supply a IP netmask)
(ex. fping -g 192.168.1.0 192.168.1.255 or fping -g 192.168.1.0/24)
-H n       Set the IP TTL value (Time To Live hops)
-i n       interval between sending ping packets (in millisec) (default 25)
-I if      bind to a particular interface
-l         loop sending pings forever
-m         ping multiple interfaces on target host
-n         show targets by name (-d is equivalent)
-O n       set the type of service (tos) flag on the ICMP packets
-p n       interval between ping packets to one target (in millisec)
(in looping and counting modes, default 1000)
-q         quiet (don’t show per-target/per-ping results)
-Q n       same as -q, but show summary every n seconds
-r n       number of retries (default 3)
-s         print final stats
-S addr    set source address
-t n       individual target initial timeout (in millisec) (default 500)
-T n       ignored (for compatibility with fping 2.4)
-u         show targets that are unreachable
-v         show version
targets    list of targets to check (if no -f specified)