001.netstat.to.elasticsearch.qbana.force.directed.graph.md

Network graph and flow: netstat to Elasticsearch+QBana

This example shows how to draw a force directed graph of the IP connections between a number of machines.

• The information will be obtained from periodical netstat collections.
• Data will be indexed in Elasticsearch.
• Display of the graph will be done using QBana (tough your tailored application with D3JS will surelly have more features)

Something like this:

You will need to have installed:

• probespawner - https://github.com/filipealmeida/probespawner
• elasticsearch - https://www.elastic.co/
• qbana - https://github.com/QXIP/Qbana

The elasticsearch available is configured by default at the suchhost host.
QBana will be setup on a webserver of your choice, in this example, it was a localhost connected to suchhost's elasticsearch cluster.

We will now:

1. Define netstat input
2. Define elasticsearch output
3. Define netstat input as the active input (probe)
4. Save the JSON file
5. Run probespawner
6. Access QBana dashboard

1. Define netstat input:

The input will be configured to issue a netstat -nte every 30 seconds.
The JSON block will be as follows:

    "NetstatInput": {
       "description": "Executes netstat every 30 seconds",
       "probemodule": { "module": "netstatntc", "name" : "LinuxNetstatNTC" },
       "output": ["ElasticsearchOutput"],
       "command": "netstat -nte",
       "interval": 30
    }

2. Define elasticsearch output:

The elasticsearch output block will include a number of index properties and a type mapping that will fit other needs not discussed here:

    "ElasticsearchOutput": {
      "class": "elasticsearch",
      "outputmodule": { "module": "jelh", "name" : "Elasticsearch" },
      "cluster" : "elasticsearch",
      "type": "linuxmetrics",
      "host" : "suchhost",
      "port" : 9300,
      "bulkActions": 1000,
      "flushInterval": 5,
      "indexPrefix": "idxnetstat",
      "index_settings": {
        "index" : {
          "analysis" : {
            "analyzer" : {
              "default" : {
                "type" : "standard",
                "filter": ["standard", "asciifolding","lowercase"]
              },
              "sortable": {
                "tokenizer": "keyword",
                "filter": ["lowercase"]
              }
            }
          }
        }
      },
      "type_mapping": {
        "linuxmetrics": { "properties" : {
            "@timestamp" : { "type" : "date" },
            "metric" : {"type" : "string", "index" : "not_analyzed"},
            "class" : {"type" : "string", "index" : "not_analyzed"},
            "host" : {"type" : "string", "index" : "not_analyzed"},
            "value" : {"type" : "double" }
            }
          }
      }
    }

3. Define netstat input as the active input (probe):

Lastly, define the active input setting a value in the input element:

"input": ["NetstatInput"]

4. All together now

The whole JSON:

{
    "NetstatInput": {
       "description": "Executes netstat every 30 seconds",
       "probemodule": { "module": "netstatntc", "name" : "LinuxNetstatNTC" },
       "output": ["ElasticsearchOutput"],
       "command": "netstat -nte",
       "interval": 30
    },
    "ElasticsearchOutput": {
      "class": "elasticsearch",
      "outputmodule": { "module": "jelh", "name" : "Elasticsearch" },
      "cluster" : "elasticsearch",
      "type": "linuxmetrics",
      "host" : "suchhost",
      "port" : 9300,
      "bulkActions": 1000,
      "flushInterval": 5,
      "indexPrefix": "idxnetstat",
      "index_settings": {
        "index" : {
          "analysis" : {
            "analyzer" : {
              "default" : {
                "type" : "standard",
                "filter": ["standard", "asciifolding","lowercase"]
              },
              "sortable": {
                "tokenizer": "keyword",
                "filter": ["lowercase"]
              }
            }
          }
        }
      },
      "type_mapping": {
        "linuxmetrics": { "properties" : {
            "@timestamp" : { "type" : "date" },
            "metric" : {"type" : "string", "index" : "not_analyzed"},
            "class" : {"type" : "string", "index" : "not_analyzed"},
            "host" : {"type" : "string", "index" : "not_analyzed"},
            "value" : {"type" : "double" }
            }
          }
      }
    },
    "input": ["NetstatInput"]
}

5. Run probespawner

With this file, let's call it netstat.json, we will run on each host we want to monitor:

[user@localhost probespawner-master]$ ./probespawner.sh netstat.json
2015-06-04 16:23:54,022 dummyprobe     INFO     Started cycle at 2015-06-04T16:23:54.020+01:00
2015-06-04 16:23:54,036 dummyprobe     INFO     Started cycle at 2015-06-04T16:23:54.035+01:00
2015-06-04 16:23:54,121 jelh           INFO     Setting Elasticsearch options: cluster.name = elasticsearch
Jun 04, 2015 4:23:54 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [Firelord] loaded [], sites []
2015-06-04 16:23:55,515 dummyprobe     INFO     Finished cycle at 2015-06-04T16:23:55.514+01:00
2015-06-04 16:23:55,516 dummyprobe     INFO     End of cycle: sleeping for 299 seconds
2015-06-04 16:23:55,957 jelh           INFO     Flushing 1000 records
2015-06-04 16:23:56,497 jelh           INFO     Flushing 1000 records
2015-06-04 16:23:56,940 jelh           INFO     Flushing 1000 records
2015-06-04 16:23:57,328 jelh           INFO     Flushing 1000 records
2015-06-04 16:23:57,578 jelh           INFO     Flushing 418 records
2015-06-04 16:23:57,648 dummyprobe     INFO     Finished cycle at 2015-06-04T16:23:57.648+01:00
2015-06-04 16:23:57,648 dummyprobe     INFO     End of cycle: sleeping for 27 seconds
...

6. Access QBana dashboard

On the QBana dashboard, we configure the index idxnetstat as stated in the output configuration of elasticsearch and then a few histograms, a flow and a force panels.

Force and flow panels' source is localip field and target is foreignip.

The result is as such: