NginxLogger - Nginx Log parser in Go for MongoDB

This tool parses nginx access log in following format[1] and inserts them into MongoDB in format[2]

Nginx default log format[1]

$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"

MongoDB document format[2]

{
    "_id" : ObjectId("59ce205c460c807614ce4728"),
	"remoteip" : {
		"continent" : "",
		"ip" : "",
		"country" : "",
		"countrycode" : "",
		"city" : "",
		"location" : {
			"latitude" : 0,
			"longitude" : 0
		}
	},
	"timestamp" : ISODate("2017-08-05T14:55:00Z"),
	"requestinguser" : "-",
	"method" : "",
	"request" : "/",
	"httpversion" : "",
	"responsecode" : "",
	"size" : 0,
	"referrer" : "",
	"client" : {
		"useragent" : {
			"family" : "",
			"major" : "",
			"minor" : "",
			"patch" : ""
		},
		"os" : {
			"family" : "",
			"major" : "",
			"minor" : "",
			"patch" : "",
			"patchminor" : ""
		},
		"device" : {
			"family" : ""
		}
	}
}

Installing dependencies

Make sure your GOROOT environment variable is corrent.

$ go get github.com/oschwald/geoip2-golang
$ go get github.com/tobie/ua-parser/go/uaparser
$ go get github.com/fsnotify/fsnotify
$ go get gopkg.in/mgo.v2/bson
$ go get golang.org/x/crypto/bcrypt

Database Configuration

Database configuration is defined in dbmanager.go

const DATABASE = "nginxLogger"
const C_USER = "user"
const C_LOG = "access_log"
const C_FILE = "file_position"

DATABASE - database name.

C_USER - collection to store username/password (to facilitate login for other applications using data ).

C_LOG - collection to store logs.

C_FILE - collection to store file pointer.

Usage

1. Install and start MongoDB
2. Install dependencies
3. Build using make
4. Start using ./dbmanager

Binary takes two arguments

-path - Path to nginx access logs. (Defaults to /var/log/nginx/access.log) -adduser (optional) - To add user for facilitating login for other applications using this data