Add ElasticSearch event forwarding

Add the ability to forward all events
Update rulesets
Cleaned Zircogui code

.gitignore

@@ -33,4 +33,4 @@ tmp
 *.evtx
 *.evtx_data
 *.tar
-*.tar.sha256
+*.tar.sha256

Readme.md

@@ -4,15 +4,13 @@
 ![](pics/Zircolite.svg)
 
 [![python](https://img.shields.io/badge/python-3.8-blue)](https://www.python.org/)
-![version](https://img.shields.io/badge/Platform-Win-green)
-![version](https://img.shields.io/badge/Platform-Lin-green)
-![version](https://img.shields.io/badge/Platform-Mac-green)
 ![version](https://img.shields.io/badge/Architecture-64bit-red)
 
 **Zircolite is a standalone tool written in Python 3. It allows to use SIGMA rules on MS Windows EVTX (EVTX and JSONL format), Auditd logs and Sysmon for Linux logs**
 
 - **Zircolite** can be used directly on the investigated endpoint (use [releases](https://github.com/wagga40/Zircolite/releases)) or in your forensic/detection lab
-- **Zircolite** is fast and can parse large datasets in just seconds (check [benchmarks](docs/Internals.md#benchmarks))
+- **Zircolite** is relatively fast and can parse large datasets in just seconds (check [benchmarks](docs/Internals.md#benchmarks))
+- **Zircolite** is based on a Sigma backend (SQLite) and do not use internal sigma to something conversion 
 
 **Zircolite can be used directly in Python or you can use the binaries provided in [releases](https://github.com/wagga40/Zircolite/releases) (Microsoft Windows and Linux  only).** 
 **Documentation is [here](docs).**
@@ -30,29 +28,27 @@ The use of [evtx_dump](https://github.com/omerbenamram/evtx) is **optional but r
 Help is available with `zircolite.py -h`. If your EVTX files have the extension ".evtx" :
 
 ```shell
-python3 zircolite.py --evtx <EVTX_FOLDER/EVTX_FILE> --ruleset <Converted Sigma rules>
+# python3 zircolite.py --evtx <EVTX FOLDER or EVTX FILE> --ruleset <SIGMA RULES JSON FILE>
 python3 zircolite.py --evtx sysmon.evtx --ruleset rules/rules_windows_sysmon.json
 ```
 The SYSMON ruleset used here is a default one and it is for logs coming from endpoints where SYSMON installed. A generic ruleset is available too.
 
 #### Auditd logs : 
 
 ```shell
-python3 zircolite.py --evtx <EVTX_FOLDER/EVTX_FILE> --ruleset <Converted Sigma rules> --auditd
-python3 zircolite.py --evtx auditd.log --ruleset rules/rules_linux.json --auditd
+python3 zircolite.py --events auditd.log --ruleset rules/rules_linux.json --auditd
 ```
 
 #### Sysmon for Linux logs : 
 
 ```shell
-python3 zircolite.py --evtx <EVTX_FOLDER/EVTX_FILE> --ruleset <Converted Sigma rules> --sysmon4linux
-python3 zircolite.py --evtx auditd.log --ruleset rules/rules_linux.json --sysmon4linux
+python3 zircolite.py --events sysmon.log --ruleset rules/rules_linux.json --sysmon4linux
 ```
 
 #### JSONL/NDJSON files : 
 
 ```shell
-python3 zircolite.py --evtx <JSON_FOLDER/JSON_FILE> --ruleset rules/rules_windows_sysmon.json --jsononly
+python3 zircolite.py --evtx <JSON_FOLDER or JSON_FILE> --ruleset rules/rules_windows_sysmon.json --jsononly
 ```
 
 :information_source: If you want to try the tool you can test with these samples : 

docs/Advanced.md

@@ -32,7 +32,7 @@ The tool has been created to be used on very big datasets and there are a lot of
 
 Except when `evtx_dump` is used, Zircolite only use one core. So if you have a lot of EVTX files and their total size is big, it is recommanded that you use a script to launch multiple Zircolite instances. On Linux or MacOS The easiest way is to use **GNU Parallel**. 
 
-:information_source: on MacOS, please use GNU find (`brew install find` will install `gfind`)
+ℹ️ on MacOS, please use GNU find (`brew install find` will install `gfind`)
 
 - **"DFIR Case mode" : One directory per computer/endpoint**
 
@@ -58,10 +58,6 @@ Except when `evtx_dump` is used, Zircolite only use one core. So if you have a l
 
 	In this example the `-j -1` is for using all cores but one. You can adjust the number of used cores with this arguments.
 
-#### Using Zircolite MP 
-
-***deprecated***
-
 ---
 
 ### Filtering
@@ -77,7 +73,7 @@ To speed up the detection process, you may want to use Zircolite on files matchi
 - `-s` or `--select` : select files partly matching the provided a string (case insensitive)
 - `-a` or `--avoid` : exclude files partly matching the provided a string (case insensitive)
 
-:information_source: When using te two arguments, the "select" argument is always applied first and then the "avoid" argument is applied. So, it is possible to exclude files from included files but not the opposite.
+ℹ️ When using the two arguments, the "select" argument is always applied first and then the "avoid" argument is applied. So, it is possible to exclude files from included files but not the opposite.
 
 - Only use EVTX files that contains "sysmon" in their names
 
@@ -150,7 +146,7 @@ You can also specify a string, to avoid unexpected side-effect **comparison is c
 ```shell
 python3 zircolite.py --evtx logs/ --ruleset rules/rules_windows_sysmon.json -R BFFA7F72 -R MSHTA
 ```
-:information_source: As of version 2.2.0 of Zircolite, since the rulesets are directly generated from the official `sigmac` tool there is no more CRC32 in the rule title. Rule filtering is still available but you have to rely on other criteria.
+ℹ️ As of version 2.2.0 of Zircolite, since the rulesets are directly generated from the official `sigmac` tool there is no more CRC32 in the rule title. Rule filtering is still available but you have to rely on other criteria.
 
 #### Limit the number of detected events
 
@@ -172,7 +168,7 @@ For now, the forwarders are not asynchronous so it can slow Zircolite execution.
 
 If you forward your events to a central collector you can disable local logging with the Zircolite `--nolog` argument.
 
-#### Forward to a HTTP server
+#### Forward events to a HTTP server
 
 If you have multiple endpoints to scan, it is usefull to send the detected events to a central collector. As of v1.2, Zircolite can forward detected events to an HTTP server :
 
@@ -182,7 +178,7 @@ python3 zircolite.py --evtx sample.evtx  --ruleset rules/rules_windows_sysmon.js
 ```
 An **example** server called is available in the [tools](tools/) directory.
 
-#### Forward to a Splunk instance via HEC
+#### Forward events to a Splunk instance via HEC
 
 As of v1.3.5, Zircolite can forward detections to a Splunk instance with Splunk **HTTP Event Collector**.
 
@@ -196,10 +192,31 @@ python3 zircolite.py --evtx /sample.evtx  --ruleset rules/rules_windows_sysmon.j
 
 :warning: On Windows do not forget to put quotes
 
+#### Forward to ELK
+
+As of version 2.8.0, Zircolite can forward events to an ELK stack using the ES client.
+
+```shell
+python3 zircolite.py --evtx /sample.evtx  --ruleset rules/rules_windows_sysmon.json \
+	--remote "https://x.x.x.x:8088" --index "zircolite-whatever" \
+	--eslogin "yourlogin" --espass "yourpass"
+```
+
+ℹ️ the `--eslogin` and `--espass` arguments are optional.
+
+⚠️ **Elastic is not handling logs the way Splunk does. Since Zircolite is flattening the field names in the JSON output some fields, especially when working with EVTX files, can have different types between Channels, logsources etc. So when Elastic uses automatic field mapping, mapping errors may prevent events insertion into Elastic.**
+
 #### No local logs
 
 When you forward detected events to an server, sometimes you don't want any log file left on the system you have run Zircolite on. It is possible with the `--nolog` option.
 
+### Forwarding all events 
+
+Zircolite is able to forward all events and not just the detected events to Splunk, ELK or a custom HTTP Server. you just to use the `--forwardall` argument. Please note that this ability forward events as JSON and not specific  `Windows` sourcetype.
+
+⚠️ **Elastic is not handling logs the way Splunk does. Since Zircolite is flattening the field names in the JSON output some fields, especially when working with EVTX files, can have different types between Channels, logsources etc. So when Elastic uses automatic field mapping, mapping errors may prevent events insertion into Elastic.**
+
+
 ---
 
 ### Templating and Formatting
@@ -313,7 +330,7 @@ Basically, if you want to integrate Zircolite with **DFIR Orc** :
 </wolf>
 ```
 
-:information_source: Please note that if you add this configuration to an existing one, you only need to keep the part between `<!-- BEGIN ... -->` and `<!-- /END ... -->` blocks.
+ℹ️ Please note that if you add this configuration to an existing one, you only need to keep the part between `<!-- BEGIN ... -->` and `<!-- /END ... -->` blocks.
 
 -  Put your custom or default mapping file `zircolite_win10_nuitka.exe ` (the default one is in the Zircolite repository `config` directory)   `rules_windows_generic.json` (the default one is in the Zircolite repository `rules` directory) in the the `config` directory.
 
@@ -352,7 +369,7 @@ Basically, if you want to integrate Zircolite with **DFIR Orc** :
 	</archive>
 </toolembed>
 ```
-:information_source: Please note that if you add this configuration to an existing one, you only need to keep the part between `<!-- BEGIN ... -->` and `<!-- /END ... -->` blocks.
+ℹ️ Please note that if you add this configuration to an existing one, you only need to keep the part between `<!-- BEGIN ... -->` and `<!-- /END ... -->` blocks.
 
 - Now you need to generate the **DFIR Orc** binary by executing `.\configure.ps1` at the root of the repository
 - The final output will be in the `output` directory

docs/Usage.md

@@ -46,20 +46,39 @@ It also works directly on an unique EVTX file.
 
 By default : 
 
+- `--ruleset` is not mandatory but the default ruleset will be `rules/rules_windows_generic.json`
 - Results are written in the `detected_events.json` in the same directory as Zircolite
 - There is a `zircolite.log`file that will be created in the current working directory
 
+#### Auditd logs : 
+
+```shell
+python3 zircolite.py --events auditd.log --ruleset rules/rules_linux.json --auditd
+```
+
+:information_source: `--events` and `--evtx` are stricly equivalent but `--events` is more logical to use for non EVTX logs.
+
+#### Sysmon for Linux logs : 
+
+Sysmon for linux has been released in October 2021. It outputs XML in text format with one event per-line. As of version 2.6.0, **Zircolite** has an *initial* support of Sysmon for Linux log files. To test it, just add `-S` to you command line : 
+
+```shell
+python3 zircolite.py --events sysmon.log --ruleset rules/rules_linux.json --sysmon4linux
+```
+
+:information_source: Since the logs come from Linux, the default file extension when using `-S` case is `.log`
+
 #### JSONL/NDJSON
 
 It is possible to use Zircolite directly on JSONL/NDJSON files (NXLog files) with the `--jsononly` or `-j` arguments : 
 
 ```shell
-python3 zircolite.py --evtx <EVTX_FOLDER> --ruleset <CONVERTED_SIGMA_RULES> --jsononly
+python3 zircolite.py --events <EVTX_FOLDER> --ruleset <CONVERTED_SIGMA_RULES> --jsononly
 ```
 
 A simple use case is when you have already run Zircolite and use the `--keeptmp` option. Since it keeps all the converted EVTX in a temp directory, if you need to re-execute Zircolite, you can do it directly using this directory as the EVTX source (with `--evtx <EVTX_IN_JSON_DIRECTORY>` and `--jsononly`) and avoid to convert the EVTX again.
 
-:information_source: If you you can change the file extension with `--fileext`.
+:information_source: You can change the file extension with `--fileext`.
 
 #### SQLite database files
 
@@ -69,17 +88,7 @@ Since everything in Zircolite is stored in a in-memory SQlite database, you can
 python3 zircolite.py --evtx <EVTX_FOLDER> --ruleset <CONVERTED_SIGMA_RULES> --dbfile output.db
 ```
 
-If you need to re-execute Zircolite,  you can do it directly using the SQLite database as the EVTX source (with `--evtx <SAVED_SQLITE_DB_PATH>` and `--dbonly`) and avoid to convert the EVTX, post-process the EVTX and insert data to database. **Using this technique can save a lot of time...** 
-
-#### Sysmon for Linux XML log files
-
-Sysmon for linux has been released in October 2021. It outputs XML in text format with one event per-line. As of version 2.6.0, **Zircolite** has an *initial* support of Sysmon for Linux log files. To test it, just add `-S` to you command line : 
-
-```shell
-python3 zircolite.py --evtx <EVTX_FOLDER> --ruleset <CONVERTED_SIGMA_RULES> -S
-```
-
-:information_source: Since the logs come from Linux, the default file extension when using `-S` case is `.log`
+If you need to re-execute Zircolite,  you can do it directly using the SQLite database as the EVTX source (with `--evtx <SAVED_SQLITE_DB_PATH>` and `--dbonly`) and avoid to convert the EVTX, post-process the EVTX and insert data to database. **Using this technique can save a lot of time... But you will be unable to use the `--forwardall`option** 
 
 ---
 
@@ -89,16 +98,14 @@ Default rulesets are already provided in the `rules` directory. These rulesets o
 
 #### With sigmatools
 
-Zircolite use the SIGMA rules in JSON format. To generate your ruleset you need the official sigmatools (version 0.20 minimum) : 
+Zircolite use the SIGMA rules in JSON format. To generate your ruleset you need the official sigmatools (**version 0.21 minimum**) : 
 
 ```shell 
 git clone https://github.com/SigmaHQ/sigma.git
 cd sigma
 ```
 **You must have the sigma dependencies installed, check [here](https://github.com/SigmaHQ/sigma#installation) :**
 
-[DEPRECATED] The pip version of sigmatools works but the backend has been updated since
-
 ##### Sysmon rulesets (when investigated endpoints have Sysmon logs)
 
 ```shell 
@@ -140,7 +147,7 @@ tools/sigmac \
 
 #### On the fly rules conversion
 
-Since Zircolite 2.2.0, if you have sigmatools >= 0.20, Zircolite is able to convert the rules on-the-fly if you provide a SIGMA config file and the `sigmac` path. It is very convenient for testing but you should avoid it since this is slower : 
+Since Zircolite 2.2.0, if you have sigmatools >= 0.21, Zircolite is able to convert the rules on-the-fly if you provide a SIGMA config file and the `sigmac` path. It is very convenient for testing but you should avoid it since this is slower : 
 
 ```shell
 python3 zircolite.py --evtx ../Samples/EVTX-ATTACK-SAMPLES/ \

requirements.txt

@@ -1,7 +1,11 @@
 jinja2>=2.11.3
 colorama>=0.4.4
 tqdm>=4.58.0
+urllib3>=1.26.2
 requests>=2.24.0
 evtx>=0.6.2
 aiohttp[speedups]~=3.7
 lxml~=4.6
+elasticsearch[async]>=7.8.0
+orjson>=3.6.0
+xxhash