Added the ability to filter events by date, and to filter rule by name

Updated readme & rulesets

Makefile

@@ -3,7 +3,7 @@
 DOCKER?=docker
 DOCKER_BUILD_FLAGS?=
 DOCKER_REGISTRY?=docker.io
-DOCKER_TAG?=1.2.5
+DOCKER_TAG?=1.4.0
 
 define HELP_MENU
 	Usage: make [<env>] <target> [<target> ...]

Readme.md

@@ -57,6 +57,7 @@ python3 zircolite.py --evtx ../Logs --ruleset rules/rules_windows_sysmon.json
 Some EVTX files are not used by SIGMA rules but can become quite large (`Microsoft-Windows-SystemDataArchiver%4Diagnostic.evtx` etc.), if you use Zircolite with directory as input argument, all EVTX files will be converted, saved and matched against the SIGMA Rules. 
 
 To speed up the detection process, you may want to use Zircolite on files matching or not matching a specific pattern. For that you can use **filters** provided by the two command line arguments :
+
 - `-s` or `--select` : select files partly matching the provided a string (case insensitive)
 - `-a` or `--avoid` : exclude files partly matching the provided a string (case insensitive)
 
@@ -73,7 +74,7 @@ python3 zircolite.py --evtx logs/ --ruleset rules/rules_windows_sysmon.json \
 
 ```
 
-For example, the **Sysmon** ruleset available in the `rules` directory only use the following channels (names have been shortened) : Sysmon, Security, System, Powershell, Defender, AppLocker, DriverFrameworks, Application, NTLM, DNS, MSexchange, WMI-activity, TaskScheduler. So if you use the sysmon ruleset with the following rules, you should speed up `Zircolite`execution : 
+For example, the **Sysmon** ruleset available in the `rules` directory only use the following channels (names have been shortened) : *Sysmon, Security, System, Powershell, Defender, AppLocker, DriverFrameworks, Application, NTLM, DNS, MSexchange, WMI-activity, TaskScheduler*. So if you use the sysmon ruleset with the following rules, you should speed up `Zircolite`execution : 
 
 ```shell
 python3 zircolite.py --evtx logs/ --ruleset rules/rules_windows_sysmon.json \
@@ -86,6 +87,25 @@ python3 zircolite.py --evtx logs/ --ruleset rules/rules_windows_sysmon.json \
 
 :information_source: the "select" argument is always applied first and then the "avoid" argument is applied. So, it is possible to exclude files from included files but not the opposite.
 
+### Date filtering
+
+Sometimes you want to work on a selected time range to speed up analysis. With Zircolite, it is possible to filter on specific time range just by using the `--after` and `--before` and their respective shorter versions `-A` and `-B`. The filter will apply on the `SystemTime` field of each event : 
+
+```shell
+python3 zircolite.py --evtx logs/ --ruleset rules/rules_windows_sysmon.json \ 
+-A 2021-06-02T22:40:00 -B 2021-06-02T23:00:00
+```
+
+The `--after` and `--before` arguments can be used independantly.
+
+### Rule filtering
+
+Some rules can be noisy or slow on specific datasets (check [here](rules/Readme.md)) so it is possible to skip stop by using the `-R` or `--rulefilter` argument. The filter will apply on the rule title. Since there is a CRC32 in the rule title it is easiers to use it : 
+
+```shell
+python3 zircolite.py --evtx logs/ --ruleset rules/rules_windows_sysmon.json -R BFFA7F72
+``` 
+
 ### Templating
 
 Zircolite provides a templating system based on Jinja 2. It allows you to change the output format to suits your needs (Splunk or ELK integration, Grep-able output...). To use the template system, use these arguments :

rules/Readme.md

@@ -1,8 +1,22 @@
 # Rulesets
 
+## Default rulesets
+
 These rulesets have been generated with `genRules.py` wich is available in the folder `tools` of the Zircolite repository.
 
 :warning: **These rulesets are given "as is" to help new analysts to discover SIGMA and Zircolite. They are not filtered for slow rules, rules with a lot of false positives etc. If you know what you do, you MUST generate your own rulesets.**
 
 - `rules_windows_generic.json` : Full SIGMA "**Windows**" ruleset (no SYSMON rewriting)
 - `rules_windows_sysmon.json` : Full SIGMA "**Windows**" ruleset (SYSMON)
+
+## Why you should make your own rulesets
+
+The default rulesets provided are the conversion of the rules located in `rules/windows` directory of the Sigma repository. You should take into account that : 
+
+- **Some rules are very noisy or produce a lot of false positives** depending on your environnement or the config file you used with genRules
+- **Some rules can be very slow** depending on your logs
+
+For example : 
+
+-  "Suspicious Eventlog Clear or Configuration Using Wevtutil" : **very noisy** on fresh environnement (labs etc.), commonly generate a lot of useless detection
+-  Notepad Making Network Connection : **can slow very significantly** the execution of Zircolite

tools/genRules/genRules.py

@@ -68,6 +68,7 @@ def retrieveRule(ruleFile):
         pool.close() 
         pool.join()
 
+    outputList = list(filter(None, outputList)) # Remove empty rules
     with open(args.output, 'w') as f:
         if args.rule is not None:
             json.dump([outputList], f, indent=4)

zircolite.py

@@ -120,7 +120,7 @@ def extractEvtx(file, tmpDir, evtx_dumpBinary):
     except Exception as e:
         logging.error(f"{Fore.RED}   [-] {e}")
 
-def flattenJSON(file):
+def flattenJSON(file, timeAfter, timeBefore):
     """
         Flatten json object with nested keys into a single level.
         Returns the flattened json object
@@ -153,7 +153,6 @@ def flatten(x, name=''):
                     else:
                         # Removing all annoying character from field name
                         key = ''.join(e for e in name[:-1].split(".")[-1] if e.isalnum())
-                    #key = key.lower()
                     JSONLine[key] = value
                     # Generate the CREATE TABLE SQL statement
                     if key.lower() not in keyDict:
@@ -170,7 +169,13 @@ def flatten(x, name=''):
                 flatten(json.loads(line))
             except Exception as e:
                 logging.debug(f'JSON ERROR : {e}')
-            JSONOutput.append(JSONLine)
+            # Handle timestamp filters
+            if timeAfter != "1970-01-01T00:00:00" and timeBefore != "9999-12-12T23:59:59":
+                timestamp = time.strptime(JSONLine["SystemTime"].split(".")[0].replace("Z",""), '%Y-%m-%dT%H:%M:%S')
+                if timestamp > timeAfter and timestamp < timeBefore:
+                    JSONOutput.append(JSONLine)
+            else:
+                JSONOutput.append(JSONLine)
             JSONLine = {}
 
     return {"dbFields": fieldStmt, "dbValues": JSONOutput}
@@ -342,22 +347,15 @@ def saveDbToDisk(dbConnection, dbFilename):
 # MAIN()
 ################################################################
 if __name__ == '__main__':
-    print("""
-    ███████╗██╗██████╗  ██████╗ ██████╗ ██╗     ██╗████████╗███████╗
-    ╚══███╔╝██║██╔══██╗██╔════╝██╔═══██╗██║     ██║╚══██╔══╝██╔════╝
-      ███╔╝ ██║██████╔╝██║     ██║   ██║██║     ██║   ██║   █████╗
-     ███╔╝  ██║██╔══██╗██║     ██║   ██║██║     ██║   ██║   ██╔══╝
-    ███████╗██║██║  ██║╚██████╗╚██████╔╝███████╗██║   ██║   ███████╗
-    ╚══════╝╚═╝╚═╝  ╚═╝ ╚═════╝ ╚═════╝ ╚══════╝╚═╝   ╚═╝   ╚══════╝
-    """)
 
     # Init Args handling
     tmpDir = "tmp-" + ''.join(random.SystemRandom().choice(string.ascii_uppercase + string.digits) for _ in range(8))
     parser = argparse.ArgumentParser()
     parser.add_argument("-e", "--evtx", help="EVTX log file or directory where EVTX log files are stored in JSON or EVTX format", type=str, required=True)
-    parser.add_argument("-s", "--select", help="Only EVTX files containing the provided string will be used. If there is/are exclusion(s) (--avoid) they will be handled after selection", action='append', nargs='+')
-    parser.add_argument("-a", "--avoid", help="EVTX files containing the provided string will NOT be used", action='append', nargs='+')
+    parser.add_argument("-s", "--select", help="Only EVTX files containing the provided string will be used. If there is/are exclusion(s) ('--avoid') they will be handled after selection", action='append', nargs='+')
+    parser.add_argument("-a", "--avoid", help="EVTX files containing the provided string will NOT be used", nargs='+')
     parser.add_argument("-r", "--ruleset", help="JSON File containing SIGMA rules", type=str, required=True)
+    parser.add_argument("-R", "--rulefilter", help="Remove rule from ruleset, match is done on rule title. The easier is to provide a CRC32 (check your ruleset to find it)", action='append', nargs='*')
     parser.add_argument("-c", "--config", help="JSON File containing field mappings and exclusions", type=str, default="config/fieldMappings.json")
     parser.add_argument("-o", "--outfile", help="JSON file that will contains all detected events", type=str, default="detected_events.json")
     parser.add_argument("-f", "--fileext", help="EVTX file extension", type=str, default="evtx")
@@ -366,6 +364,8 @@ def saveDbToDisk(dbConnection, dbFilename):
     parser.add_argument("-d", "--dbfile", help="Save data as a SQLite Db to the specified file on disk", type=str)
     parser.add_argument("-l", "--logfile", help="Log file name", default="zircolite.log", type=str)
     parser.add_argument("-j", "--jsononly", help="If logs files are already in JSON lines format ('jsonl' in evtx_dump)", action="store_true")
+    parser.add_argument("-A", "--after", help="Zircolite will only work on events that happened after the provided timestamp (UTC). Format : 1970-01-01T00:00:00", type=str, default="1970-01-01T00:00:00")
+    parser.add_argument("-B", "--before", help="Zircolite will only work on events that happened before the provided timestamp (UTC). Format : 1970-01-01T00:00:00", type=str, default="9999-12-12T23:59:59")
     parser.add_argument("--remote", help="Forward results to a HTTP server, please provide the full address e.g http://address:port/uri (except for Splunk)", type=str)
     parser.add_argument("--token", help="Use this to provide Splunk HEC Token", type=str)
     parser.add_argument("--stream", help="By default event forwarding is done at the end, this option activate forwarding events when detected", action="store_true")
@@ -379,6 +379,19 @@ def saveDbToDisk(dbConnection, dbFilename):
 
     # Init logging
     consoleLogger = initLogger(args.debug, args.logfile)
+
+    logging.info("""
+    ███████╗██╗██████╗  ██████╗ ██████╗ ██╗     ██╗████████╗███████╗
+    ╚══███╔╝██║██╔══██╗██╔════╝██╔═══██╗██║     ██║╚══██╔══╝██╔════╝
+      ███╔╝ ██║██████╔╝██║     ██║   ██║██║     ██║   ██║   █████╗
+     ███╔╝  ██║██╔══██╗██║     ██║   ██║██║     ██║   ██║   ██╔══╝
+    ███████╗██║██║  ██║╚██████╗╚██████╔╝███████╗██║   ██║   ███████╗
+    ╚══════╝╚═╝╚═╝  ╚═╝ ╚═════╝ ╚═════╝ ╚══════╝╚═╝   ╚═╝   ╚══════╝
+    """)
+
+    # flatten array of "rulefilter" arguments
+    if args.rulefilter: args.rulefilter = [item for sublist in args.rulefilter for item in sublist]
+
     # Init Forwarding
     forwarder = eventForwarder(args.remote, args.token)
     if args.remote is not None: 
@@ -388,6 +401,12 @@ def saveDbToDisk(dbConnection, dbFilename):
 
     logging.info("[+] Checking prerequisites")
 
+    # Checking provided timestamps
+    try:
+        eventsAfter = time.strptime(args.after, '%Y-%m-%dT%H:%M:%S')
+        eventsBefore = time.strptime(args.before, '%Y-%m-%dT%H:%M:%S')
+    except:
+        quitOnError(f"{Fore.RED}   [-] Wrong timestamp format. Please use 'AAAA-MM-DDTHH:MM:SS'")
     # Cheking for evtx_dump binaries
     evtx_dumpBinary = getOSExternalTools()
     checkIfExists(evtx_dumpBinary, f"{Fore.RED}   [-] Cannot find Evtx_dump")
@@ -456,7 +475,7 @@ def saveDbToDisk(dbConnection, dbFilename):
         quitOnError(f"{Fore.RED}   [-] No JSON files found.")
     for evtxJSON in tqdm(EVTXJSONList, colour="yellow"):
         if os.stat(evtxJSON).st_size != 0:
-            results = flattenJSON(evtxJSON)
+            results = flattenJSON(evtxJSON, eventsAfter, eventsBefore)
             fieldStmt += results["dbFields"]
             valuesStmt += results["dbValues"]
 
@@ -484,6 +503,11 @@ def saveDbToDisk(dbConnection, dbFilename):
     logging.info(f"[+] Loading ruleset from : {args.ruleset}")
     with open(args.ruleset) as f:
         ruleset = json.load(f)
+    # Remove empty rule and remove filtered rules
+    ruleset = list(filter(None, ruleset))
+    if args.rulefilter is not None:
+        ruleset = [rule for rule in ruleset if not any(ruleFilter in rule["title"] for ruleFilter in args.rulefilter)]
+
     logging.info(f"[+] Executing ruleset - {len(ruleset)} rules")
     # Results are writen upon detection to allow analysis during execution and to avoid loosing results in case of error.
     fullResults = []
@@ -492,7 +516,7 @@ def saveDbToDisk(dbConnection, dbFilename):
             with tqdm(ruleset, colour="yellow") as ruleBar:
                 f.write('[')
                 for rule in ruleBar:  # for each rule in ruleset
-                    if args.showall and "title" in rule: ruleBar.write(f'{Fore.BLUE}    - {rule["title"]}')  # Print all rules
+                    if args.showall: ruleBar.write(f'{Fore.BLUE}    - {rule["title"]}')  # Print all rules
                     ruleResults = executeRule(rule)
                     if ruleResults != {}:
                         ruleBar.write(f'{Fore.CYAN}    - {ruleResults["title"]} : {ruleResults["count"]} events')
@@ -513,7 +537,7 @@ def saveDbToDisk(dbConnection, dbFilename):
         else:
             f.write('[')
             for rule in ruleset:
-                if args.showall and "title" in rule: ruleBar.write(f'{Fore.BLUE}    - {rule["title"]}')  # Print all rules
+                if args.showall: logging.info(f'{Fore.BLUE}    - {rule["title"]}')  # Print all rules
                 ruleResults = executeRule(rule)
                 if ruleResults != {}:
                     logging.info(f'{Fore.CYAN}    - {ruleResults["title"]} : {ruleResults["count"]} events')