Merge pull request #65 from wagga40/csv-delimiter

Add options : delimiter for CSV, stop recursion, file pattern

docs/Advanced.md

@@ -71,6 +71,7 @@ Except when `evtx_dump` is used, Zircolite only use one core. So if you have a l
 - `--dbfile <FILE>` allows you to export all the logs in a SQLite 3 database file. You can query the logs with SQL statements to find more things than what the Sigma rules could have found
 - `--keeptmp` allows you to keep the source logs (EVTX/Auditd/Evtxtract/XML...) converted in JSON format
 - `--keepflat` allow you to keep the source logs (EVTX/Auditd/Evtxtract/XML...) converted in a flattened JSON format
+
 ---
 
 ### Filtering
@@ -238,14 +239,14 @@ Zircolite is able to forward all events and not just the detected events to Splu
 
 ### Templating and Formatting
 
-Zircolite provides a templating system based on Jinja 2. It allows you to change the output format to suits your needs (Splunk or ELK integration, Grep-able output...). There are some templates available in the [Templates directory](../templates) of the repository : CSV, Splunk, Mini-GUI. To use the template system, use these arguments :
+Zircolite provides a templating system based on Jinja 2. It allows you to change the output format to suits your needs (Splunk or ELK integration, Grep-able output...). There are some templates available in the [Templates directory](../templates) of the repository : Splunk, Timesketch, ... To use the template system, use these arguments :
 
 - `--template <template_filename>`
 - `--templateOutput <output_filename>`
 
 ```shell
 python3 zircolite.py --evtx sample.evtx  --ruleset rules/rules_windows_sysmon.json \
---template templates/exportCSV.tmpl --templateOutput test.csv
+--template templates/exportForSplunk.tmpl --templateOutput exportForSplunk.json
 ```
 
 It is possible to use multiple templates if you provide for each `--template` argument there is a `--templateOutput` argument associated.
@@ -261,7 +262,7 @@ The Mini-GUI can be used totally offline, it allows the user to display and sear
 
 #### Automatic generation
 
-As of Zircolite 2.1.0, with the non-embedded versions, the easier way to use the Mini-GUI is to generate a package with the `--package` option. A zip file containing all the necessary data will be generated at the root of the repository.  
+As of Zircolite 2.1.0, the easier way to use the Mini-GUI is to generate a package with the `--package` option. A zip file containing all the necessary data will be generated at the root of the repository.  
 
 #### Manual generation
 

docs/Usage.md

@@ -45,11 +45,14 @@ python3 zircolite.py --evtx ../Logs --ruleset rules/rules_windows_sysmon.json
 
 It also works directly on an unique EVTX file.
 
+:information_source: `--evtx`, `--events` and `-e` are equivalent
+
 By default : 
 
 - `--ruleset` is not mandatory but the default ruleset will be `rules/rules_windows_generic.json`
 - Results are written in the `detected_events.json` in the same directory as Zircolite
 - There is a `zircolite.log`file that will be created in the current working directory
+- `Zircolite` will automatically choose a file extension, you can change it with `--fileext`. This option can be used with wildcards or [Python Glob syntax](https://docs.python.org/3/library/glob.html) but with `*.` added before the given parameter value : `*.<FILEEXT PARAMETER VALUE>`. For example `--fileext log` will search for `*.log` files in the given path and `--fileext log.*` will search for `*.log.*` which can be useful when handling linux log files (auditd.log.1...).
 
 #### XML logs : 
 
@@ -116,13 +119,11 @@ python3 zircolite.py --events sysmon.log --ruleset rules/rules_linux.json --sysm
 It is possible to use Zircolite directly on JSONL/NDJSON files (NXLog files) with the `--jsononly` or `-j` arguments : 
 
 ```shell
-python3 zircolite.py --events <EVTX_FOLDER> --ruleset <RULESET> --jsononly
+python3 zircolite.py --events <LOGS_FOLDER> --ruleset <RULESET> --jsononly
 ```
 
 A simple use case is when you have already run Zircolite and use the `--keeptmp` option. Since it keeps all the converted EVTX in a temp directory, if you need to re-execute Zircolite, you can do it directly using this directory as the EVTX source (with `--evtx <EVTX_IN_JSON_DIRECTORY>` and `--jsononly`) and avoid to convert the EVTX again.
 
-:information_source: You can change the file extension with `--fileext`.
-
 #### SQLite database files
 
 Since everything in Zircolite is stored in a in-memory SQlite database, you can choose to save the database on disk for later use. It is possible with the option `--dbfile <db_filename>`.

zircolite.py

@@ -700,6 +700,7 @@ def __init__(
         timeField=None,
         hashes=False,
         dbLocation=":memory:",
+        delimiter=";",
     ):
         self.logger = logger or logging.getLogger(__name__)
         self.dbConnection = self.createConnection(dbLocation)
@@ -713,6 +714,7 @@ def __init__(
         self.csvMode = csvMode
         self.timeField = timeField
         self.hashes = hashes
+        self.delimiter = delimiter
 
     def close(self):
         self.dbConnection.close()
@@ -973,7 +975,7 @@ def executeRuleset(
                                     ):  # Creating the CSV header and the fields ("agg" is for queries with aggregation)
                                         csvWriter = csv.DictWriter(
                                             fileHandle,
-                                            delimiter=";",
+                                            delimiter=self.delimiter,
                                             fieldnames=[
                                                 "rule_title",
                                                 "rule_description",
@@ -1580,7 +1582,24 @@ def avoidFiles(pathList, avoidFilesList):
         help="The output will be in CSV. You should note that in this mode empty fields will not be discarded from results",
         action="store_true",
     )
+    parser.add_argument(
+        "--csv-delimiter",
+        help="Choose the delimiter for CSV ouput",
+        type=str,
+        default=";",
+    )
     parser.add_argument("-f", "--fileext", help="Extension of the log files", type=str)
+    parser.add_argument(
+        "-fp",
+        "--file-pattern",
+        help="Use a Python Glob pattern to select files. This option only works with directories",
+        type=str,
+    )
+    parser.add_argument(
+        "--no-recursion",
+        help="By default Zircolite search recursively, by using this option only the provided directory will be used",
+        action="store_true",
+    )
     parser.add_argument(
         "-t",
         "--tmpdir",
@@ -1885,6 +1904,7 @@ def avoidFiles(pathList, avoidFilesList):
         timeField=args.timefield,
         hashes=args.hashes,
         dbLocation=args.ondiskdb,
+        delimiter=args.csv_delimiter,
     )
 
     # If we are not working directly with the db
@@ -1900,22 +1920,30 @@ def avoidFiles(pathList, avoidFilesList):
             else:
                 args.fileext = "evtx"
 
-        EVTXPath = Path(args.evtx)
-        if EVTXPath.is_dir():
-            # EVTX recursive search in given directory with given file extension
-            EVTXList = list(EVTXPath.rglob(f"*.{args.fileext}"))
-        elif EVTXPath.is_file():
-            EVTXList = [EVTXPath]
+        LogPath = Path(args.evtx)
+        if LogPath.is_dir():
+            # Log recursive search in given directory with given file extension or pattern
+            pattern = f"*.{args.fileext}"
+            # If a Glob pattern is provided
+            if args.file_pattern not in [None, ""]:
+                pattern = args.file_pattern
+            fnGlob = LogPath.rglob
+
+            if args.no_recursion:
+                fnGlob = LogPath.glob
+            LogList = list(fnGlob(pattern))
+        elif LogPath.is_file():
+            LogList = [LogPath]
         else:
             quitOnError(
                 f"{Fore.RED}   [-] Unable to find events from submitted path{Fore.RESET}"
             )
 
         # Applying file filters in this order : "select" than "avoid"
-        FileList = avoidFiles(selectFiles(EVTXList, args.select), args.avoid)
+        FileList = avoidFiles(selectFiles(LogList, args.select), args.avoid)
         if len(FileList) <= 0:
             quitOnError(
-                f"{Fore.RED}   [-] No file found. Please verify filters, the directory or the extension with '--fileext'{Fore.RESET}"
+                f"{Fore.RED}   [-] No file found. Please verify filters, directory or the extension with '--fileext' or '--file-pattern'{Fore.RESET}"
             )
 
         if not args.jsononly:
@@ -1938,19 +1966,19 @@ def avoidFiles(pathList, avoidFilesList):
             for evtx in tqdm(FileList, colour="yellow"):
                 extractor.run(evtx)
             # Set the path for the next step
-            EVTXJSONList = list(Path(extractor.tmpDir).rglob("*.json"))
+            LogJSONList = list(Path(extractor.tmpDir).rglob("*.json"))
         else:
-            EVTXJSONList = FileList
+            LogJSONList = FileList
 
         checkIfExists(
             args.config, f"{Fore.RED}   [-] Cannot find mapping file{Fore.RESET}"
         )
-        if EVTXJSONList == []:
+        if LogJSONList == []:
             quitOnError(f"{Fore.RED}   [-] No JSON files found.{Fore.RESET}")
 
         # Print field list and exit
         if args.fieldlist:
-            fields = zircoliteCore.run(EVTXJSONList, Insert2Db=False)
+            fields = zircoliteCore.run(LogJSONList, Insert2Db=False)
             zircoliteCore.close()
             if not args.jsononly and not args.keeptmp:
                 extractor.cleanup()
@@ -1963,10 +1991,10 @@ def avoidFiles(pathList, avoidFilesList):
         # Flatten and insert to Db
         if args.forwardall:
             zircoliteCore.run(
-                EVTXJSONList, saveToFile=args.keepflat, forwarder=forwarder
+                LogJSONList, saveToFile=args.keepflat, forwarder=forwarder
             )
         else:
-            zircoliteCore.run(EVTXJSONList, saveToFile=args.keepflat)
+            zircoliteCore.run(LogJSONList, saveToFile=args.keepflat)
         # Unload In memory DB to disk. Done here to allow debug in case of ruleset execution error
         if args.dbfile is not None:
             zircoliteCore.saveDbToDisk(args.dbfile)
@@ -2045,7 +2073,7 @@ def avoidFiles(pathList, avoidFilesList):
 
     # Remove files submitted for analysis
     if args.remove_events:
-        for EVTX in EVTXList:
+        for EVTX in LogList:
             try:
                 os.remove(EVTX)
             except OSError as e:

zircolite_dev.py

@@ -470,7 +470,7 @@ def runAll(self, EVTXJSONList):
 class zirCore:
     """ Load data into database and apply detection rules  """
 
-    def __init__(self, config, logger=None, noOutput=False, timeAfter="1970-01-01T00:00:00", timeBefore="9999-12-12T23:59:59", limit=-1, csvMode=False, timeField=None, hashes=False, dbLocation=":memory:"):
+    def __init__(self, config, logger=None, noOutput=False, timeAfter="1970-01-01T00:00:00", timeBefore="9999-12-12T23:59:59", limit=-1, csvMode=False, timeField=None, hashes=False, dbLocation=":memory:", delimiter=";"):
         self.logger = logger or logging.getLogger(__name__)
         self.dbConnection = self.createConnection(dbLocation)
         self.fullResults = []
@@ -483,6 +483,7 @@ def __init__(self, config, logger=None, noOutput=False, timeAfter="1970-01-01T00
         self.csvMode = csvMode
         self.timeField = timeField
         self.hashes = hashes
+        self.delimiter = delimiter
 
     def close(self):
         self.dbConnection.close()
@@ -679,7 +680,7 @@ def executeRuleset(self, outFile, writeMode='w', forwarder=None, showAll=False,
                                 # Output to json or csv file
                                 if self.csvMode: 
                                     if not csvWriter: # Creating the CSV header and the fields ("agg" is for queries with aggregation)
-                                        csvWriter = csv.DictWriter(fileHandle, delimiter=';', fieldnames=["rule_title", "rule_description", "rule_level", "rule_count", "agg"] + list(ruleResults["matches"][0].keys()))
+                                        csvWriter = csv.DictWriter(fileHandle, delimiter=self.delimiter, fieldnames=["rule_title", "rule_description", "rule_level", "rule_count", "agg"] + list(ruleResults["matches"][0].keys()))
                                         csvWriter.writeheader()
                                     for data in ruleResults["matches"]:
                                         dictCSV = { "rule_title": ruleResults["title"], "rule_description": ruleResults["description"], "rule_level": ruleResults["rule_level"], "rule_count": ruleResults["count"], **data}                                        
@@ -1071,7 +1072,10 @@ def avoidFiles(pathList, avoidFilesList):
     parser.add_argument("-c", "--config", help="JSON File containing field mappings and exclusions", type=str, default="config/fieldMappings.json")
     parser.add_argument("-o", "--outfile", help="File that will contains all detected events", type=str, default="detected_events.json")
     parser.add_argument("--csv", help="The output will be in CSV. You should note that in this mode empty fields will not be discarded from results", action='store_true')
+    parser.add_argument("--csv-delimiter", help="Choose the delimiter for CSV ouput", type=str, default=";")
     parser.add_argument("-f", "--fileext", help="Extension of the log files", type=str)
+    parser.add_argument("-fp", "--file-pattern", help="Use a Python Glob pattern to select files. This option only works with directories", type=str)
+    parser.add_argument("--no-recursion", help="By default Zircolite search recursively, by using this option only the provided directory will be used", action="store_true")
     parser.add_argument("-t", "--tmpdir", help="Temp directory that will contains events converted as JSON (parent directories must exist)", type=str)
     parser.add_argument("-k", "--keeptmp", help="Do not remove the temp directory containing events converted in JSON format", action='store_true')
     parser.add_argument("-K", "--keepflat", help="Save flattened events as JSON", action='store_true')
@@ -1189,7 +1193,7 @@ def avoidFiles(pathList, avoidFilesList):
     start_time = time.time()
 
     # Initialize zirCore
-    zircoliteCore = zirCore(args.config, logger=consoleLogger, noOutput=args.nolog, timeAfter=eventsAfter, timeBefore=eventsBefore, limit=args.limit, csvMode=args.csv, timeField=args.timefield, hashes=args.hashes, dbLocation=args.ondiskdb)
+    zircoliteCore = zirCore(args.config, logger=consoleLogger, noOutput=args.nolog, timeAfter=eventsAfter, timeBefore=eventsBefore, limit=args.limit, csvMode=args.csv, timeField=args.timefield, hashes=args.hashes, dbLocation=args.ondiskdb, delimiter=args.csv_delimiter)
 
     # If we are not working directly with the db
     if not args.dbonly:
@@ -1200,19 +1204,27 @@ def avoidFiles(pathList, avoidFilesList):
             elif args.xml: args.fileext = "xml"
             else: args.fileext = "evtx"
 
-        EVTXPath = Path(args.evtx)
-        if EVTXPath.is_dir():
-            # EVTX recursive search in given directory with given file extension
-            EVTXList = list(EVTXPath.rglob(f"*.{args.fileext}"))
-        elif EVTXPath.is_file():
-            EVTXList = [EVTXPath]
+        LogPath = Path(args.evtx)
+        if LogPath.is_dir():
+            # Log recursive search in given directory with given file extension or pattern
+            pattern = f"*.{args.fileext}"
+            # If a Glob pattern is provided
+            if args.file_pattern not in [None, ""]: 
+                pattern = args.file_pattern
+            fnGlob = LogPath.rglob
+
+            if args.no_recursion: 
+                fnGlob = LogPath.glob
+            LogList = list(fnGlob(pattern))
+        elif LogPath.is_file():
+            LogList = [LogPath]
         else:
             quitOnError(f"{Fore.RED}   [-] Unable to find events from submitted path{Fore.RESET}")
 
         # Applying file filters in this order : "select" than "avoid"
-        FileList = avoidFiles(selectFiles(EVTXList, args.select), args.avoid)
+        FileList = avoidFiles(selectFiles(LogList, args.select), args.avoid)
         if len(FileList) <= 0:
-            quitOnError(f"{Fore.RED}   [-] No file found. Please verify filters, the directory or the extension with '--fileext'{Fore.RESET}")
+            quitOnError(f"{Fore.RED}   [-] No file found. Please verify filters, directory or the extension with '--fileext' or '--file-pattern'{Fore.RESET}")
 
         if not args.jsononly:
             # Init EVTX extractor object
@@ -1221,27 +1233,27 @@ def avoidFiles(pathList, avoidFilesList):
             for evtx in tqdm(FileList, colour="yellow"):
                 extractor.run(evtx)
             # Set the path for the next step
-            EVTXJSONList = list(Path(extractor.tmpDir).rglob("*.json"))
+            LogJSONList = list(Path(extractor.tmpDir).rglob("*.json"))
         else:
-            EVTXJSONList = FileList
+            LogJSONList = FileList
 
         checkIfExists(args.config, f"{Fore.RED}   [-] Cannot find mapping file{Fore.RESET}")
-        if EVTXJSONList == []:
+        if LogJSONList == []:
             quitOnError(f"{Fore.RED}   [-] No JSON files found.{Fore.RESET}")
 
         # Print field list and exit
         if args.fieldlist:
-            fields = zircoliteCore.run(EVTXJSONList, Insert2Db=False)
+            fields = zircoliteCore.run(LogJSONList, Insert2Db=False)
             zircoliteCore.close()
             if not args.jsononly and not args.keeptmp: extractor.cleanup()
             [print(sortedField) for sortedField in sorted([field for field in fields.values()])]
             sys.exit(0)
 
         # Flatten and insert to Db
         if args.forwardall:
-            zircoliteCore.run(EVTXJSONList, saveToFile=args.keepflat, forwarder=forwarder)
+            zircoliteCore.run(LogJSONList, saveToFile=args.keepflat, forwarder=forwarder)
         else:
-            zircoliteCore.run(EVTXJSONList, saveToFile=args.keepflat)
+            zircoliteCore.run(LogJSONList, saveToFile=args.keepflat)
         # Unload In memory DB to disk. Done here to allow debug in case of ruleset execution error
         if args.dbfile is not None: zircoliteCore.saveDbToDisk(args.dbfile)
     else:
@@ -1289,7 +1301,7 @@ def avoidFiles(pathList, avoidFilesList):
 
     # Remove files submitted for analysis
     if args.remove_events:
-        for EVTX in EVTXList:
+        for EVTX in LogList:
             try:
                 os.remove(EVTX)
             except OSError as e: