Add field alias and field splitting (Hash/hashes in Sysmon)

[wagga40]

This PR add field alias and field splitting.

Field mappings, field exclusions, value exclusions, field aliases and field splitting

Sometimes your logs need some transformations to allow your rules to match against them. Zircolite has multiple mechanisms for this. The configuration of these mechanisms is provided by a file that can be found in the config directory of the repository. It is also possible to provide your own configuration woth the --config or -c options.

The configuration file has the following structure :

{
	"exclusions" : [],
	"useless" : [],
	"mappings" : 
	{
		"field_name_1": "new_field_name_1", 
		"field_name_2": "new_field_name_2"
	},
	"alias":
	{
		"field_alias_1": "alias_1"
	},
	"split":
	{
		"field_name_split": {"separator":",", "equal":"="}
	}
}

Field mappings

field mappings allow you to rename a field from your raw logs (the ones that you want to analyze with Zircolite). Zircolite already uses this mechanism to rename nested JSON fields. You can check all the builtin field mappings here.

For example, if you want to rename the field "CommandLine" in your raw logs to "cmdline", you can add the following in the here file :

{
	"exclusions" : [],
	"useless" : [],
	"mappings" : 
	{
		"CommandLine": "cmdline"
	},
	"alias":{},
	"split": {}
}

Please keep in mind that as opposed to field alias, the original field name is not kept.

Field exclusions

field exclusions allow you to exclude a field. Zircolite already uses this mechanism to exclude the xlmns field. You can check all the builtin field exclusions here.

Value exclusions

value exclusions allow you to remove field which value is to be excluded. Zircolite already uses this mechanism to remove null and empty values. You can check all the builtin value exclusions here.

Field aliases

field aliases allow you to have multiple fields with different name but the same value. It is pretty similar to field mapping but you keep the original value. Field aliases can be used on original field names but also on mapped field names and splitted fields.

Let's say you have this event log in JSON format (the event has been deliberately truncated):

{
    "EventID": 1,
    "Provider_Name": "Microsoft-Windows-Sysmon",
    "Channel": "Microsoft-Windows-Sysmon/Operational",
    "CommandLine": "\"C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe\"",
    "Image": "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe",
    "IntegrityLevel": "Medium",
}

Let's say you are not sure all your rules use the "CommandLine" field but you remember that some of them use the "cmdline" field. To avoid any problems you could use an alias for the "CommandLine" field like this :

{
	"exclusions" : [],
	"useless" : [],
	"mappings" : {},
	"alias":{
		"CommandLine": "cmdline"
	},
	"split": {}
}

With this configuration, the event log used to apply Sigma rules will look like this :

{
	"EventID": 1,
	"Provider_Name": "Microsoft-Windows-Sysmon",
	"Channel": "Microsoft-Windows-Sysmon/Operational",
	"CommandLine": "\"C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe\"",
	"cmdline": "\"C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe\"",
	"Image": "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe",
	"IntegrityLevel": "Medium",
}

Be careful when using aliases because the data is stored multiple times.

Field splitting

field aliases allow you to split fields that contain key,value pairs. Zircolite already uses this mechanism to handle hash/hashes fields in Sysmon logs. You can check all the builtin field splittings here. Moreover, Field aliases can be applied to splitted fields.

For example, let's say we have this Sysmon event log :

	{
      "Hashes": "SHA1=XX,MD5=X,SHA256=XXX,IMPHASH=XXXX",
      "EventID": 1
	}

With the following configuration, Zircolite will split the hashes field like this :

{
	"exclusions" : [],
	"useless" : [],
	"mappings" : {},
	"alias":{},
	"split": {
		"Hashes": {"separator":",", "equal":"="}
	}
}

The final event log used to apply Sigma rules will look like this :

{
      "SHA1": "F43D9BB316E30AE1A3494AC5B0624F6BEA1BF054",
      "MD5": "04029E121A0CFA5991749937DD22A1D9",
      "SHA256": "9F914D42706FE215501044ACD85A32D58AAEF1419D404FDDFA5D3B48F66CCD9F",
      "IMPHASH": "7C955A0ABC747F57CCC4324480737EF7",
      "Hashes": "SHA1=F43D9BB316E30AE1A3494AC5B0624F6BEA1BF054,MD5=04029E121A0CFA5991749937DD22A1D9,SHA256=9F914D42706FE215501044ACD85A32D58AAEF1419D404FDDFA5D3B48F66CCD9F,IMPHASH=7C955A0ABC747F57CCC4324480737EF7",
      "EventID": 1
}