Issue with PreFixRegex in Generic detector

[RuchitaKshirsagarTR]

Description

We have found a second issue with TruffleHog’s generic detector and would like to share with you our analysis to discuss the logic behind it and what would be the solutions possible.

As per our discussion on slack regarding the generic detectors, the generic detector wasn’t detecting patterns it should have detected, such as api_key=9e107d9d372bb6826bd81d3542a419d6

Because it was missing this pattern, we wanted to fix it, and removed the following regex expressions from generic.go in PR #1343:

• \b[A-Fa-f0-9]{32}\b
• \b[A-Fa-f0-9x]{6,99}\b

Now, because of the removal of second regex, tests for the generic detector are failing and we are getting following error while running the test from generic_test.go

--- FAIL: TestGeneric_FromChunk (342.65s)
generic_test.go:43: Generic.FromData() expected no secret for data: secret_guid=3fc0b7f7-da09-4ae7-a9c8-d69824b1819b

Expected Behavior

However, the secret_guid should not have been flagged there is an excludePattern mentioned in generic.go file that should exclude any UUID. But upon debugging the code we found that the FromData function of line 66, stores the data string in two indices, the zeroth index is as it is, and first index has the second half of the data string. On line 73 it copies first index string to token variable and on line 83 it is matching against the exclude pattern regexes. Therefore, by this logic it would never match the value with UUID regex.

It seems that the issue is caused by how the grouping for the regex is constructed when the PrefixRegex function is called. Which means that this problem, may cause issues, outside of the generic detector, where PrefixRegex is used.

Actual Behavior

To further investigate, we tried to print the match array for other strings as well, and found that it is successfully taking out the “secret” values from the given data string but in case of guid it is getting only the secong half the “secret” value.

To add further, previously when \b[A-Fa-f0-9x]{6,99}\b expression was causing misses, it used to match following portions of the guid, which does not seem to be correct.

TruffleHog Version

latest

[zricethezav]

The generic detector is not enabled by default. Any generic detection should be done using a custom detector