Custom regex parallel verify #1127
Conversation
| @@ -82,66 +83,98 @@ func (c *customRegexWebhook) FromData(ctx context.Context, verify bool, data []b | |||
| // ] | |||
| matches := permutateMatches(regexMatches) | |||
|
|
|||
| g := new(errgroup.Group) | |||
There was a problem hiding this comment.
question: Are we okay not setting an upper bound on the number of goroutines here? Is the idea the len of matches will be low enough for this not to cause issues?
If we did plan on bounding the number of goroutines we can add:
g.SetLimit(<some-limit>)
There was a problem hiding this comment.
I was thinking unlimited would be okay because it's at most 100 goroutines that will be doing "async I/O" work.
| if err := g.Wait(); err != nil { | ||
| return nil, err | ||
| } | ||
| close(resultsCh) |
There was a problem hiding this comment.
suggestion: I think we should move this up right after you created the chan and defer the close. Otherwise if we encounter an err in the goroutines w/in the errgroup we are going to leak with this unclosed channel.
so just: defer close(resultsCh) at L90
There was a problem hiding this comment.
aaah! yes that's way cleaner!!!
There was a problem hiding this comment.
Unfortunately we need to close the channel before iterating over it otherwise the loop on line 103 would never end.
I do think that maybe we can ignore the error returned by g.Wait() though, and collect as many results that we got.
Bump trufflesecurity/trufflehog from 3.28.2 to 3.28.7. v3.28.7 What's Changed Support filtering detectors by version by @mcastorina in trufflesecurity/trufflehog#1150 Full Changelog: trufflesecurity/trufflehog@v3.28.6...v3.28.7 v3.28.6 What's Changed Rename .pre-commit-hooks.yml to .pre-commit-hooks.yaml by @zhuwenxing in trufflesecurity/trufflehog#1141 Keyword optimization by @zricethezav in trufflesecurity/trufflehog#1144 Release should only run on tags by @dustin-decker in trufflesecurity/trufflehog#1146 New Contributors @zhuwenxing made their first contribution in trufflesecurity/trufflehog#1141 @zricethezav made their first contribution in trufflesecurity/trufflehog#1144 Full Changelog: trufflesecurity/trufflehog@v3.28.5...v3.28.6 v3.28.5 What's Changed [chore] - Only scanned staged git changes by @ahrav in trufflesecurity/trufflehog#1143 Full Changelog: trufflesecurity/trufflehog@v3.28.4...v3.28.5 v3.28.4 What's Changed [chore] Address more linter errors by @mcastorina in trufflesecurity/trufflehog#1134 Custom regex parallel verify by @0x1 in trufflesecurity/trufflehog#1127 [chore] Close response bodies by @mcastorina in trufflesecurity/trufflehog#1137 Bump github.com/stretchr/testify from 1.8.1 to 1.8.2 by @dependabot in trufflesecurity/trufflehog#1130 Add pre-commit yml config by @ahrav in trufflesecurity/trufflehog#1138 Disable profiler in debug mode and add profile switch by @yilmi in trufflesecurity/trufflehog#1136 Full Changelog: trufflesecurity/trufflehog@v3.28.3...v3.28.4 v3.28.3 What's Changed Support file scanning in filesystem source by @mcastorina in trufflesecurity/trufflehog#1030 Add ability to include and exclude detectors by @mcastorina in trufflesecurity/trufflehog#1106 [chore] Implement String for ScanErrors by @mcastorina in trufflesecurity/trufflehog#1131 [chore] Update docs for individual file scanning by @mcastorina in trufflesecurity/trufflehog#1132 [chore] Address lint errors by @mcastorina in trufflesecurity/trufflehog#1133
No description provided.