Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CSV Parser not treating ignored values correctly leading to exception #1651

Closed
ghost opened this issue Nov 6, 2020 · 0 comments · Fixed by #1674
Closed

CSV Parser not treating ignored values correctly leading to exception #1651

ghost opened this issue Nov 6, 2020 · 0 comments · Fixed by #1674
Assignees
Labels
bug Indicates an unexpected problem or unintended behavior component: bots
Milestone

Comments

@ghost
Copy link

ghost commented Nov 6, 2020

If the generic csv parser encounters a - as value in a column, the data is rejected by:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/intelmq/lib/bot.py", line 978, in process
    events = list(filter(bool, value))
  File "/usr/lib/python3/dist-packages/intelmq/bots/parsers/generic/parser_csv.py", line 130, in parse_line
    raise InvalidValue(key, value)
intelmq.lib.exceptions.InvalidValue: invalid value '-' (<class 'str'>) for key 'extra.cert_orgunit'

Example data is as simple as:

someheader
-

The reason is in these lines of code:

if event.add(key, value, raise_failure=False):
break

If Message.add gets a value to be ignored (like -), the return value is None, which evaluates to False. I think changing the if-clause to ... is not False is enough.

Please add a testcase for this.

@ghost ghost added bug Indicates an unexpected problem or unintended behavior component: bots labels Nov 6, 2020
@ghost ghost added this to the 2.2.3 milestone Nov 6, 2020
@ghost ghost assigned waldbauer-certat Nov 6, 2020
ghost pushed a commit that referenced this issue Dec 9, 2020
Currently CsvParser handles "-" values incorrectly, throwing
an error, which was not intended.

Signed-off-by: Sebastian Waldbauer <waldbauer@cert.at>

Fixes #1651
@ghost ghost closed this as completed in #1674 Dec 9, 2020
@ghost ghost closed this as completed in 57c430e Dec 9, 2020
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Indicates an unexpected problem or unintended behavior component: bots
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant