Fix format in mongoDB bots #1322
Conversation
Actually date are saved in format string in mongoDB. This fix change the format for time.observation and time.source from string to ISODate.
| @@ -43,7 +44,10 @@ def process(self): | |||
| event = self.receive_message() | |||
|
|
|||
| try: | |||
| self.collection.insert(event.to_dict(hierarchical=self.parameters.hierarchical_output)) | |||
| tmp_dict = event.to_dict(hierarchical=self.parameters.hierarchical_output) | |||
| tmp_dict["time"]["observation"] = dateutil.parser.parse(tmp_dict["time"]["observation"]) | |||
There was a problem hiding this comment.
The existence of time.observation and time.source is assumed here but cannot be guaranteed. See also the failing test.
There was a problem hiding this comment.
It is much faster to not use the dateutil parser here because it needs to guess the format based on heuristics. As the format is known, you can use intelmq.lib.harmonization.DateTime.parse_utc_isoformat (I just changed this method from private to public in the maintenance branch)
There was a problem hiding this comment.
Thks for the tips, I'll change that asap
There was a problem hiding this comment.
sorry, no time this week. I'll do it next week
There was a problem hiding this comment.
unfortunately, using intelmq.lib.harmonization.DateTime.parse_utc_isoformat is not enough. The format returned is still a str. So pymongo insert a string instead of Isodate.
I don't see any way to avoid using dateutil
There was a problem hiding this comment.
Oh, ok. That method in the libs looks strange, I need to look at it later when I refactor that file.
|
I just thought about moving the logic to |
ghost
self-assigned this
ghost
added
bug
…ls#1322 Add parameters replacement_char for changing default dot separator when flatten the hierarchy. Fix certtools#1324
Codecov Report
@@ Coverage Diff @@
## maintenance #1322 +/- ##
===============================================
- Coverage 75.57% 75.48% -0.09%
===============================================
Files 273 273
Lines 12874 12892 +18
Branches 1750 1758 +8
===============================================
+ Hits 9729 9732 +3
- Misses 2748 2760 +12
- Partials 397 400 +3
|
|
Why is the replacement char now needed? |
|
I add the replacement_char for fixing the issue #1324 |
|
Ah, that was this one. I am now writing the missing changelog entries and I already forgot about this issue :) |
parameter replacement_char: handling if not given style issues reduce code duplication
Update to PR#1322
ghost
merged commit 
Actually date are saved in format string in mongoDB.
This fix change the format for time.observation and time.source from string to ISODate.