Elastic update for 7+ #1513
Merged
Elastic update for 7+ #1513
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## develop #1513 +/- ##
==========================================
+ Coverage 66.97% 67.5% +0.52%
==========================================
Files 375 375
Lines 17845 18100 +255
Branches 2440 2443 +3
==========================================
+ Hits 11952 12218 +266
+ Misses 5438 5434 -4
+ Partials 455 448 -7
|
ghost
added this to the 
ghost
self-requested a review
|
The failing test is a codestyle test related complaining about a long line in the cymru expert - this line is a url in a comment. |
I will look into it |
On travis I only see a fail in the ripe expert tests which is working on the develop branch (just merge it in). Please give more information on the error you are seeing. |
|
Are you able to see my build output? For me the failing test is: |
|
Ok I've updated the branch and now see the ripe expert error. Will revisit this this weekend. |
|
I see the ripe error also on https://travis-ci.org/github/certtools/intelmq/jobs/669767267 |
|
Thank you very much for your efforts and your contribution. Sorry for letting you wait so long, unfortunately I was busy with other stuff these weeks. I hope I can form a new release soon. I made some additional documentation/maintenance changes in 9385ec5 |
CSIRT-CZ
pushed a commit
to CZ-NIC/intelmq
that referenced
this pull request
Jun 18, 2020
2.2.0 Feature release Dropped support for Python 3.4. ### Core - `__init__`: Changes to the path-handling, see [User Guide, section _/opt and LSB paths_](docs/User-Guide.md#opt-and-lsb-paths) for more information - The environment variable `INTELMQ_ROOT_DIR` can be used to set custom root directories instead of `/opt/intelmq/` (certtools#805) in case of non LSB-path installations. - The environment variable `ROOT_DIR` can be used to set custom root directories instead of `/` (certtools#805) in case of LSB-path installations. - `intelmq.lib.exceptions`: Added `MissingDependencyError` for show error messages about a missing library and how to install it (certtools#1471). - Added optional parameter `installed` to show the installed version. - Added optional parameter `additional_text` to show arbitrary text. - Adding more type annotations for core libraries. - `intelmq.lib.pipeline.Pythonlist.sleep`: Drop deprecated method. - `intelmq.lib.utils`: `write_configuration`: Append a newline at end of configuration/file to allow proper comparisons & diffs. - `intelmq.lib.test`: `BotTestCase` drops privileges upon initialization (certtools#1489). - `intelmq.lib.bot`: - New class `OutputBot`: - Method `export_event` to format/export events according to the parameters given by the user. - `ParserBot`: New methods `parse_json_stream` and `recover_line_json_stream`. - `ParserBot.recover_line_json`: Fix format by adding a list around the line data. - `Bot.send_message`: In debugging log level, the path to which the message is sent is now logged too. ### Bots - Bots with dependencies: Use of `intelmq.lib.exceptions.MissingDependencyError`. #### Collectors - `intelmq.bots.collectors.misp.collector`: Deprecate parameter `misp_verify` in favor of generic parameter `http_verify_cert`. - `intelmq.bots.collectors.tcp.collector`: Drop compatibility with Python 3.4. - `intelmq.bots.collectors.stomp.collector`: - Check the stomp.py version and show an error message if it does not match. - For stomp.py versions `>= 5.0.0` redirect the `stomp.PrintingListener` output to debug logging. - `intelmq.bots.collectors.microsoft.collector_azure`: Support current Python library `azure-storage-blob>= 12.0.0`, configuration is incompatible and needs manual change. See NEWS file and bot's documentation for more details. - `intelmq.bots.collectors.amqp.collector_amqp`: Require `pika` minimum version 1.0. - `intelmq.bots.collectors.github_api.collector_github_contents_api`: Added (PR#1481). #### Parsers - `intelmq.bots.parsers.autoshun.parser`: Drop compatibility with Python 3.4. - `intelmq.bots.parsers.html_table.parser`: Drop compatibility with Python 3.4. - `intelmq.bots.parsers.shadowserver.parser`: Add support for MQTT and Open-IPP feeds (PR#1512, PR#1544). - `intelmq.bots.parsers.taichung.parser`: - Migrate to `ParserBot`. - Also parse geolocation information if available. - `intelmq.bots.parsers.cymru.parser_full_bogons`: - Migrate to `ParserBot`. - Add last updated information in raw. - `intelmq.bots.parsers.anubisnetworks.parser`: Add new parameter `use_malware_familiy_as_classification_identifier`. - `intelmq.bots.parsers.microsoft.parser_ctip`: Compatibility for new CTIP data format used provided by the Azure interface. - `intelmq.bots.parsers.cymru.parser_cap_program`: Support for `openresolver` type. - `intelmq.bots.parsers.github_feed.parser`: Added (PR#1481). - `intelmq.bots.parsers.urlvir.parser`: Removed, as the feed is discontinued (certtools#1537). #### Experts - `intelmq.bots.experts.csv_converter`: Added as converter to CSV. - `intelmq.bots.experts.misp`: Added (PR#1475). - `intelmq.bots.experts.modify`: New parameter `maximum_matches`. #### Outputs - `intelmq.bots.outputs.amqptopic`: - Use `OutputBot` and `export_event`. - Allow formatting the routing key with event data by the new parameter `format_routing_key` (boolean). - `intelmq.bots.outputs.file`: Use `OutputBot` and `export_event`. - `intelmq.bots.outputs.files`: Use `OutputBot` and `export_event`. - `intelmq.bots.outputs.misp.output_feed`: Added, creates a MISP Feed (PR#1473). - `intelmq.bots.outputs.misp.output_api`: Added, pushes to MISP via the API (PR#1506, PR#1536). - `intelmq.bots.outputs.elasticsearch.output`: Dropped ElasticSearch version 5 compatibility, added version 7 compatibility (certtools#1513). ### Documentation - Document usage of the `INTELMQ_ROOT_DIR` environment variable. - Added document on MISP integration possibilities. - Feeds: - Added "Full Bogons IPv6" feed. - Remove discontinued URLVir Feeds (certtools#1537). ### Packaging - `setup.py` do not try to install any data to `/opt/intelmq/` as the behavior is inconsistent on various systems and with `intelmqsetup` we have a tool to create the structure and files anyway. - `debian/rules`: - Provide a blank state file in the package. - Patches: - Updated `fix-intelmq-paths.patch`. ### Tests - Travis: Use `intelmqsetup` here too. - Install required build dependencies for the Debian package build test. - This version is no longer automatically tested on Python `<` 3.5. - Also run the tests on Python 3.8. - Run the Debian packaging tests on Python 3.5 and the code-style test on 3.8. - Added tests for the new bot `intelmq.bots.outputs.misp.output_feed` (certtools#1473). - Added tests for the new bot `intelmq.bots.experts.misp.expert` (certtools#1473). - Added tests for `intelmq.lib.exceptions`. - Added tests for `intelmq.lib.bot.OutputBot` and `intelmq.lib.bot.OutputBot.export_event`. - Added IPv6 tests for `intelmq.bots.parsers.cymru.parser_full_bogons`. - Added tests for `intelmq.lib.bot.ParserBot`'s new methods `parse_json_stream` and `recover_line_json_stream`. - `intelmq.tests.test_conf`: Set encoding to UTF-8 for reading the `feeds.yaml` file. ### Tools - `intelmqctl`: - `upgrade-config`: - Allow setting the state file location with the `--state-file` parameter. - Do not require a second run anymore, if the state file is newly created (certtools#1491). - New parameter `no_backup`/`--no-backup` to skip creation of `.bak` files for state and configuration files. - Only require `psutil` for the `IntelMQProcessManager`, not for process manager independent calls like `upgrade-config` or `check`. - Add new command `debug` to output some information for debugging. Currently implemented: - paths - environment variables - `IntelMQController`: New argument `--no-file-logging` to disable logging to file. - If dropping privileges does not work, `intelmqctl` will now abort (certtools#1489). - `intelmqsetup`: - Add argument parsing and an option to skip setting file ownership, possibly not requiring root permissions. - Call `intelmqctl upgrade-config` and add argument for the state file path (certtools#1491). - `intelmq_generate_misp_objects_templates.py`: Tool to create a MISP object template (certtools#1470). - `intelmqdump`: New parameter `-t` or `--truncate` to optionally give the maximum length of `raw` data to show, 0 for no truncating. ### Contrib - Added `development-tools`. - ElasticSearch: Dropped version 5 compatibility, added version 7 compatibility (certtools#1513). - Malware Name Mapping Downloader: - New parameter `--mwnmp-ignore-adware`. - The parameter `--add-default` supports an optional parameter to define the default value. ### Known issues - Bots started with IntelMQ-Manager stop when the webserver is restarted. (certtools#952). - Corrupt dump files when interrupted during writing (certtools#870).
CSIRT-CZ
pushed a commit
to CZ-NIC/intelmq
that referenced
this pull request
Jun 22, 2020
2.2.0 Feature release Dropped support for Python 3.4. ### Core - `__init__`: Changes to the path-handling, see [User Guide, section _/opt and LSB paths_](docs/User-Guide.md#opt-and-lsb-paths) for more information - The environment variable `INTELMQ_ROOT_DIR` can be used to set custom root directories instead of `/opt/intelmq/` (certtools#805) in case of non LSB-path installations. - The environment variable `ROOT_DIR` can be used to set custom root directories instead of `/` (certtools#805) in case of LSB-path installations. - `intelmq.lib.exceptions`: Added `MissingDependencyError` for show error messages about a missing library and how to install it (certtools#1471). - Added optional parameter `installed` to show the installed version. - Added optional parameter `additional_text` to show arbitrary text. - Adding more type annotations for core libraries. - `intelmq.lib.pipeline.Pythonlist.sleep`: Drop deprecated method. - `intelmq.lib.utils`: `write_configuration`: Append a newline at end of configuration/file to allow proper comparisons & diffs. - `intelmq.lib.test`: `BotTestCase` drops privileges upon initialization (certtools#1489). - `intelmq.lib.bot`: - New class `OutputBot`: - Method `export_event` to format/export events according to the parameters given by the user. - `ParserBot`: New methods `parse_json_stream` and `recover_line_json_stream`. - `ParserBot.recover_line_json`: Fix format by adding a list around the line data. - `Bot.send_message`: In debugging log level, the path to which the message is sent is now logged too. ### Bots - Bots with dependencies: Use of `intelmq.lib.exceptions.MissingDependencyError`. #### Collectors - `intelmq.bots.collectors.misp.collector`: Deprecate parameter `misp_verify` in favor of generic parameter `http_verify_cert`. - `intelmq.bots.collectors.tcp.collector`: Drop compatibility with Python 3.4. - `intelmq.bots.collectors.stomp.collector`: - Check the stomp.py version and show an error message if it does not match. - For stomp.py versions `>= 5.0.0` redirect the `stomp.PrintingListener` output to debug logging. - `intelmq.bots.collectors.microsoft.collector_azure`: Support current Python library `azure-storage-blob>= 12.0.0`, configuration is incompatible and needs manual change. See NEWS file and bot's documentation for more details. - `intelmq.bots.collectors.amqp.collector_amqp`: Require `pika` minimum version 1.0. - `intelmq.bots.collectors.github_api.collector_github_contents_api`: Added (PR#1481). #### Parsers - `intelmq.bots.parsers.autoshun.parser`: Drop compatibility with Python 3.4. - `intelmq.bots.parsers.html_table.parser`: Drop compatibility with Python 3.4. - `intelmq.bots.parsers.shadowserver.parser`: Add support for MQTT and Open-IPP feeds (PR#1512, PR#1544). - `intelmq.bots.parsers.taichung.parser`: - Migrate to `ParserBot`. - Also parse geolocation information if available. - `intelmq.bots.parsers.cymru.parser_full_bogons`: - Migrate to `ParserBot`. - Add last updated information in raw. - `intelmq.bots.parsers.anubisnetworks.parser`: Add new parameter `use_malware_familiy_as_classification_identifier`. - `intelmq.bots.parsers.microsoft.parser_ctip`: Compatibility for new CTIP data format used provided by the Azure interface. - `intelmq.bots.parsers.cymru.parser_cap_program`: Support for `openresolver` type. - `intelmq.bots.parsers.github_feed.parser`: Added (PR#1481). - `intelmq.bots.parsers.urlvir.parser`: Removed, as the feed is discontinued (certtools#1537). #### Experts - `intelmq.bots.experts.csv_converter`: Added as converter to CSV. - `intelmq.bots.experts.misp`: Added (PR#1475). - `intelmq.bots.experts.modify`: New parameter `maximum_matches`. #### Outputs - `intelmq.bots.outputs.amqptopic`: - Use `OutputBot` and `export_event`. - Allow formatting the routing key with event data by the new parameter `format_routing_key` (boolean). - `intelmq.bots.outputs.file`: Use `OutputBot` and `export_event`. - `intelmq.bots.outputs.files`: Use `OutputBot` and `export_event`. - `intelmq.bots.outputs.misp.output_feed`: Added, creates a MISP Feed (PR#1473). - `intelmq.bots.outputs.misp.output_api`: Added, pushes to MISP via the API (PR#1506, PR#1536). - `intelmq.bots.outputs.elasticsearch.output`: Dropped ElasticSearch version 5 compatibility, added version 7 compatibility (certtools#1513). ### Documentation - Document usage of the `INTELMQ_ROOT_DIR` environment variable. - Added document on MISP integration possibilities. - Feeds: - Added "Full Bogons IPv6" feed. - Remove discontinued URLVir Feeds (certtools#1537). ### Packaging - `setup.py` do not try to install any data to `/opt/intelmq/` as the behavior is inconsistent on various systems and with `intelmqsetup` we have a tool to create the structure and files anyway. - `debian/rules`: - Provide a blank state file in the package. - Patches: - Updated `fix-intelmq-paths.patch`. ### Tests - Travis: Use `intelmqsetup` here too. - Install required build dependencies for the Debian package build test. - This version is no longer automatically tested on Python `<` 3.5. - Also run the tests on Python 3.8. - Run the Debian packaging tests on Python 3.5 and the code-style test on 3.8. - Added tests for the new bot `intelmq.bots.outputs.misp.output_feed` (certtools#1473). - Added tests for the new bot `intelmq.bots.experts.misp.expert` (certtools#1473). - Added tests for `intelmq.lib.exceptions`. - Added tests for `intelmq.lib.bot.OutputBot` and `intelmq.lib.bot.OutputBot.export_event`. - Added IPv6 tests for `intelmq.bots.parsers.cymru.parser_full_bogons`. - Added tests for `intelmq.lib.bot.ParserBot`'s new methods `parse_json_stream` and `recover_line_json_stream`. - `intelmq.tests.test_conf`: Set encoding to UTF-8 for reading the `feeds.yaml` file. ### Tools - `intelmqctl`: - `upgrade-config`: - Allow setting the state file location with the `--state-file` parameter. - Do not require a second run anymore, if the state file is newly created (certtools#1491). - New parameter `no_backup`/`--no-backup` to skip creation of `.bak` files for state and configuration files. - Only require `psutil` for the `IntelMQProcessManager`, not for process manager independent calls like `upgrade-config` or `check`. - Add new command `debug` to output some information for debugging. Currently implemented: - paths - environment variables - `IntelMQController`: New argument `--no-file-logging` to disable logging to file. - If dropping privileges does not work, `intelmqctl` will now abort (certtools#1489). - `intelmqsetup`: - Add argument parsing and an option to skip setting file ownership, possibly not requiring root permissions. - Call `intelmqctl upgrade-config` and add argument for the state file path (certtools#1491). - `intelmq_generate_misp_objects_templates.py`: Tool to create a MISP object template (certtools#1470). - `intelmqdump`: New parameter `-t` or `--truncate` to optionally give the maximum length of `raw` data to show, 0 for no truncating. ### Contrib - Added `development-tools`. - ElasticSearch: Dropped version 5 compatibility, added version 7 compatibility (certtools#1513). - Malware Name Mapping Downloader: - New parameter `--mwnmp-ignore-adware`. - The parameter `--add-default` supports an optional parameter to define the default value. ### Known issues - Bots started with IntelMQ-Manager stop when the webserver is restarted. (certtools#952). - Corrupt dump files when interrupted during writing (certtools#870).
This pull request was closed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Towards #1479