intelmq.lib.bot.Bot.set_request_parameters doesn't honor defaults.conf

[gethvi]

The expected order of configuration:

1. bot parameters from runtime.conf if present
2. default values from defaults.confif present
3. default values specified by the documentation or fail

This function sets it's own default values if bot parameters from runtime.conf are not present and doesn't use defaults.conf values.

[ghost]

The method uses self.parameters. In Bot.__init__, self.__load_defaults_configuration() is called, and later self.__load_runtime_configuration() overwriting the default's values. set_request_parameters then has fallback values, in case there's no value for a certain parameter given.

[gethvi]

@wagner-certat Thank you for the explanation. However it still feel unintuitive. I would like to suggest having a dedicated method utils.load_defaults_configuration(check: bool = True, path: str = DEFAULTS_CONF_FILE)
that loads defaults.conf, validates all the keys and if there is a key missing then it is either replaced by some "superdefault" value (e.g. mail_ssl: true ) or an exception is thrown.
Basically something like the part of intelmqctl.check that checks defaults.conf, but as a separate method and more fault proof.
With such approach there wouldn't be a need for any fallback values anywhere else ever again. Everything checked in one place.

[ghost]

I don't think we can get rid of internal fallback defaults. Just think of the case that the configurations files are "unreachable" and the tools need to abort in a proper way including logging.

But: handling of internal fall-back parameters, the default configuration and higher levels of parameters (bot-level default values and runtime configuration) can / should be improved.
See #632, #644, #757, #1505, #1580

[ghost]

create_request_session in utils uses the global parameters:

intelmq/intelmq/lib/utils.py

Line 768 in e489bee

defaults = get_global_settings()

And the HTTPMixin as well of course: https://github.com/certtools/intelmq/blob/develop/intelmq/lib/mixins/http.py

-> Solved