ENH: Fully droped Python 3.6. Updated syntax.

All references to Python 3.6 are updated or explicitly mentioned
as no longer supported. The syntax was updated to use newer features.

In addition, the changelog was fixed by removing already fixed known
issues and moving wrong-placed entry.

Closes certtools#2272

CHANGELOG.md

@@ -44,6 +44,7 @@ CHANGELOG
   - Removes `tzone` argument from `DateTime.from_timestamp` and `DateTime.from_epoch_millis`
   - `DateTime.from_timstamp` now also allows string argument
 - Removes `pytz` global dependency
+- Removed support for Python 3.6, including removing conditional dependencies and updating syntax to use features from newest versions. (fixes [#2272](https://github.com/certtools/intelmq/issues/2272)
 
 ### Development
 - Removed Python 3.6 from CI.
@@ -179,15 +180,14 @@ CHANGELOG
   - `check`: handle `SyntaxError` in bot modules and report it without breaking execution (fixes #2177)
   - Privilege drop before logfile creation (PR#2277 by Sebastian Waldbauer, fixes 2176)
 - `intelmqsetup`: Revised installation of manager by building the static files at setup, not build time, making it behave more meaningful. Requires intelmq-manager >= 3.1.0 (PR#2198 by Sebastian Wagner, fixes #2197).
+- `intelmqdump`: Respected global and per-bot custom settings of `logging_path` (fix #1605).
 
 ### Contrib
 - logrotate: Move compress and ownership rules to the IntelMQ-blocks to prevent that they apply to other files (PR#2111 by Sebastian Wagner, fixes #2110).
 
 ### Known issues
 This is short list of the most important known issues. The full list can be retrieved from [GitHub](https://github.com/certtools/intelmq/labels/bug?page=2&q=is%3Aopen+label%3Abug).
 - intelmq_psql_initdb does not work for SQLite (#2202).
-- SyntaxError in bots causes intelmqctl check to crash (#2177).
-- intelmqctl create log file before dropping privileges (#2176).
 - intelmqsetup: should install a default state file (#2175).
 - Misp Expert - Crash if misp event already exist (#2170).
 - Turris greylist has been updated (#2167).
@@ -197,7 +197,6 @@ This is short list of the most important known issues. The full list can be retr
 - intelmqctl log: parsing syslog does not work (#2097).
 - Bash completion scripts depend on old JSON-based configuration files (#2094).
 - Bot configuration examples use JSON instead of YAML (#2066).
-- intelmqdump: logging_path parameter not honoured (#1605).
 - Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952).
 - Corrupt dump files when interrupted during writing (#870).
 
@@ -759,7 +758,6 @@ IntelMQ no longer supports Python 3.5 (and thus Debian 9 and Ubuntu 16.04), the
 ### Tools
 - `intelmqdump`:
     - Check if given queue is configured upon recovery (#1433, PR#1587 by Mladen Markovic).
-    - Respected global and per-bot custom settings of `logging_path` (fix #1605).
 - `intelmqctl`:
   - `intelmq list queues`: `--sum`, `--count`, `-s` flag for showing total count of messages (#1408, PR#1581 by Mladen Markovic).
   - `intelmq check`: Added a possibility to ignore queues from the orphaned queues check (by Sebastian Wagner).

contrib/malware_name_mapping/download_mapping.py

@@ -180,7 +180,7 @@ def main(args):
                      )
     if args.filename:
         try:
-            with open(args.filename, 'wt') as output:
+            with open(args.filename, 'w') as output:
                 output.write(rules)
         except PermissionError:
             print('Could not write to file %r.' % args.filename, file=sys.stderr)

debian/control

@@ -23,7 +23,7 @@ Build-Depends: debhelper (>= 4.1.16),
                quilt,
                rsync,
                safe-rm
-X-Python3-Version: >= 3.5
+X-Python3-Version: >= 3.7
 Standards-Version: 3.9.6
 Homepage: https://github.com/certtools/intelmq/
 

docs/user/installation.rst

@@ -26,7 +26,7 @@ The following installation methods are available:
 Base Requirements
 -----------------
 
-The following instructions assume the following requirements. Python versions >= 3.6 are supported.
+The following instructions assume the following requirements. Python versions >= 3.7 are supported.
 
 Supported and recommended operating systems are:
 
@@ -124,6 +124,12 @@ CentOS 7 / RHEL 7:
    # optional dependencies
    yum install python3-psycopg2
 
+.. note::
+
+   We no longer support already end-of-life Python 3.6, which is the last Python version officially
+   packaged for CentOS Linux 7. You can either use alternative Python source, or stay on the IntelMQ
+   3.0.2.
+
 CentOS 8:
 
 .. code-block:: bash

intelmq/bin/intelmq_psql_initdb.py

@@ -89,7 +89,7 @@ def main():
                                                  text=True)
             fp = os.fdopen(os_fp, 'wt')
         else:
-            fp = open(OUTPUTFILE, 'wt')
+            fp = open(OUTPUTFILE, 'w')
         psql = generate()
         print("INFO - Writing %s file" % OUTPUTFILE)
         fp.write(psql)

intelmq/bin/intelmqdump.py

@@ -210,7 +210,7 @@ def main(argv=None):
     defaults = utils.get_global_settings()
     runtime_config = utils.get_runtime()
     pipeline_pipes = {}
-    logging_paths = set([defaults.get('logging_path', DEFAULT_LOGGING_PATH)])
+    logging_paths = {defaults.get('logging_path', DEFAULT_LOGGING_PATH)}
     for bot, settings in runtime_config.items():
         pipeline_pipes[settings.get('source_queue', f"{bot}-queue")] = bot
         logging_paths.add(settings.get("parameters", {}).get('logging_path', DEFAULT_LOGGING_PATH))

intelmq/bots/collectors/shadowserver/collector_reports_api.py

@@ -95,9 +95,9 @@ def _reports_list(self, date=None):
         dayafter = date + timedelta(1)
 
         data = self.preamble
-        data += ',"date": "{}:{}" '.format(daybefore.isoformat(), dayafter.isoformat())
+        data += f',"date": "{daybefore.isoformat()}:{dayafter.isoformat()}" '
         if len(self._report_list) > 0:
-            data += ',"reports": {}'.format(json.dumps(self._report_list))
+            data += f',"reports": {json.dumps(self._report_list)}'
         data += '}'
         self.logger.debug('Downloading report list with data: %s.', data)
 

intelmq/bots/outputs/file/output.py

@@ -39,7 +39,7 @@ def open_file(self, filename: str = None):
         if self._file is not None:
             self._file.close()
         try:
-            self._file = open(filename, mode='at', encoding='utf-8', errors=self.errors)
+            self._file = open(filename, mode='a', encoding='utf-8', errors=self.errors)
         except FileNotFoundError:  # directory does not exist
             path = Path(os.path.dirname(filename))
             try:
@@ -48,7 +48,7 @@ def open_file(self, filename: str = None):
                 self.logger.exception('Directory %r could not be created.', path)
                 self.stop()
             else:
-                self._file = open(filename, mode='at', encoding='utf-8', errors=self.errors)
+                self._file = open(filename, mode='a', encoding='utf-8', errors=self.errors)
 
     def process(self):
         event = self.receive_message()