Upgrade to Connexion3 #702

juhoinkinen · 2023-05-10T14:49:35Z

Upgrades to Connexion 3, closes #689 and #698.

Connexion 3 introduces many changes compared to Connexion 2, see the documentation here.

Most importantly, after the upgrade to Connexion 3, running Annif with Gunicorn requires the use Uvicorn workers; the workers can be set using the --worker-class/-k option of Gunicorn:

gunicorn --worker-class uvicorn.workers.UvicornWorker "annif:create_app()"

In Dockerfile there is the enviroment variable GUNICORN_CMD_ARGS="--worker-class uvicorn.workers.UvicornWorker", which gives Gunicorn the option by default, so Docker image users do not have to worry about this change.

The original PR discussion below beginning from first draft.

This seems to be surprisingly difficult. Connexion 3 documentation states that there are two modes of running/usage of Connexion:

Using stand-alone Connexion - use the provided Flask app via connexion.FlaskApp
Using Connexion with existing ASGI or WSGI frameworks - use the "middleware" connexion.ConnexionMiddleware to wrap an existing Flask app

Previously Annif has been using the mode 1, and tried to continue with that. But now

Connexion 3.0 needs to be run using an ASGI server instead of a WSGI server. While any ASGI server should work, connexion comes with uvicorn as an extra

And the app needs to be started with

 uvicorn --factory annif:create_app

The command

 annif run

does not know about uvicorn but apparently tries to use plain Flask, and the app won't start.

There are also smaller breaking changes, that needed to be addressed:

Set content-types in response headers in rest methods, because
- "We no longer guess a content type for response serialization if multiple are defined in the spec. We do take into account returned headers." (The Annif OpenAPI spec included both application/json and application/problem+json, which seemed to be the cause.)
- "Content type is now validated for requests and responses if defined in the spec"
Changing req.get_json() -> req.json() in tests, apparently because of "connexion.request is now a Starlette Request instead of a Flask Request".

sonarcloud · 2023-05-10T14:52:02Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
4 Code Smells

No Coverage information
0.0% Duplication

codecov · 2023-05-10T14:55:40Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.64%. Comparing base (2e5e987) to head (e1e5d5a).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #702      +/-   ##
==========================================
- Coverage   99.65%   99.64%   -0.02%     
==========================================
  Files          89       89              
  Lines        6423     6443      +20     
==========================================
+ Hits         6401     6420      +19     
- Misses         22       23       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

tests/test_openapi.py

…o upgrade-to-connexion3

juhoinkinen · 2023-12-21T14:03:07Z

I added added annif run CLI command for starting uvicorn dev server as a custom Click command and removed the default run ,shell and routes Flask commands. The cli object used by annif command is still an instance of FlaskGroup, but maybe that could be turned into a custom Click command group. I don't know if this would then uncouple Flask fully from Annif.

Note that the default port for uvicorn is 8000, whereas for Flask it is 5000.

Also the tests/test_openapi.py do not pass due to an error:

AttributeError: module 'anyio' has no attribute 'start_blocking_portal'

but this seems to be some version mismatch of starlette.

And some tests/test_cli.py need to be adjusted.

All in all, maybe there could be an alternative to Connexion, which could allow still using Flask.

sonarcloud · 2024-01-09T08:06:08Z

Quality Gate passed

The SonarCloud Quality Gate passed, but some issues were introduced.

7 New issues
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

juhoinkinen · 2024-01-12T15:10:24Z

Also the tests/test_openapi.py do not pass due to an error:
AttributeError: module 'anyio' has no attribute 'start_blocking_portal'
but this seems to be some version mismatch of starlette.

Starlette-testclient has not pinned (or declared at all?) its anyio dependency, and anyio 4.* do not work with it: Kludex/starlette-testclient#5

Schemathesis v3.2.3 takes this into account and requires anyio <4, but for some reason anyio v4.2.0 gets installed.

A workaround is to pin anyio<4.* as a direct dependency.

juhoinkinen · 2024-01-22T12:04:14Z

When setting CORS in annif/__init__.py with

cxapp.add_middleware(
     CORSMiddleware,
    position=MiddlewarePosition.BEFORE_EXCEPTION,
    allow_origins=["*"],
    allow_methods=["*"],
)

CORS is not working for unit tests:

tests/test_openapi.py:43: AssertionError
...
FAILED tests/test_openapi.py::test_openapi_fuzzy[GET /v1/] - KeyError: 'access-control-allow-origin'
FAILED tests/test_openapi.py::test_openapi_fuzzy[GET /v1/projects] - KeyError: 'access-control-allow-origin'
FAILED tests/test_openapi.py::test_openapi_fuzzy[GET /v1/projects/{project_id}] - KeyError: 'access-control-allow-origin'
FAILED tests/test_openapi.py::test_openapi_fuzzy[POST /v1/projects/{project_id}/suggest] - KeyError: 'access-control-allow-origin'
FAILED tests/test_openapi.py::test_openapi_fuzzy[POST /v1/projects/{project_id}/suggest-batch] - KeyError: 'access-control-allow-origin'
FAILED tests/test_openapi.py::test_openapi_fuzzy[POST /v1/projects/{project_id}/learn] - KeyError: 'access-control-allow-origin'
FAILED tests/test_openapi.py::test_openapi_list_projects - assert 0

However, when looking at the response headers in the Firefox network tab, the 'access-control-allow-origin' header is there (at least for /suggest request), so there might be an issue due to starlette-testclient.

juhoinkinen · 2024-02-22T11:22:40Z

Also the tests/test_openapi.py do not pass due to an error:
AttributeError: module 'anyio' has no attribute 'start_blocking_portal'
but this seems to be some version mismatch of starlette.
Starlette-testclient has not pinned (or declared at all?) its anyio dependency, and anyio 4.* do not work with it: Kludex/starlette-testclient#5

Schemathesis v3.2.3 takes this into account and requires anyio <4, but for some reason anyio v4.2.0 gets installed.

A workaround is to pin anyio<4.* as a direct dependency.

Anyio pinning is not needed anymore, when doing these updates

jsonschema 4.17.* -> 4.21.*
pytest 7.* -> 8.*

juhoinkinen · 2024-02-22T11:45:42Z

When setting CORS in annif/__init__.py with

cxapp.add_middleware(
     CORSMiddleware,
    position=MiddlewarePosition.BEFORE_EXCEPTION,
    allow_origins=["*"],
    allow_methods=["*"],
)

CORS is not working for unit tests:

tests/test_openapi.py:43: AssertionError
...
FAILED tests/test_openapi.py::test_openapi_fuzzy[GET /v1/] - KeyError: 'access-control-allow-origin'
FAILED tests/test_openapi.py::test_openapi_fuzzy[GET /v1/projects] - KeyError: 'access-control-allow-origin'
FAILED tests/test_openapi.py::test_openapi_fuzzy[GET /v1/projects/{project_id}] - KeyError: 'access-control-allow-origin'
FAILED tests/test_openapi.py::test_openapi_fuzzy[POST /v1/projects/{project_id}/suggest] - KeyError: 'access-control-allow-origin'
FAILED tests/test_openapi.py::test_openapi_fuzzy[POST /v1/projects/{project_id}/suggest-batch] - KeyError: 'access-control-allow-origin'
FAILED tests/test_openapi.py::test_openapi_fuzzy[POST /v1/projects/{project_id}/learn] - KeyError: 'access-control-allow-origin'
FAILED tests/test_openapi.py::test_openapi_list_projects - assert 0

However, when looking at the response headers in the Firefox network tab, the 'access-control-allow-origin' header is there (at least for /suggest request), so there might be an issue due to starlette-testclient.

Also curl output shows the CORS header to be sent (when using H "Origin: http://example.com"):

curl -H "Origin: http://example.com" -X 'POST'   'http://127.0.0.1:8000/v1/projects/yake-fi/suggest'   -H 'accept: application/json'   -H 'Content-Type: application/x-www-form-urlencoded'   -d 'text=kissa&limit=10&threshold=0&language=' -v

juhoinkinen · 2024-04-04T14:10:45Z

When setting CORS in annif/__init__.py with

cxapp.add_middleware(
     CORSMiddleware,
    position=MiddlewarePosition.BEFORE_EXCEPTION,
    allow_origins=["*"],
    allow_methods=["*"],
)

CORS is not working for unit tests:

tests/test_openapi.py:43: AssertionError
...
FAILED tests/test_openapi.py::test_openapi_fuzzy[GET /v1/] - KeyError: 'access-control-allow-origin'
FAILED tests/test_openapi.py::test_openapi_fuzzy[GET /v1/projects] - KeyError: 'access-control-allow-origin'
FAILED tests/test_openapi.py::test_openapi_fuzzy[GET /v1/projects/{project_id}] - KeyError: 'access-control-allow-origin'
FAILED tests/test_openapi.py::test_openapi_fuzzy[POST /v1/projects/{project_id}/suggest] - KeyError: 'access-control-allow-origin'
FAILED tests/test_openapi.py::test_openapi_fuzzy[POST /v1/projects/{project_id}/suggest-batch] - KeyError: 'access-control-allow-origin'
FAILED tests/test_openapi.py::test_openapi_fuzzy[POST /v1/projects/{project_id}/learn] - KeyError: 'access-control-allow-origin'
FAILED tests/test_openapi.py::test_openapi_list_projects - assert 0

However, when looking at the response headers in the Firefox network tab, the 'access-control-allow-origin' header is there (at least for /suggest request), so there might be an issue due to starlette-testclient.

Also curl output shows the CORS header to be sent (when using H "Origin: http://example.com"):

curl -H "Origin: http://example.com" -X 'POST'   'http://127.0.0.1:8000/v1/projects/yake-fi/suggest'   -H 'accept: application/json'   -H 'Content-Type: application/x-www-form-urlencoded'   -d 'text=kissa&limit=10&threshold=0&language=' -v

The CORS testing problem was resolved by removing CORS check from the fuzzying test and (re)creating a dedicated test for it, which adds Origin header to the request:

app_client.headers = {"Origin": "http://somedomain.com"}

But now there is a (new?) problem: the test_openapi_fuzzy discovered that %0A, i.e. encoded newline, in the URL parameters crashes the application. This does not occur with Connexion 2.

Edit: The test cases with %0A in path parameter are now skipped over to keep tests working.

…mand

osma

I tested this PR and here are some findings:

CLI operation

The Flask-related commands routes and shell have been dropped. This is good because they're irrelevant for Annif anyway. But the -e, --env-file and -A, --app options that seem to be Flask-specific are still there - could we easily get rid of those too?
I timed a few commands (help, list-projects, suggest) and they don't seem to take longer to run than they used to - some are maybe even a little faster than before. Excellent!

`annif run` command

The console output now looks a bit different than it used to, due to the switch to uvicorn. But I think the new output looks good.
Annif now listens on port 8000, when it used to listen on port 5000. I think it would be better to keep the old port, because it's documented for example in the Annif tutorial Web UI and REST API exercises. Simply defining a default value of 5000 for --port should work, right?

Swagger-UI

The category view in /v1/ui now has a different order - "Learning from feedback" comes last. It's better this way!
Method parameters are also in a different order - not worse than before, maybe even a bit better. The layout for the "Send empty value" checkbox seems a bit broken, though I'm unsure if we can do anything about it. Here is an example that shows old (main) on the left, new (PR branch) on the right:

REST API

CORS now works differently than before. I'm not sure if this is bad, just something to note (maybe put in the release notes?).
- Before: If there's an Origin header in the request, send Access-Control-Allow-Origin: <Origin header value>, otherwise send Access-Control-Allow-Origin: *
- After: If there's an Origin header in the request, send Access-Control-Allow-Origin: *, otherwise send no such header.
The URL /projects/ used to give a 404 response, now it redirects to the correct URL /projects. Again, not necessarily a problem, just different, and perhaps worth putting into the release notes.
Successful JSON responses are pretty-printed, but error responses in Problem JSON are not. Previously both kinds of JSON responses were pretty-printed for easier readability. I think readable JSON would be good especially for the error messages, since they are likely to be seen by humans at some point.

Other

SonarCloud has a few complaints - are these real problems?
CodeClimate also complains of complexity, but it seems like the affected code was already complex, so not really a regression
Codecov complains about L25 in validation.py not being covered by tests, would it be easy to test this? Or add a notation that excludes it from test coverage?
See comment about Connexion version specifier

pyproject.toml

juhoinkinen · 2024-04-12T09:03:34Z

I tested this PR and here are some findings:

CLI operation

The Flask-related commands routes and shell have been dropped. This is good because they're irrelevant for Annif anyway. But the -e, --env-file and -A, --app options that seem to be Flask-specific are still there - could we easily get rid of those too?

Yes, done.

I timed a few commands (help, list-projects, suggest) and they don't seem to take longer to run than they used to - some are maybe even a little faster than before. Excellent!

annif run command

The console output now looks a bit different than it used to, due to the switch to uvicorn. But I think the new output looks good.

Annif now listens on port 8000, when it used to listen on port 5000. I think it would be better to keep the old port, because it's documented for example in the Annif tutorial Web UI and REST API exercises. Simply defining a default value of 5000 for --port should work, right?

Yes, done.

Swagger-UI

The category view in /v1/ui now has a different order - "Learning from feedback" comes last. It's better this way!

Method parameters are also in a different order - not worse than before, maybe even a bit better. The layout for the "Send empty value" checkbox seems a bit broken, though I'm unsure if we can do anything about it. Here is an example that shows old (main) on the left, new (PR branch) on the right:

REST API

CORS now works differently than before. I'm not sure if this is bad, just something to note (maybe put in the release notes?).

Before: If there's an Origin header in the request, send Access-Control-Allow-Origin: <Origin header value>, otherwise send Access-Control-Allow-Origin: *

After: If there's an Origin header in the request, send Access-Control-Allow-Origin: *, otherwise send no such header.

The URL /projects/ used to give a 404 response, now it redirects to the correct URL /projects. Again, not necessarily a problem, just different, and perhaps worth putting into the release notes.

~~TODO in Relase notes.~~
Edit: Added in Relase notes.

Successful JSON responses are pretty-printed, but error responses in Problem JSON are not. Previously both kinds of JSON responses were pretty-printed for easier readability. I think readable JSON would be good especially for the error messages, since they are likely to be seen by humans at some point.

Yes, but seems that this cannot be configured, see an old issue: spec-first/connexion#313

Other

SonarCloud has a few complaints - are these real problems?

I would say no, I silenced them.

CodeClimate also complains of complexity, but it seems like the affected code was already complex, so not really a regression

Yes, and the code in question is to override a function of Connexion, so it is not meaningful to simplify it. I silenced the complain as "wontfix".

Codecov complains about L25 in validation.py not being covered by tests, would it be easy to test this? Or add a notation that excludes it from test coverage?

I tried to add test for this line, but it is not detected by CodeCov. I added # pragma: no cover annotation for the line to exclude it.

This also reverts the previous commit fc45493. Now uvicorn workers are used always when running with Docker

osma

LVGTM

The test was added in ab33375 The condition is checked before this custom validation

sonarcloud · 2024-04-19T09:01:44Z

Quality Gate passed

Issues
0 New issues
1 Accepted issue

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

juhoinkinen and others added 6 commits May 10, 2023 11:36

Upgrade to Connexion 3 alpha-6

4b4e621

Make application/x-www-form-urlencoded request with axios

57fcaa9

Fix OpenAPI spec for /learn endpoint

f046c1f

Set content-types in response headers in rest methods

6d0a68f

Adapt tests for rest returning also status codes & content types

c315633

Adapt tests and fixtures for using Connexion app

e417e02

juhoinkinen added the maintenance label May 10, 2023

juhoinkinen added this to the 1.0 milestone May 10, 2023

juhoinkinen mentioned this pull request May 11, 2023

Optimize CLI startup time #696

Merged

Base automatically changed from lazy-imports to main May 11, 2023 11:28

juhoinkinen modified the milestones: 1.0, Short term Aug 15, 2023

juhoinkinen added 2 commits November 15, 2023 16:54

Upgrade to Connexion 3.0.* (from alpha); remove direct Flask dependency

9cd6a60

Merge branch 'main' into upgrade-to-connexion3

5d7ec95

github-advanced-security bot found potential problems Nov 15, 2023

View reviewed changes

tests/test_openapi.py Fixed Show fixed Hide fixed

juhoinkinen added 5 commits November 15, 2023 17:26

Drop using flask-cors (TODO use CORSMiddleware)

067983b

Drop using flask-cors (TODO use CORSMiddleware)

d485a3f

Merge branch 'main' into upgrade-to-connexion3

9ca8d9a

Add annif run CLI command for starting uvicorn dev server

3902748

Merge branch 'upgrade-to-connexion3' of github.com:NatLibFi/Annif int…

cb39aa0

…o upgrade-to-connexion3

juhoinkinen added 2 commits January 9, 2024 10:02

Require connexion version >= 3.0.5

aee036f

Merge branch 'main' into upgrade-to-connexion3

5aa1cf3

Re-enable CORS

beac55c

juhoinkinen mentioned this pull request Apr 5, 2024

RuntimeError by URL with encoded newline (%0A) parameter spec-first/connexion#1908

Open

juhoinkinen added 5 commits April 5, 2024 15:43

Exclude fuzzy cases where path parameters contain newline "%0A"

556a29a

Adapt tests for removed "annif routes" and customized "annif run" com…

f4c28dc

…mand

Fix slow fuzzy test by making it use cxapp and asgi test calls

a0a7246

Test with mocking that run command tries to start up server

f12ca5e

Update CustomRequestBodyValidator for Connexion 3 and re-enable its test

3703423

juhoinkinen marked this pull request as ready for review April 8, 2024 10:52

osma requested changes Apr 11, 2024

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

juhoinkinen added 7 commits April 12, 2024 10:51

Use port 5000 by default like with Connexion 2

89f9c3c

Remove --env-file and --app options

7c91eba

Allow only patch level updates on Connexion versions

c57cc14

Fix hints for return types

2f7aaa5

Add test for empty RequestBody to suggest request

ab33375

Omit line from CodeCov covererage report because this case is tested

60e95f6

Use the right annotation to exlude line from codecov report

8a2d0d5

This comment was marked as outdated.

Sign in to view

Use uvicorn workers in docker-compose setup

fc45493

juhoinkinen added the dependencies Pull requests that update a dependency file label Apr 12, 2024

juhoinkinen mentioned this pull request Apr 12, 2024

Update start up command to use uvicorn workers NatLibFi/Annif-tutorial#25

Closed

juhoinkinen added 3 commits April 17, 2024 09:03

Set env in Dockerfile to make Gunicorn use Uvicorn workers

ad464e4

This also reverts the previous commit fc45493. Now uvicorn workers are used always when running with Docker

Merge branch 'main' into upgrade-to-connexion3

9cc9e64

Adapt fixture from merged another PR to this PR

5f02228

osma approved these changes Apr 19, 2024

View reviewed changes

Remove useless test and condition in validator

e1e5d5a

The test was added in ab33375 The condition is checked before this custom validation

juhoinkinen merged commit c3a86a6 into main Apr 19, 2024
14 of 16 checks passed

juhoinkinen deleted the upgrade-to-connexion3 branch April 19, 2024 09:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade to Connexion3 #702

Upgrade to Connexion3 #702

juhoinkinen commented May 10, 2023 •

edited

Loading

sonarcloud bot commented May 10, 2023

codecov bot commented May 10, 2023 •

edited

Loading

juhoinkinen commented Dec 21, 2023

sonarcloud bot commented Jan 9, 2024

juhoinkinen commented Jan 12, 2024

juhoinkinen commented Jan 22, 2024 •

edited

Loading

juhoinkinen commented Feb 22, 2024

juhoinkinen commented Feb 22, 2024

juhoinkinen commented Apr 4, 2024 •

edited

Loading

osma left a comment •

edited

Loading

juhoinkinen commented Apr 12, 2024 •

edited

Loading

CLI operation

`annif run` command

Swagger-UI

REST API

Other

This comment was marked as outdated.

osma left a comment

sonarcloud bot commented Apr 19, 2024

Upgrade to Connexion3 #702

Upgrade to Connexion3 #702

Conversation

juhoinkinen commented May 10, 2023 • edited Loading

sonarcloud bot commented May 10, 2023

codecov bot commented May 10, 2023 • edited Loading

Codecov Report

juhoinkinen commented Dec 21, 2023

sonarcloud bot commented Jan 9, 2024

Quality Gate passed

juhoinkinen commented Jan 12, 2024

juhoinkinen commented Jan 22, 2024 • edited Loading

juhoinkinen commented Feb 22, 2024

juhoinkinen commented Feb 22, 2024

juhoinkinen commented Apr 4, 2024 • edited Loading

osma left a comment • edited Loading

Choose a reason for hiding this comment

CLI operation

annif run command

Swagger-UI

REST API

Other

juhoinkinen commented Apr 12, 2024 • edited Loading

CLI operation

annif run command

Swagger-UI

REST API

Other

This comment was marked as outdated.

osma left a comment

Choose a reason for hiding this comment

sonarcloud bot commented Apr 19, 2024

Quality Gate passed

juhoinkinen commented May 10, 2023 •

edited

Loading

codecov bot commented May 10, 2023 •

edited

Loading

juhoinkinen commented Jan 22, 2024 •

edited

Loading

juhoinkinen commented Apr 4, 2024 •

edited

Loading

osma left a comment •

edited

Loading

`annif run` command

juhoinkinen commented Apr 12, 2024 •

edited

Loading

`annif run` command