Prototype REST implementation #183

giffels · 2021-04-09T20:53:32Z

This pull request implements a prototype REST implementation to query the SqlRegistry. Currently two read-only API calls (GET /resources/state, GET /resources ) are supported. The REST API implementation uses FastApi and uvicorn as ASGI server. The REST API relies on JSON Web Token and OAuth2 scopes to support authentication and authorization.

Could be useful to avoid #172.

To activate the TARDIS REST service, simply add the following lines to your TARDISconfiguration.

Services:
  restapi:
      !TardisRestApi
      host: 127.0.0.1
      port: 1234
      secret_key: 752e003f636f402cc23728e185ce8c9eef27b7e02cf509b3015f7757e625b8e4
      algorithm: HS256
      users:
         - user_name: tardis
           hashed_password: $2b$12$c9SSllh1U6tOhIo37sDWF.kdRIU5RQAAOHL9bVYMs2.HluyFE43Uq
           scopes:
             - user:read

to enable logging I suggest to use the following logging configuration in your COBalD configuration.

logging:
  version: 1
  loggers:
    root:
      level: DEBUG
      handlers: [console, file]
    uvicorn:
      level: INFO
      handlers: [ uvicorn_file ]
      propagate: False
  formatters:
    precise:
      format: '%(name)s: %(asctime)s %(message)s'
      datefmt: '%Y-%m-%d %H:%M:%S'
  handlers:
    console:
      class : logging.StreamHandler
      formatter: precise
      stream  : ext://sys.stdout
    file:
      class : logging.handlers.RotatingFileHandler
      formatter: precise
      filename: tardis.log
      maxBytes: 10485760
      backupCount: 3
    uvicorn_file:
      class: logging.handlers.RotatingFileHandler
      formatter: precise
      filename: uvicorn.log
      maxBytes: 10485760
      backupCount: 3

Add unittest using httpx (see https://fastapi.tiangolo.com/advanced/async-tests/)
Add documentation
Add changelog
Add support for tokens

Benchmarks:

Running one million requests against this service took about 11 minutes. Which is equivalent to a performance of about 1500 requests/s.

codecov-commenter · 2021-08-24T16:18:50Z

Codecov Report

Merging #183 (b8721c9) into master (d66c293) will decrease coverage by 0.05%.
The diff coverage is 98.96%.

@@            Coverage Diff             @@
##           master     #183      +/-   ##
==========================================
- Coverage   99.56%   99.50%   -0.06%     
==========================================
  Files          41       53      +12     
  Lines        1823     2022     +199     
==========================================
+ Hits         1815     2012     +197     
- Misses          8       10       +2

Impacted Files	Coverage Δ
tardis/rest/hash_credentials/__main__.py	`83.33% <83.33%> (ø)`
tardis/rest/token_generator/__main__.py	`83.33% <83.33%> (ø)`
tardis/plugins/sqliteregistry.py	`100.00% <100.00%> (ø)`
tardis/rest/app/crud.py	`100.00% <100.00%> (ø)`
tardis/rest/app/database.py	`100.00% <100.00%> (ø)`
tardis/rest/app/main.py	`100.00% <100.00%> (ø)`
tardis/rest/app/routers/login.py	`100.00% <100.00%> (ø)`
tardis/rest/app/routers/resources.py	`100.00% <100.00%> (ø)`
tardis/rest/app/security.py	`100.00% <100.00%> (ø)`
tardis/rest/hash_credentials/hash_credentials.py	`100.00% <100.00%> (ø)`
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d66c293...b8721c9. Read the comment docs.

maxfischer2781 · 2021-11-12T14:00:58Z

I've just pushed out cobald v0.12.3 that includes the required eager evaluation of YAML tags.

giffels · 2021-11-12T14:52:57Z

I've just pushed out cobald v0.12.3 that includes the required eager evaluation of YAML tags.

Thanks a lot.

…3.6 and click, clear lru_cahe in unittests

giffels · 2022-01-11T15:34:37Z

I have now also changed the security scopes to better match the granted permissions. We now have

resources:get instead of user:read
resources:put instead of user:write (not used yet)

The idea is that this can be extended by further scopes like resources:delete, configuration:get, configuration.put, etc.

mschnepf

I have one comment, but nothing critical

docs/source/services/services.rst

RHofsaess

👍

mschnepf

Thanks for all your work. It looks really good 👍

eileen-kuehn

Great stuff 👍
I mainly left some comments on how docstrings might be improved. So fixing should be a no-brainer 😸

docs/source/services/services.rst

eileen-kuehn · 2022-01-21T12:15:22Z

docs/source/services/services.rst

+    both access and error logs into a separate file. Therefore, an ``uvicorn`` logger needs to be configured. An example
+    can be found below.


Can we actually ensure that the example is rendered below? Maybe it is better to state something like "See the example logging configuration.".

Thanks, I thing this is a better.

docs/source/services/services.rst

tardis/rest/service.py

eileen-kuehn · 2022-01-21T12:37:42Z

tardis/rest/service.py

+        return self._secret_key
+
+    @lru_cache(maxsize=16)
+    def get_user(self, user_name: str) -> [None, UserCredentials]:


Do you mind checking the return type? I would expect Optional[UserCredentials] from just reading the code below.

Thanks, I have changed it to Optional[UserCredentials].

tardis/rest/token_generator/generate_token.py

Co-authored-by: Eileen Kuehn <eileen.kuehn@kit.edu>

giffels · 2022-01-28T10:11:44Z

Thank you very much @eileen-kuehn for improving documentation and code.

mschnepf

Looks good to me 👍

eileen-kuehn

Great stuff, looks good to go 🥇

giffels added the enhancement New feature or request label Apr 9, 2021

giffels self-assigned this Apr 9, 2021

giffels added this to the 0.7.0 - Release milestone Aug 10, 2021

giffels force-pushed the add-rest-api branch 2 times, most recently from c3bedf9 to 301ffcf Compare August 24, 2021 16:18

giffels force-pushed the add-rest-api branch from 2f2a4af to f6cd804 Compare October 5, 2021 13:30

giffels mentioned this pull request Oct 6, 2021

Add english language packs to ubuntu, centos8 and rockylinux images #212

Merged

giffels force-pushed the add-rest-api branch 3 times, most recently from 3f76538 to b0192aa Compare October 13, 2021 09:08

giffels mentioned this pull request Nov 10, 2021

Intermediate release 0.12.3 MatterMiners/cobald#102

Merged

3 tasks

giffels added 16 commits December 2, 2021 11:06

Prototype REST implementation

e7e32b3

Add closure to get sqlregistry

22c91e9

Refactoring access to database

f913dcc

Adding rest api unittest and prepare support for jwt

71156a4

Add authentication based on bearer tokens (jwt)

d345977

Enable input validation for get state call

75b8c20

Make secret key mandatory in RestService. Improbe RestService tests

3302c4d

Add security tests and code changes

d4ba325

Add token generator cli for REST api

ac47843

Added unittest for generate token

5b5a5e5

Fix List type hint for Python3.6

8e4ff74

Pin version of uvicorn for python3.6, fix LANG environment for python…

ea4ccd6

…3.6 and click, clear lru_cahe in unittests

Set LANG environment before deployment test

4e34367

Fix flake exception style

07176b4

Improve security unittests and error handling

bb0f6a7

Fix timezone dependency of unittests

a4d1813

giffels dismissed stale reviews from RHofsaess and mschnepf via ee18261 January 11, 2022 11:26

Change security scopes to better match granted permissions

ee5a491

giffels requested review from mschnepf and RHofsaess January 11, 2022 15:59

mschnepf reviewed Jan 11, 2022

View reviewed changes

docs/source/services/services.rst Show resolved Hide resolved

Add outlook on REST API to documentation

b5f317d

giffels requested a review from mschnepf January 11, 2022 17:02

RHofsaess previously approved these changes Jan 13, 2022

View reviewed changes

mschnepf previously approved these changes Jan 13, 2022

View reviewed changes

eileen-kuehn requested changes Jan 21, 2022

View reviewed changes

Apply suggestions from code review

5078f5f

Co-authored-by: Eileen Kuehn <eileen.kuehn@kit.edu>

giffels dismissed stale reviews from mschnepf and RHofsaess via 5078f5f January 28, 2022 09:54

giffels added 2 commits January 28, 2022 11:02

Change wording in Services documentation

bc819ee

Fix type hint of get_user

019aa7f

giffels added 3 commits January 28, 2022 11:12

Make black happy again

fc4e1ae

Fix identation in services documentation

b8721c9

Let sshexecutor test wait a bit longer

a185900

giffels requested review from eileen-kuehn, mschnepf and RHofsaess January 28, 2022 14:41

mschnepf approved these changes Jan 31, 2022

View reviewed changes

eileen-kuehn approved these changes Jan 31, 2022

View reviewed changes

RHofsaess approved these changes Jan 31, 2022

View reviewed changes

giffels merged commit f978b19 into MatterMiners:master Jan 31, 2022

giffels deleted the add-rest-api branch January 31, 2022 11:37

giffels mentioned this pull request Aug 11, 2022

REST API cookie authentication and refactor #250

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prototype REST implementation #183

Prototype REST implementation #183

giffels commented Apr 9, 2021 •

edited

Loading

codecov-commenter commented Aug 24, 2021 •

edited

Loading

maxfischer2781 commented Nov 12, 2021

giffels commented Nov 12, 2021

giffels commented Jan 11, 2022

mschnepf left a comment

RHofsaess left a comment

mschnepf left a comment

eileen-kuehn left a comment

eileen-kuehn Jan 21, 2022

giffels Jan 28, 2022

eileen-kuehn Jan 21, 2022

giffels Jan 28, 2022

giffels commented Jan 28, 2022

mschnepf left a comment

eileen-kuehn left a comment

		both access and error logs into a separate file. Therefore, an ``uvicorn`` logger needs to be configured. An example
		can be found below.

Prototype REST implementation #183

Prototype REST implementation #183

Conversation

giffels commented Apr 9, 2021 • edited Loading

Benchmarks:

codecov-commenter commented Aug 24, 2021 • edited Loading

Codecov Report

maxfischer2781 commented Nov 12, 2021

giffels commented Nov 12, 2021

giffels commented Jan 11, 2022

mschnepf left a comment

Choose a reason for hiding this comment

RHofsaess left a comment

Choose a reason for hiding this comment

mschnepf left a comment

Choose a reason for hiding this comment

eileen-kuehn left a comment

Choose a reason for hiding this comment

eileen-kuehn Jan 21, 2022

Choose a reason for hiding this comment

giffels Jan 28, 2022

Choose a reason for hiding this comment

eileen-kuehn Jan 21, 2022

Choose a reason for hiding this comment

giffels Jan 28, 2022

Choose a reason for hiding this comment

giffels commented Jan 28, 2022

mschnepf left a comment

Choose a reason for hiding this comment

eileen-kuehn left a comment

Choose a reason for hiding this comment

giffels commented Apr 9, 2021 •

edited

Loading

codecov-commenter commented Aug 24, 2021 •

edited

Loading