Allow for more advanced filtering when retrieving records

[QuantumDancer]

Right now, we can only retrieve the full list of records stored in the database or filter based on either the start or stop time of the record. We should provide more ways to filter records when querying so that this can be done on the server side and not on the client side.

TODO: Think about what we or other users might want to filter on.

Should be done after #465

[QuantumDancer]

New filter options

Based on our discussions in Freiburg, we propose to implement the following filter options. Multiple selection options should be supported at the same time. I.e., one should be able to select records based on criteria on different fields or based on multiple criteria for one field. See the sections below for details.

start time, stop time, and runtime of a record

Allow to select records with a start_time/stop_time/duration less than/less than or equal/greater than/greater than or equal a given value.

Template query

GET /records?<field>{<operator>}=<value>

• <field> should support the following values: start_time, stop_time, or runtime
• <operator> should support the following values:
  • gt: greater than (>)
  • lt: less than (<)
  • gte: greater than or equal (>=)
  • lte: less than or equal (<=)
• <value> should only be accepted if in the format of RFC 3339/ISO 8601

Examples

All examples use UTC timezone.

• Select all records that where started on October 1st or later:

  GET /records?start_time{gte}=2003-10-01T00:00:00+01:00&stop_time{lt}=2003-10-04T00:00:00+00:00
  

• Select all records that where started on October 1 or later and stopped before October 4:

  GET /records?start_time{gte}=2003-10-01T00:00:00+01:00&stop_time{lt}=2003-10-04T00:00:00+00:00
  

• Select all records that where started between October 1 and October 2:

  GET /records?start_time{gte}=2003-10-01T00:00:00+01:00&start_time{lt}=2003-10-03T00:00:00+00:00
  

• Select all records that where stopped after Septer 17 and lasted less than 1 day and 12 hours:

  GET /records?stop_time{gt}=2003-09-17T00:00:00+00:00&runtime{lt}=P1DT12
  

Meta

Allow to filter on values in arbitrary meta keys. Reminder: The meta structure is defined as HashMap<String, Vec<String>>. So we need to be able to set both the key and value to search for. In addition, the values in the hash map are actually lists of strings. To not overly complicate things, we will only support for matching or excluding full list items for now.

Template query

GET /records?meta{<field>}{<operator>}=<value>

• <field> should support arbitrary strings that can be converted into ValidName
• <operator> should support the following values:
  • c: contains
  • dnc: does not contain

Examples

• Select records where the meta entry for site_id has site1 as one of the values. This will match meta: {"site_id": ["site1"]} and meta: {"site_id": ["site1", "site2"]}, but not meta: {"site_id": ["site2", "site3"]}.

  GET /records?meta{site_id}{c}=site1
  

• Select records where the meta entry for site_id contains both site1 and site2. This will match meta: {"site_id": ["site1", "site2"]} but not meta: {"site_id": ["site1"]} and meta: {"site_id": ["site2", "site3"]}.

  GET /records?meta{site_id}{c}=site1&meta{site_id}{c}=site2
  

• Select records where the meta entry for site_id contains site1 but not site2. This will match meta: {"site_id": ["site1"]} but not meta: {"site_id": ["site1", "site2"]} and meta: {"site_id": ["site2", "site3"]}.

  GET /records?meta{site_id}{c}=site1&meta{site_id}{dnc}=site2
  

• Mixing different meta fields is of course also possible. I.e. select records where the meta entry for site_id contains site1 and group_id contains group1, but not group2:

  GET /records?meta{site_id}{c}=site1&meta{group_id}{c}=group1&meta{group_id}{dnc}=group2
  

Components

TBD

Clients

TBD (we also need support the additional filtering options for both the rust and python client. imho a builder pattern to build up the query would make sense.)