Store: store structured data as JSON objects #38
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 3 commits
June 2, 2025 15:43
Implement full-blown JSON serialization and validation functions. Support extensions for user classes.
* Use JSON encoding for strings with special characters, vectors, and
maps to ensure that deserialization returns what has been serialized.
* Make JsonParser work for JSON objects containing strings with special
characters.
* JsonParser now returns a bool indicating whether the parsing was
successful.
* Replace `template <typename T> void operator>>(T&)` with `void
operator>>(std::string&)` since it only worked for strings anyways.
* Make `operator>>` produce numbers as JSON numbers.
* Rewrite UnitTests/StoreTest to thoroughly test Store implementation.
The type of the stored object is encoded by the first character in the
stored value: `"` for strings, `[` for arrays, and `{` for objects. A
matching character is expected at the end of the value. Any other
character at the beginning of the value means that the object was
encoded with a custom `operator<<(std::ostream&, T)`, and it is expected
that `operator>>(std::istream&, T&)` can be used to decode the object
completely. If such encoding produces a string beginning with one of the
characters above, it is wrapped into a pair of '"'. These conversions
are handled transparently to the user.
…ew store contents
Calling Initialise on a file setting an integer variable, e.g.,
var 1
resulted in this variable stored as "1" and interpreted as a string, so
Get<int>("1") returned 0 (json_decode failed to parse "1" as a number).
This is inconsistent both with the previous and the new behaviours where
numbers set with `Set` are stored unquoted. This patch updates
Initialise to store a value as is, unless it begins with a `"`, or it
begins with a `[` or `{` and is not a valid JSON object. In these cases
the value is quoted.
Also:
* Initialise now does not drop the whitespace in the value, so
var a string with a tab and a long space
does not result in `a string with a tab and a long space`.
* Comments, newlines, and spaces at the end can now be escaped:
escaped_comment test \# test
escaped_newline test \
test2
escaped_whitespace test \
* One backward incompatible change is that a space is now not required
before an inline comment:
var val#ue
now results in `val` rather than `val#ue`.
StoreTest has been updated to test for these cases.
Avoid leaving Store half-filled.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request makes an overhaul of
Storeimplementation, extending its support of edge cases in user data.The primary change is that the structured data stored in a
Storesuch as vectors and maps, are stored as JSON arrays and objects, with the strings in keys and values properly encoded, so a"q } w"should result the same string when requested, and it won't confuse the JSON parser and serializer.The type of an object stored is encoded by the first character of its serialized string:
"means that it's a string,[--- a vector,{--- a map. Any other character means that it is an object with custom encoding (see below). Strings and custom objects are stored as is, but when serialized to JSON format, their contents are properly escaped to produce a valid JSON string. Vectors and maps are stored in JSON format initially.The user can add support for storage of their own classes
Tby implementingoperator<<(std::ostream&, T)andoperator>>(std::istream&, T&), orjson_encode_r(std::ostream&, T)andjson_decode_r(const char*&, T&). When both the operators and the functions are defined, the functions take priority.json_encode_randjson_decode_rextend the JSON parser and serializer to support custom user classes and are useful outside theStoreclass.Speaking of JSON, this pull request also adds generic functions for JSON production, validation, and parsing, see
json_encode,json_scan*andjson_decodeinJson.h.Store::Initialisehas been updated to allow escaped comments, newlines, and spaces at the end of a line, and to keep all of the spaces in the value string, soresults in
"long space # test \\ \n newline ". This change is not backwards compatible, but it is unlikely that anyone relied on that feature.UnitTests/StoreTesthas been rewritten to test for many edge cases and the new features.I personally think that we should stop abusing
Storefor JSON data and instead either implement a customJSONobject that stores values as a tree in memory, or just use the newjson_encodeandjson_decodefunctions for (de-)serialization.I have checked that libDAQInterface Example works with the new
Store, but I think is worth testing this code on some real codebase before merging this pull request. Can someone do that?This pull request fixes #19, #22, #25. #16 appears to be not relevant anymore.