Datasources - part 4: documentation #3105
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rolnico
changed the title
flo-dup
force-pushed
the
nro/datasources_3_new_features
branch
from
47bf1a7
to
2a4ba27
Compare
Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>
flo-dup
force-pushed
the
nro/datasources_4_documentation
branch
from
8916995
to
86fd669
Compare
Signed-off-by: Florian Dupuy <florian.dupuy@rte-france.com>
flo-dup
reviewed
Aug 9, 2024
|
Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>
rolnico
changed the title
Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>
Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>
… into nro/datasources_4_documentation
jonenst
reviewed
Sep 19, 2024
| It allows users to read and write files. It is for example used under the hood by Importers to access the filesystem | ||
| during Network imports when using `Network.read()` methods. | ||
|
|
||
| For importers and exporters, datasources are used to access files corresponding to a single network |
There was a problem hiding this comment.
Suggested change
| For importers and exporters, datasources are used to access files corresponding to a single network | |
| For importers and exporters, datasources are used to access files corresponding to a single network. |
jonenst
reviewed
Sep 19, 2024
| Note: this does not apply to compression extensions. | ||
|
|
||
| _**Example:** | ||
| For a file named `europe.west.xiidm.gz`, the base name could be `europe.west` for instance (or `europe` or `europe.w` or ...), while the data extension would be `xiidm`._ |
There was a problem hiding this comment.
Suggested change
| For a file named `europe.west.xiidm.gz`, the base name could be `europe.west` for instance (or `europe` or `europe.w` or ...), while the data extension would be `xiidm`._ | |
| For a file named `europe.west.xiidm.gz`, the base name could be `europe.west` for instance (or `europe` or `europe.w` or ...), while the data extension would be `xiidm` and the compression extension `gz`._ |
There was a problem hiding this comment.
I prefer to remove the .gz in the filename since we are not yet talking about compression
jonenst
reviewed
Sep 19, 2024
| Two classes implement the `DataSource` interface: | ||
| - `MemDataSource`: extension of `ReadOnlyMemDataSource` implementing the writing features of `DataSource` | ||
| - `AbstractFileSystemDataSource`: abstract class used to define datasources based on files present in the file system, | ||
| either directly or in an archive. |
There was a problem hiding this comment.
Suggested change
| either directly or in an archive. | |
| either directly (see below the DirectoryDataSource class and its children) or in an archive (see below the AbstractArchiveDataSource and its children). |
jonenst
reviewed
Sep 19, 2024
| `ZstdDirectoryDataSource`. | ||
|
|
||
| `DirectoryDataSource` integrates the notions of base name and data extension: | ||
| - The base name is used to access files that all start with the same String. For example, `network` would |
There was a problem hiding this comment.
Suggested change
| - The base name is used to access files that all start with the same String. For example, `network` would | |
| - The base name is used to access files that all start with the same prefix. For example, `network` would |
jonenst
reviewed
Sep 19, 2024
| `(String suffix, String ext)` as parameters, you still have the possibility to use files that do not correspond to the | ||
| base name and data extension by using the methods with `(String filename)` as parameter, excluding the compression | ||
| extension if there is one. | ||
|
|
There was a problem hiding this comment.
document that listNames filters by the basename contrary to exists(String filename)
jonenst
reviewed
Sep 19, 2024
| given as a parameter in the datasource constructor, the archive file name is even defined using the base name and the | ||
| data extension, as `<directory>/<basename>.<dataExtension>.<archiveExtension>.<compressionExtension>` with the | ||
| compression extension being optional depending on the archive format. For example `network.xiidm.zip` contains | ||
| `network.xiidm`. |
There was a problem hiding this comment.
document that listnames lists everything without filtering by the basename
jonenst
reviewed
Sep 19, 2024
| datasource.exists("network.south") // Returns false: the file "network.south.gz" does not exist | ||
| datasource.exists("network.xiidm") // Returns true: the file "network.xiidm.gz" exists | ||
|
|
||
| // Check if some files exist in the datasource by using the `exists(String fileName)` method |
There was a problem hiding this comment.
Suggested change
| // Check if some files exist in the datasource by using the `exists(String fileName)` method | |
| // Check if some files exist in the datasource by using the `exists(String suffix, String ext)` method |
jonenst
reviewed
Sep 19, 2024
| } | ||
|
|
||
| // List the files in the datasource | ||
| Set<String> files = datasource.listNames(".*") // returns a set containing: "network", "network.south", "network.xiidm", "network.v3.xiidm", "network_test.txt" |
There was a problem hiding this comment.
highlight that it filters out toto.xiidm.gz because of the basename filtering
jonenst
reviewed
Sep 19, 2024
|
|
||
| // List the files in the datasource | ||
| Set<String> files = datasource.listNames(".*") // returns a set containing: "network", "network.south", "network.xiidm", "network.v3.xiidm", "network_test.txt" | ||
| ``` |
There was a problem hiding this comment.
More things to eventually document (in this PR or in another):
- Use different datasource on the same directory to select different network
- exists(filename) on a file with a different basename that returns true
flo-dup
reviewed
Sep 19, 2024
jonenst
reviewed
Sep 19, 2024
| ## Principles | ||
|
|
||
| Datasources are Java-objects used for I/O operations around PowSyBl. | ||
| It allows users to read and write files. It is for example used under the hood by Importers to access the filesystem |
There was a problem hiding this comment.
Suggested change
| It allows users to read and write files. It is for example used under the hood by Importers to access the filesystem | |
| A Datasource allows users to read and write files. It is for example used under the hood by Importers to access the filesystem |
jonenst
reviewed
Sep 19, 2024
| reading features. | ||
| It has two parameters: | ||
| - a base name, which is a prefix that can be used to consider only files with this prefix (while reading) or as a prefix for | ||
| the output file (while writing), |
There was a problem hiding this comment.
Suggested change
| the output file (while writing), | |
| the output file name (while writing), |
jonenst
reviewed
Sep 19, 2024
| `ReadOnlyDataSource` is the most basic datasource interface available. As you can tell by the name, it only provides | ||
| reading features. | ||
| It has two parameters: | ||
| - a base name, which is a prefix that can be used to consider only files with this prefix (while reading) or as a prefix for |
There was a problem hiding this comment.
Suggested change
| - a base name, which is a prefix that can be used to consider only files with this prefix (while reading) or as a prefix for | |
| - a base name, which is a prefix that can be used to consider only files with names starting with this prefix (while reading) or as a prefix for |
jonenst
reviewed
Sep 19, 2024
| Two classes implement the `DataSource` interface: | ||
| - `MemDataSource`: extension of `ReadOnlyMemDataSource` implementing the writing features of `DataSource` | ||
| - `AbstractFileSystemDataSource`: abstract class used to define datasources based on files present in the file system, | ||
| either (see below the DirectoryDataSource class and its children) or in an archive (see below the |
There was a problem hiding this comment.
Suggested change
| either (see below the DirectoryDataSource class and its children) or in an archive (see below the | |
| either directly (see below the DirectoryDataSource class and its children) or in an archive (see below the |
jonenst
reviewed
Sep 19, 2024
| be a good base name if your files are `network.xiidm`, `network_mapping.csv`, etc. | ||
| - The data extension is the last extension of your main files, excluding the compression extension if they have one. | ||
| It usually corresponds to the data format extension: `csv`, `xml`, `json`, `xiidm`, etc. This extension is mainly used | ||
| to disambiguate the files to use in the datasource, for example when you have files that differ only by the data |
There was a problem hiding this comment.
a bit similar to the earlier sentence "(optionally) a data extension, mainly used to disambiguate identically named data of different type." so maybe either remove or make it a lot more specific: not "mainly used". maybesomething like
"just like you can create 2 different datasources selecting a different subset of files in a folder based on a different prefix (e.g. france.xiidm and europe.xiidm), you can use the data extension to select either france.xiidm or france.uct"
jonenst
reviewed
Sep 19, 2024
|
|
||
| // Using a datasource with different parameters allows to use other files, even on the same directory | ||
| GzDirectoryDataSource totoDatasource = new GzDirectoryDataSource(testDir, "toto", "xiidm", observer); | ||
| oolean totoDatasource.exists(null, "xiidm"); // Returns true: the file "toto.xiidm.gz" exists in the directory |
There was a problem hiding this comment.
Suggested change
| oolean totoDatasource.exists(null, "xiidm"); // Returns true: the file "toto.xiidm.gz" exists in the directory | |
| totoDatasource.exists(null, "xiidm"); // Returns true: the file "toto.xiidm.gz" exists in the directory |
jonenst
approved these changes
Sep 19, 2024
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Please check if the PR fulfills these requirements
What kind of change does this PR introduce?
Documentation update --> add documentation on datasources.
Does this PR introduce a breaking change or deprecate an API?