parquet-converters

Converts intermediate connectome building output files for consumption by different programs:

• the binary output of touchdetector to Parquet, to be consumed by functionalizer
• the Parquet output of functionalizer to SONATA, to be read with libsonata

Installation

Easiest with the package contained in the BlueBrain fork of Spack.

Clone the project:

$ gh repo clone BlueBrain/parquet-converters -- --recursive --shallow-submodules

The following dependencies should be provided:

• HDF5
• A MPI implementation, e.g., OpenMPI
• Apache Arrow and Parquet development packages, e.g., following the official instructions
• nlohmann_json
• range-v3
• Catch2 (also as submodule)
• CLI11 (also as submodule)
• HighFive (also as submodule)

After ensuring the availability of the dependencies, use the following to build the utilities:

$ cmake -B build -S . -GNinja -DCMAKE_CXX_COMPILER=$(which mpicxx)
$ cmake --build build

See also the Dockerfile for a full build workflow.

Usage

To convert TouchDetector output to Parquet (the output directory is the current directory):

mpirun -np 4 touch2parquet $MY_TD_OUTPUT_DIRECTORY/touchesData.0

This will produce 4 Parquet files, adjust the parallelism accordingly to create more files.

To produce a SONATA file with synapses contained in a population named All:

mpirun -np 100 parquet2hdf5 $MY_FZ_OUTPUT_DIRECTORY/circuit.parquet edges.h5
All

Creating the synapse index requires a higher parallelism than the initial conversion.

Acknowledgment

The development of this software was supported by funding to the Blue Brain Project, a research center of the École polytechnique fédérale de Lausanne (EPFL), from the Swiss government's ETH Board of the Swiss Federal Institutes of Technology.

Copyright (c) 2017-2024 Blue Brain Project/EPFL