Add a Dense batched class and kernels #1413

pratikvn · 2023-10-03T18:21:24Z

This PR adds support for the Batch Dense matrix format and currently adds very minimal functionality that is necessary.

TODO

Add CUDA kernels and tests
Add HIP kernels and tests
Add DPCPP kernels and tests

pratikvn · 2023-10-04T09:40:46Z

format!

include/ginkgo/core/matrix/batch_dense.hpp

core/matrix/batch_dense.cpp

pratikvn · 2023-10-06T07:11:10Z

format!

MarcelKoch

Looks good, but some mostly minor comments left.

include/ginkgo/core/matrix/batch_dense.hpp

include/ginkgo/core/base/batch_multi_vector.hpp

core/base/batch_utilities.hpp

common/cuda_hip/matrix/batch_dense_kernels.hpp.inc

dpcpp/matrix/batch_dense_kernels.dp.cpp

reference/matrix/batch_dense_kernels.hpp.inc

reference/test/matrix/batch_dense_kernels.cpp

test/matrix/batch_dense_kernels.cpp

thoasm

Pretty much only nits and two questions:

Why is it batch_dim and not batch::dim?
Wouldn't a type-alias be better (which fixes the bit-size of the integer) in core/matrix/batch_struct.hpp?

include/ginkgo/core/matrix/batch_dense.hpp

reference/matrix/batch_struct.hpp

reference/test/matrix/batch_dense_kernels.cpp

test/matrix/batch_dense_kernels.cpp

thoasm · 2023-10-09T03:44:16Z

dpcpp/matrix/batch_dense_kernels.hpp.inc

+            temp += val * b.values[j];
+        }
+
+        temp = ::gko::kernels::dpcpp::reduce(


I have 2 questions:

Why specify global scope here (::gko)?

Is the top-level gko::batch namespace specifier really necessary (in this file, not just here)?

This is in general expected to be used as a header file and included in a variety of namespaces, so I think it is better to fully specify the namespaces, to avoid any confusion.

sycl has the same reduce signature such that we need to specify the namespace.
the reduce function is not supported correctly/fully in the old SYCL version

pratikvn · 2023-10-09T21:49:54Z

format!

Co-authored-by: Aditya Kashi <kashi1@ornl.gov> Co-authored-by: Isha Aggarwal <isha.aggarwal2@kit.edu>

Co-authored-by: Phuong Nguyen <phuong.nguyen@icl.utk.edu>

Co-authored-by: Pratik Nayak <pratikvn@pm.me>

Co-authored-by: Marcel Koch <marcel.koch@kit.edu>

Co-authored-by: Thomas Grützmacher <thomas.gruetzmacher@kit.edu>

Co-authored-by: Pratik Nayak <pratikvn@pm.me>

MSVC compiler fails lookip in gko::detail if there exists a gko::x::detail namespace

Co-authored-by: Yu-Hsiang Tsai <yhmtsai@gmail.com> Co-authored-by: Marcel Koch <marcel.koch@kit.edu>

Co-authored-by: Pratik Nayak <pratikvn@pm.me>

tcojean

LGTM. Some small points.

tcojean · 2023-10-11T09:20:19Z

include/ginkgo/core/matrix/batch_dense.hpp

+    std::unique_ptr<const unbatch_type> create_const_view_for_item(
+        size_type item_id) const;


Somewhat unrelated, but everywhere in Ginkgo, do we actually need to have different names for the const version? Usually the compiler does a good job at selecting the correct version with const overloading.
https://isocpp.org/wiki/faq/const-correctness#const-overloading

The stdlib also usually has the same method for both const & non const access, see e.g.:
https://en.cppreference.com/w/cpp/ranges/view_interface/data
https://en.cppreference.com/w/cpp/container/vector/data

I see that we never mentioned this in either #402 or #1400

I think we discussed this sometime back and decided that for uniformity, we have named functions for const qualified related operations. I think this was around the time we were creating const_view in gko::array

include/ginkgo/core/matrix/batch_dense.hpp

core/matrix/batch_dense.cpp

core/test/matrix/batch_dense.cpp

tcojean · 2023-10-11T09:38:56Z

core/test/matrix/batch_dense.cpp

+                                          this->exec);
+
+    auto m =
+        gko::batch::create_from_item<gko::batch::matrix::Dense<value_type>>(


We could use ASSERT_DEATH or ASSERT_ANY_THROW to test the out of bound calls.

thoasm

LGTM!

Co-authored-by: Terry Cojean <terry.cojean@kit.edu>

sonarcloud · 2023-10-12T05:41:50Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
24 Code Smells

67.3% Coverage
5.7% Duplication

The version of Java (11.0.3) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more here

Release 1.7.0 to master The Ginkgo team is proud to announce the new Ginkgo minor release 1.7.0. This release brings new features such as: - Complete GPU-resident sparse direct solvers feature set and interfaces, - Improved Cholesky factorization performance, - A new MC64 reordering, - Batched iterative solver support with the BiCGSTAB solver with batched Dense and ELL matrix types, - MPI support for the SYCL backend, - Improved ParILU(T)/ParIC(T) preconditioner convergence, and more! If you face an issue, please first check our [known issues page](https://github.com/ginkgo-project/ginkgo/wiki/Known-Issues) and the [open issues list](https://github.com/ginkgo-project/ginkgo/issues) and if you do not find a solution, feel free to [open a new issue](https://github.com/ginkgo-project/ginkgo/issues/new/choose) or ask a question using the [github discussions](https://github.com/ginkgo-project/ginkgo/discussions). Supported systems and requirements: + For all platforms, CMake 3.16+ + C++14 compliant compiler + Linux and macOS + GCC: 5.5+ + clang: 3.9+ + Intel compiler: 2019+ + Apple Clang: 14.0 is tested. Earlier versions might also work. + NVHPC: 22.7+ + Cray Compiler: 14.0.1+ + CUDA module: CMake 3.18+, and CUDA 10.1+ or NVHPC 22.7+ + HIP module: ROCm 4.5+ + DPC++ module: Intel oneAPI 2022.1+ with oneMKL and oneDPL. Set the CXX compiler to `dpcpp` or `icpx`. + MPI: standard version 3.1+, ideally GPU Aware, for best performance + Windows + MinGW: GCC 5.5+ + Microsoft Visual Studio: VS 2019+ + CUDA module: CUDA 10.1+, Microsoft Visual Studio + OpenMP module: MinGW. ### Version support changes + CUDA 9.2 is no longer supported and 10.0 is untested [#1382](#1382) + Ginkgo now requires CMake version 3.16 (and 3.18 for CUDA) [#1368](#1368) ### Interface changes + `const` Factory parameters can no longer be modified through `with_*` functions, as this breaks const-correctness [#1336](#1336) [#1439](#1439) ### New Deprecations + The `device_reset` parameter of CUDA and HIP executors no longer has an effect, and its `allocation_mode` parameters have been deprecated in favor of the `Allocator` interface. [#1315](#1315) + The CMake parameter `GINKGO_BUILD_DPCPP` has been deprecated in favor of `GINKGO_BUILD_SYCL`. [#1350](#1350) + The `gko::reorder::Rcm` interface has been deprecated in favor of `gko::experimental::reorder::Rcm` based on `Permutation`. [#1418](#1418) + The Permutation class' `permute_mask` functionality. [#1415](#1415) + Multiple functions with typos (`set_complex_subpsace()`, range functions such as `conj_operaton` etc). [#1348](#1348) ### Summary of previous deprecations + `gko::lend()` is not necessary anymore. + The classes `RelativeResidualNorm` and `AbsoluteResidualNorm` are deprecated in favor of `ResidualNorm`. + The class `AmgxPgm` is deprecated in favor of `Pgm`. + Default constructors for the CSR `load_balance` and `automatical` strategies + The PolymorphicObject's move-semantic `copy_from` variant + The templated `SolverBase` class. + The class `MachineTopology` is deprecated in favor of `machine_topology`. + Logger constructors and create functions with the `executor` parameter. + The virtual, protected, Dense functions `compute_norm1_impl`, `add_scaled_impl`, etc. + Logger events for solvers and criterion without the additional `implicit_tau_sq` parameter. + The global `gko::solver::default_krylov_dim`, use instead `gko::solver::gmres_default_krylov_dim`. ### Added features + Adds a batch::BatchLinOp class that forms a base class for batched linear operators such as batched matrix formats, solver and preconditioners [#1379](#1379) + Adds a batch::MultiVector class that enables operations such as dot, norm, scale on batched vectors [#1371](#1371) + Adds a batch::Dense matrix format that stores batched dense matrices and provides gemv operations for these dense matrices. [#1413](#1413) + Adds a batch::Ell matrix format that stores batched Ell matrices and provides spmv operations for these batched Ell matrices. [#1416](#1416) [#1437](#1437) + Add a batch::Bicgstab solver (class, core, and reference kernels) that enables iterative solution of batched linear systems [#1438](#1438). + Add device kernels (CUDA, HIP, and DPCPP) for batch::Bicgstab solver. [#1443](#1443). + New MC64 reordering algorithm which optimizes the diagonal product or sum of a matrix by permuting the rows, and computes additional scaling factors for equilibriation [#1120](#1120) + New interface for (non-symmetric) permutation and scaled permutation of Dense and Csr matrices [#1415](#1415) + LU and Cholesky Factorizations can now be separated into their factors [#1432](#1432) + New symbolic LU factorization algorithm that is optimized for matrices with an almost-symmetric sparsity pattern [#1445](#1445) + Sorting kernels for SparsityCsr on all backends [#1343](#1343) + Allow passing pre-generated local solver as factory parameter for the distributed Schwarz preconditioner [#1426](#1426) + Add DPCPP kernels for Partition [#1034](#1034), and CSR's `check_diagonal_entries` and `add_scaled_identity` functionality [#1436](#1436) + Adds a helper function to create a partition based on either local sizes, or local ranges [#1227](#1227) + Add function to compute arithmetic mean of dense and distributed vectors [#1275](#1275) + Adds `icpx` compiler supports [#1350](#1350) + All backends can be built simultaneously [#1333](#1333) + Emits a CMake warning in downstream projects that use different compilers than the installed Ginkgo [#1372](#1372) + Reordering algorithms in sparse_blas benchmark [#1354](#1354) + Benchmarks gained an `-allocator` parameter to specify device allocators [#1385](#1385) + Benchmarks gained an `-input_matrix` parameter that initializes the input JSON based on the filename [#1387](#1387) + Benchmark inputs can now be reordered as a preprocessing step [#1408](#1408) ### Improvements + Significantly improve Cholesky factorization performance [#1366](#1366) + Improve parallel build performance [#1378](#1378) + Allow constrained parallel test execution using CTest resources [#1373](#1373) + Use arithmetic type more inside mixed precision ELL [#1414](#1414) + Most factory parameters of factory type no longer need to be constructed explicitly via `.on(exec)` [#1336](#1336) [#1439](#1439) + Improve ParILU(T)/ParIC(T) convergence by using more appropriate atomic operations [#1434](#1434) ### Fixes + Fix an over-allocation for OpenMP reductions [#1369](#1369) + Fix DPCPP's common-kernel reduction for empty input sizes [#1362](#1362) + Fix several typos in the API and documentation [#1348](#1348) + Fix inconsistent `Threads` between generations [#1388](#1388) + Fix benchmark median condition [#1398](#1398) + Fix HIP 5.6.0 compilation [#1411](#1411) + Fix missing destruction of rand_generator from cuda/hip [#1417](#1417) + Fix PAPI logger destruction order [#1419](#1419) + Fix TAU logger compilation [#1422](#1422) + Fix relative criterion to not iterate if the residual is already zero [#1079](#1079) + Fix memory_order invocations with C++20 changes [#1402](#1402) + Fix `check_diagonal_entries_exist` report correctly when only missing diagonal value in the last rows. [#1440](#1440) + Fix checking OpenMPI version in cross-compilation settings [#1446](#1446) + Fix false-positive deprecation warnings in Ginkgo, especially for the old Rcm (it doesn't emit deprecation warnings anymore as a result but is still considered deprecated) [#1444](#1444) ### Related PR: #1451

Release 1.7.0 to develop The Ginkgo team is proud to announce the new Ginkgo minor release 1.7.0. This release brings new features such as: - Complete GPU-resident sparse direct solvers feature set and interfaces, - Improved Cholesky factorization performance, - A new MC64 reordering, - Batched iterative solver support with the BiCGSTAB solver with batched Dense and ELL matrix types, - MPI support for the SYCL backend, - Improved ParILU(T)/ParIC(T) preconditioner convergence, and more! If you face an issue, please first check our [known issues page](https://github.com/ginkgo-project/ginkgo/wiki/Known-Issues) and the [open issues list](https://github.com/ginkgo-project/ginkgo/issues) and if you do not find a solution, feel free to [open a new issue](https://github.com/ginkgo-project/ginkgo/issues/new/choose) or ask a question using the [github discussions](https://github.com/ginkgo-project/ginkgo/discussions). Supported systems and requirements: + For all platforms, CMake 3.16+ + C++14 compliant compiler + Linux and macOS + GCC: 5.5+ + clang: 3.9+ + Intel compiler: 2019+ + Apple Clang: 14.0 is tested. Earlier versions might also work. + NVHPC: 22.7+ + Cray Compiler: 14.0.1+ + CUDA module: CMake 3.18+, and CUDA 10.1+ or NVHPC 22.7+ + HIP module: ROCm 4.5+ + DPC++ module: Intel oneAPI 2022.1+ with oneMKL and oneDPL. Set the CXX compiler to `dpcpp` or `icpx`. + MPI: standard version 3.1+, ideally GPU Aware, for best performance + Windows + MinGW: GCC 5.5+ + Microsoft Visual Studio: VS 2019+ + CUDA module: CUDA 10.1+, Microsoft Visual Studio + OpenMP module: MinGW. ### Version support changes + CUDA 9.2 is no longer supported and 10.0 is untested [#1382](#1382) + Ginkgo now requires CMake version 3.16 (and 3.18 for CUDA) [#1368](#1368) ### Interface changes + `const` Factory parameters can no longer be modified through `with_*` functions, as this breaks const-correctness [#1336](#1336) [#1439](#1439) ### New Deprecations + The `device_reset` parameter of CUDA and HIP executors no longer has an effect, and its `allocation_mode` parameters have been deprecated in favor of the `Allocator` interface. [#1315](#1315) + The CMake parameter `GINKGO_BUILD_DPCPP` has been deprecated in favor of `GINKGO_BUILD_SYCL`. [#1350](#1350) + The `gko::reorder::Rcm` interface has been deprecated in favor of `gko::experimental::reorder::Rcm` based on `Permutation`. [#1418](#1418) + The Permutation class' `permute_mask` functionality. [#1415](#1415) + Multiple functions with typos (`set_complex_subpsace()`, range functions such as `conj_operaton` etc). [#1348](#1348) ### Summary of previous deprecations + `gko::lend()` is not necessary anymore. + The classes `RelativeResidualNorm` and `AbsoluteResidualNorm` are deprecated in favor of `ResidualNorm`. + The class `AmgxPgm` is deprecated in favor of `Pgm`. + Default constructors for the CSR `load_balance` and `automatical` strategies + The PolymorphicObject's move-semantic `copy_from` variant + The templated `SolverBase` class. + The class `MachineTopology` is deprecated in favor of `machine_topology`. + Logger constructors and create functions with the `executor` parameter. + The virtual, protected, Dense functions `compute_norm1_impl`, `add_scaled_impl`, etc. + Logger events for solvers and criterion without the additional `implicit_tau_sq` parameter. + The global `gko::solver::default_krylov_dim`, use instead `gko::solver::gmres_default_krylov_dim`. ### Added features + Adds a batch::BatchLinOp class that forms a base class for batched linear operators such as batched matrix formats, solver and preconditioners [#1379](#1379) + Adds a batch::MultiVector class that enables operations such as dot, norm, scale on batched vectors [#1371](#1371) + Adds a batch::Dense matrix format that stores batched dense matrices and provides gemv operations for these dense matrices. [#1413](#1413) + Adds a batch::Ell matrix format that stores batched Ell matrices and provides spmv operations for these batched Ell matrices. [#1416](#1416) [#1437](#1437) + Add a batch::Bicgstab solver (class, core, and reference kernels) that enables iterative solution of batched linear systems [#1438](#1438). + Add device kernels (CUDA, HIP, and DPCPP) for batch::Bicgstab solver. [#1443](#1443). + New MC64 reordering algorithm which optimizes the diagonal product or sum of a matrix by permuting the rows, and computes additional scaling factors for equilibriation [#1120](#1120) + New interface for (non-symmetric) permutation and scaled permutation of Dense and Csr matrices [#1415](#1415) + LU and Cholesky Factorizations can now be separated into their factors [#1432](#1432) + New symbolic LU factorization algorithm that is optimized for matrices with an almost-symmetric sparsity pattern [#1445](#1445) + Sorting kernels for SparsityCsr on all backends [#1343](#1343) + Allow passing pre-generated local solver as factory parameter for the distributed Schwarz preconditioner [#1426](#1426) + Add DPCPP kernels for Partition [#1034](#1034), and CSR's `check_diagonal_entries` and `add_scaled_identity` functionality [#1436](#1436) + Adds a helper function to create a partition based on either local sizes, or local ranges [#1227](#1227) + Add function to compute arithmetic mean of dense and distributed vectors [#1275](#1275) + Adds `icpx` compiler supports [#1350](#1350) + All backends can be built simultaneously [#1333](#1333) + Emits a CMake warning in downstream projects that use different compilers than the installed Ginkgo [#1372](#1372) + Reordering algorithms in sparse_blas benchmark [#1354](#1354) + Benchmarks gained an `-allocator` parameter to specify device allocators [#1385](#1385) + Benchmarks gained an `-input_matrix` parameter that initializes the input JSON based on the filename [#1387](#1387) + Benchmark inputs can now be reordered as a preprocessing step [#1408](#1408) ### Improvements + Significantly improve Cholesky factorization performance [#1366](#1366) + Improve parallel build performance [#1378](#1378) + Allow constrained parallel test execution using CTest resources [#1373](#1373) + Use arithmetic type more inside mixed precision ELL [#1414](#1414) + Most factory parameters of factory type no longer need to be constructed explicitly via `.on(exec)` [#1336](#1336) [#1439](#1439) + Improve ParILU(T)/ParIC(T) convergence by using more appropriate atomic operations [#1434](#1434) ### Fixes + Fix an over-allocation for OpenMP reductions [#1369](#1369) + Fix DPCPP's common-kernel reduction for empty input sizes [#1362](#1362) + Fix several typos in the API and documentation [#1348](#1348) + Fix inconsistent `Threads` between generations [#1388](#1388) + Fix benchmark median condition [#1398](#1398) + Fix HIP 5.6.0 compilation [#1411](#1411) + Fix missing destruction of rand_generator from cuda/hip [#1417](#1417) + Fix PAPI logger destruction order [#1419](#1419) + Fix TAU logger compilation [#1422](#1422) + Fix relative criterion to not iterate if the residual is already zero [#1079](#1079) + Fix memory_order invocations with C++20 changes [#1402](#1402) + Fix `check_diagonal_entries_exist` report correctly when only missing diagonal value in the last rows. [#1440](#1440) + Fix checking OpenMPI version in cross-compilation settings [#1446](#1446) + Fix false-positive deprecation warnings in Ginkgo, especially for the old Rcm (it doesn't emit deprecation warnings anymore as a result but is still considered deprecated) [#1444](#1444) ### Related PR: #1454

pratikvn added is:new-feature A request or implementation of a feature that does not exist yet. 1:ST:WIP This PR is a work in progress. Not ready for review. type:batched-functionality This is related to the batched functionality in Ginkgo labels Oct 3, 2023

pratikvn self-assigned this Oct 3, 2023

ginkgo-bot added reg:build This is related to the build system. reg:testing This is related to testing. type:matrix-format This is related to the Matrix formats mod:all This touches all Ginkgo modules. labels Oct 3, 2023

MarcelKoch reviewed Oct 4, 2023

View reviewed changes

pratikvn removed the 1:ST:WIP This PR is a work in progress. Not ready for review. label Oct 5, 2023

pratikvn changed the title ~~WIP: Batch dense PR~~ Add a Batch dense matrix class Oct 5, 2023

pratikvn changed the title ~~Add a Batch dense matrix class~~ Add a batch::matrix::Dense class and kernels Oct 5, 2023

pratikvn added the 1:ST:ready-for-review This PR is ready for review label Oct 5, 2023

pratikvn force-pushed the batch-linop branch from cd54742 to 4a1ea67 Compare October 6, 2023 07:39

pratikvn force-pushed the batch-dense branch 2 times, most recently from fb25d3c to f200e51 Compare October 6, 2023 10:59

pratikvn added this to the Release 1.7.0 milestone Oct 6, 2023

pratikvn added the 1:ST:run-full-test label Oct 6, 2023

pratikvn changed the title ~~Add a batch::matrix::Dense class and kernels~~ Add a Dense batched class and kernels Oct 7, 2023

pratikvn requested review from a team October 8, 2023 10:51

MarcelKoch self-requested a review October 9, 2023 07:58

MarcelKoch previously requested changes Oct 9, 2023

View reviewed changes

thoasm reviewed Oct 9, 2023

View reviewed changes

pratikvn force-pushed the batch-linop branch from fde5cb4 to 0078777 Compare October 9, 2023 21:46

pratikvn force-pushed the batch-dense branch from d28ddd3 to cf3562c Compare October 9, 2023 21:48

pratikvn and others added 18 commits October 11, 2023 10:48

Add CUDA, HIP kernels and tests

a4b82ea

Co-authored-by: Aditya Kashi <kashi1@ornl.gov> Co-authored-by: Isha Aggarwal <isha.aggarwal2@kit.edu>

Add SYCL kernels and tests WIP

b23dbfa

Co-authored-by: Phuong Nguyen <phuong.nguyen@icl.utk.edu>

HIP and CUDA thrust fixes

07578dd

SYCL kernel fixes

81bcf74

BatchDense -> batch::Dense

2483667

Doc updates and multivector view

b402b94

Format files

3ca9fb4

Co-authored-by: Pratik Nayak <pratikvn@pm.me>

Use CommonTestFixture value_type

e5b8813

Review updates

c00b6d9

Co-authored-by: Marcel Koch <marcel.koch@kit.edu>

Review updates

fe21d65

Co-authored-by: Thomas Grützmacher <thomas.gruetzmacher@kit.edu>

dpcpp Jacobi needs ranlux

f210ea9

Remove create_multivector_view

fb74b71

Format files

660ec7c

Co-authored-by: Pratik Nayak <pratikvn@pm.me>

const_array_view needs to be in gko::

76726e9

MSVC compiler fails lookip in gko::detail if there exists a gko::x::detail namespace

Review updates

09b7574

Co-authored-by: Yu-Hsiang Tsai <yhmtsai@gmail.com> Co-authored-by: Marcel Koch <marcel.koch@kit.edu>

Move apply validation to BatchLinOp

94452e9

Add to test_install

4a18c40

Format files

927e8c8

Co-authored-by: Pratik Nayak <pratikvn@pm.me>

pratikvn force-pushed the batch-dense branch from e1e8ac4 to c370bca Compare October 11, 2023 08:49

tcojean approved these changes Oct 11, 2023

View reviewed changes

thoasm approved these changes Oct 11, 2023

View reviewed changes

Review updates

190a010

Co-authored-by: Terry Cojean <terry.cojean@kit.edu>

pratikvn force-pushed the batch-dense branch from c370bca to 190a010 Compare October 11, 2023 12:18

pratikvn merged commit 6b0acfb into develop Oct 12, 2023
17 checks passed

pratikvn deleted the batch-dense branch October 12, 2023 07:48

tcojean mentioned this pull request Nov 6, 2023

Release 1.7.0 to master #1451

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a Dense batched class and kernels #1413

Add a Dense batched class and kernels #1413

pratikvn commented Oct 3, 2023 •

edited

Loading

pratikvn commented Oct 4, 2023

pratikvn commented Oct 6, 2023

MarcelKoch left a comment

thoasm left a comment

thoasm Oct 9, 2023

pratikvn Oct 9, 2023

yhmtsai Oct 10, 2023

pratikvn commented Oct 9, 2023

tcojean left a comment

tcojean Oct 11, 2023

pratikvn Oct 11, 2023

tcojean Oct 11, 2023

thoasm left a comment

sonarcloud bot commented Oct 12, 2023

		std::unique_ptr<const unbatch_type> create_const_view_for_item(
		size_type item_id) const;

Add a Dense batched class and kernels #1413

Add a Dense batched class and kernels #1413

Conversation

pratikvn commented Oct 3, 2023 • edited Loading

TODO

pratikvn commented Oct 4, 2023

pratikvn commented Oct 6, 2023

MarcelKoch left a comment

Choose a reason for hiding this comment

thoasm left a comment

Choose a reason for hiding this comment

thoasm Oct 9, 2023

Choose a reason for hiding this comment

pratikvn Oct 9, 2023

Choose a reason for hiding this comment

yhmtsai Oct 10, 2023

Choose a reason for hiding this comment

pratikvn commented Oct 9, 2023

tcojean left a comment

Choose a reason for hiding this comment

tcojean Oct 11, 2023

Choose a reason for hiding this comment

pratikvn Oct 11, 2023

Choose a reason for hiding this comment

tcojean Oct 11, 2023

Choose a reason for hiding this comment

thoasm left a comment

Choose a reason for hiding this comment

sonarcloud bot commented Oct 12, 2023

pratikvn commented Oct 3, 2023 •

edited

Loading