Forest Of Octrees #1009

lroberts36 · 2024-02-22T22:17:48Z

PR Summary

This PR begins transitioning Parthenon domain from being represented by a single octree to a forest of octrees that cover the mesh domain. This PR switches to a new forest based mesh structure under the hood, but does not provide any user facing mechanism for building generic forests. Rather, it just builds a forest equivalent to the old root grid. Previously, if the base grid wasn't logically cubic parts of the octree domain were masked out. Now, multiple trees will cover the domain without masking. These changes should be transparent to downstream codes. For this limited set of forests, there is a clear way to map between forest logical locations and logical locations in the old Athena++ tree which allows writing the Athena++ locations out to file, building grids based on Athena++ style input, etc. As a result, the behavior of the code and output should be completely unchanged and the changes made in this PR should have no impact on down stream codes (fingers crossed). Obviously, as-is these changes have limited utility but future PRs built on them should allow for more complex grids to be represented.

This PR also removes a couple of big things from the code:

MeshblockTree is replaced by Forest and Tree (the latter of which is based on a hash map of logical locations rather than the pointer tree used by MeshblockTree)
Most of the old boundary communication machinery (BoundaryBase, BoundaryValue, BoundaryBuffer and BoundaryVariable) and pbval from MeshBlock. There was already a lot of dead code in those classes and this PR required removing the old neighbor finding routines (since they relied on Mesh::tree). To make removing this stuff possible, some reasonably small changes to the swarms related communication was required. The changes and reorganization for this make up a lot of the lines of the PR.

This PR breaks multigrid, but fixing that would make the PR even longer (and no one is using it except me I think).

PR Checklist

Code passes cpplint
Code is formatted
Changes are summarized in CHANGELOG.md
CI has been triggered on Darwin for performance regression tests.
Docs build
(@lanl.gov employees) Update copyright on changed files

pgrete · 2024-03-25T13:46:14Z

I'll get to review a bit later this week (as I also want to run some [performance] tests downstream)

lroberts36 · 2024-03-26T16:07:32Z

This PR passes all Riot tests.

pgrete

LGTM -- and also nice cleanup of the code and thanks for the unit tests (which are way more readable than the integrated version of construction in the Mesh class.

I also ran some downstream tests and didn't notice anything odd.
In fact, the AMR performance increased by up to 30%-50% (based on the wsec_AMR output and for a test with several hundreds of blocks per device).

src/bvals/comms/bnd_info.cpp

pgrete · 2024-03-29T22:13:40Z

src/bvals/neighbor_block.hpp

+// TODO(felker): nest these enum definitions inside bvals/ classes, when possible.
+
+// DEPRECATED(felker): maintain old-style (ALL_CAPS) enumerators as unscoped,unnamed types
+// Keep for compatibility with user-provided pgen/ files. Use only new types internally.
+
+// GCC 6 added Enumerator Attr (v6.1 released on 2016-04-27)
+// TODO(felker): replace with C++14 [[deprecated]] attributes if we ever bump --std=c++14
+#if (defined(__GNUC__) && __GNUC__ >= 6) || (defined(__clang__) && __clang_major__ >= 3)
+enum {
+  FACE_UNDEF __attribute__((deprecated)) = -1,
+  INNER_X1 __attribute__((deprecated)),
+  OUTER_X1 __attribute__((deprecated)),
+  INNER_X2 __attribute__((deprecated)),
+  OUTER_X2 __attribute__((deprecated)),
+  INNER_X3 __attribute__((deprecated)),
+  OUTER_X3 __attribute__((deprecated))
+};
+enum {
+  BLOCK_BNDRY __attribute__((deprecated)) = -1,
+  BNDRY_UNDEF __attribute__((deprecated)),
+  REFLECTING_BNDRY __attribute__((deprecated)),
+  OUTFLOW_BNDRY __attribute__((deprecated)),
+  PERIODIC_BNDRY __attribute__((deprecated))
+};
+#else
+enum { FACE_UNDEF = -1, INNER_X1, OUTER_X1, INNER_X2, OUTER_X2, INNER_X3, OUTER_X3 };
+enum {
+  BLOCK_BNDRY = -1,
+  BNDRY_UNDEF,
+  REFLECTING_BNDRY,
+  OUTFLOW_BNDRY,
+  USER_BNDRY,
+  PERIODIC_BNDRY,
+  POLAR_BNDRY,
+  POLAR_BNDRY_WEDGE
+};
+#endif


Just checked and we don't use it downstream, so this can go.

src/defs.hpp

src/mesh/forest/cell_center_offsets.hpp

pgrete · 2024-03-29T22:26:53Z

src/mesh/forest/cell_center_offsets.hpp

+
+// TODO(LFR): Consider switching this to C-style enum within a namespace to avoid
+// static_cast
+enum class Offset : int { Low = -1, Middle = 0, Up = 1 };


Further up in the bvals, the ordering of offset was 0, -1, 1 IIRC.
I realize these are enums here, so it shouldn't matter, but I'm wondering if there any danger for confusion mid/long term wrt to ordering convention.

I am not sure I see where the ordering is different?

src/mesh/forest/tree.cpp

src/mesh/forest/tree.hpp

src/mesh/logical_location.hpp

tst/unit/test_forest.cpp

brryan · 2024-04-01T18:41:34Z

I don't have any objections to the swarm boundary comms refactor (thank you for working this out @lroberts36!) -- the only thing I would like to do is run the parthenon particle_leapfrog example for a large number of meshblocks and see that it still works. Right now this branch isn't building for me on volta-x86:
"/home/brryan/github/parthenon/example/sparse_advection/sparse_advection_driver.cpp", line 125: error: expected an identifier
      auto restrict =
                    ^

"/home/brryan/github/parthenon/example/sparse_advection/sparse_advection_driver.cpp", line 130: error: expected an expression
        tl.AddTask(restrict, SparseDealloc, mc1.get());
but once it is working again I'm happy to check that example.

Just tested on particle_leapfrog on 32 nodes and it runs and gives the right particle positions at final time, so everything looks good to me.

brryan

This approval is mainly for the modifications to particles. Thanks for dealing with that @lroberts36

lroberts36 added 30 commits February 7, 2024 11:43

Start on making forest of octrees

d1fa2aa

Seemingly working LogicalLocation based trees

d080719

start on forest refinement

50d1c59

Working three tree forest example

9158fb2

add mesh plotting script

437d082

Work toward automating edge relationship finding

c40bf29

small

6ab10ff

Working automated neighbor finding and plotting

cb88d4e

cleanup

cbe4882

Add squared circle setup

470f80c

allow for unrefined trees in plots

aae426f

Add a couple power of 2 bithacks

95f4997

More work on connecting trees

dc612cb

Start formalizing forest and replicating current functionality

0422280

split into separate files and other cleanup

68219cf

further split

d777c50

More cleanup

ee65a3e

small

9f5ada2

duplicate old tree functionality in Tree

f2d990a

Add transformation

f53e85d

Duplicate MeshBlockTree functionality in Forest

0fcfb50

format and lint

36280be

fix integer division bug

0ada32b

switch to neighbor maps and add gid map to test

e455ab9

reorg

e8510dc

format and lint

b4279ef

fix typo

3dbbac8

Add some simplified routines that ignore periodicity

4167195

Start comparing forest results to old connection results

f809217

resolve and set gids

40abac4

Merge branch 'develop' into lroberts36/add-forest-of-octrees

2a9c587

lroberts36 added 5 commits March 26, 2024 10:17

move change to pre-release

94e6d52

Merge branch 'develop' into lroberts36/add-forest-of-octrees

e724956

remove merge detritus

a0812e6

Merge branch 'develop' into lroberts36/add-forest-of-octrees

c784f9d

fix apparently offseting bugs for negative logical levels

37ad621

pgrete approved these changes Mar 29, 2024

View reviewed changes

lroberts36 added 8 commits April 1, 2024 11:24

comment

87dd8c4

add fail

e1395f4

remove dead code

8cf8662

update copyright

5f0d31e

mv file

fac835b

finish mv

93e7347

format

6fa6ac8

clarify naming scheme of old tree

3da72af

brryan approved these changes Apr 1, 2024

View reviewed changes

lroberts36 enabled auto-merge April 1, 2024 18:48

lroberts36 disabled auto-merge April 1, 2024 19:53

lroberts36 enabled auto-merge April 1, 2024 19:54

lroberts36 disabled auto-merge April 1, 2024 20:23

lroberts36 enabled auto-merge April 1, 2024 20:23

lroberts36 mentioned this pull request Apr 1, 2024

Add features to history output #1024

Merged

17 tasks

lroberts36 added 2 commits April 2, 2024 11:25

disable gmg tests for the moment

2fedbbd

correctly exclude gmg tests

4206737

lroberts36 disabled auto-merge April 2, 2024 18:27

lroberts36 enabled auto-merge April 2, 2024 18:27

lroberts36 merged commit 69a6599 into develop Apr 2, 2024
49 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Forest Of Octrees #1009

Forest Of Octrees #1009

lroberts36 commented Feb 22, 2024 •

edited

Loading

pgrete commented Mar 25, 2024

lroberts36 commented Mar 26, 2024

pgrete left a comment

pgrete Mar 29, 2024

pgrete Mar 29, 2024

lroberts36 Apr 1, 2024

brryan commented Apr 1, 2024

brryan left a comment

Forest Of Octrees #1009

Forest Of Octrees #1009

Conversation

lroberts36 commented Feb 22, 2024 • edited Loading

PR Summary

PR Checklist

pgrete commented Mar 25, 2024

lroberts36 commented Mar 26, 2024

pgrete left a comment

Choose a reason for hiding this comment

pgrete Mar 29, 2024

Choose a reason for hiding this comment

pgrete Mar 29, 2024

Choose a reason for hiding this comment

lroberts36 Apr 1, 2024

Choose a reason for hiding this comment

brryan commented Apr 1, 2024

brryan left a comment

Choose a reason for hiding this comment

lroberts36 commented Feb 22, 2024 •

edited

Loading