Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase code coverage in testing #22

Open
fmalatino opened this issue May 31, 2024 · 2 comments
Open

Increase code coverage in testing #22

fmalatino opened this issue May 31, 2024 · 2 comments
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@fmalatino
Copy link
Contributor

Is your feature request related to a problem? Please describe.
Through the use of coverage current code coverage in test suite is 33%. Coverage of stencils should optimally be at ~75%
Output from coverage:

Name                                                       Stmts   Miss Branch BrPart  Cover
--------------------------------------------------------------------------------------------
pyFV3/_config.py                                             281     42     84      2    80%
pyFV3/dycore_state.py                                        120     39     52      1    64%
pyFV3/initialization/analytic_init.py                         20     11      6      0    35%
pyFV3/initialization/init_utils.py                           139     92     12      0    31%
pyFV3/initialization/test_cases/initialize_baroclinic.py      89     67      4      0    24%
pyFV3/initialization/test_cases/initialize_tc.py             164    164      2      0     0%
pyFV3/stencils/a2b_ord4.py                                   252    125     98     12    47%
pyFV3/stencils/c_sw.py                                       174    120     68      3    24%
pyFV3/stencils/d2a2c_vect.py                                 252    136    112      8    39%
pyFV3/stencils/d_sw.py                                       323    132    140     18    51%
pyFV3/stencils/del2cubed.py                                   90     45     40      0    36%
pyFV3/stencils/delnflux.py                                   252    129    155      8    37%
pyFV3/stencils/divergence_damping.py                         201     68     74      7    57%
pyFV3/stencils/dyn_core.py                                   277    122    114     11    46%
pyFV3/stencils/fillz.py                                       78     57     50      0    21%
pyFV3/stencils/fv_dynamics.py                                188    145     80      0    18%
pyFV3/stencils/fv_subgridz.py                                377    331     90      0    14%
pyFV3/stencils/fvtp2d.py                                      83     16     24      2    74%
pyFV3/stencils/fxadv.py                                      141     81     66      3    30%
pyFV3/stencils/map_single.py                                  59     21     16      0    59%
pyFV3/stencils/mapn_tracer.py                                 24      1      8      2    91%
pyFV3/stencils/moist_cv.py                                    52     33     20      0    43%
pyFV3/stencils/neg_adj3.py                                   188    169    106      0     8%
pyFV3/stencils/nh_p_grad.py                                   51     22     10      0    48%
pyFV3/stencils/pe_halo.py                                     11      8     10      0    14%
pyFV3/stencils/pk3_halo.py                                    24     10     10      0    41%
pyFV3/stencils/ppm.py                                         40     26     20      0    30%
pyFV3/stencils/ray_fast.py                                    94     67     84      0    21%
pyFV3/stencils/remap_profile.py                              318    276    220      2    11%
pyFV3/stencils/remapping.py                                  173     77     88      8    41%
pyFV3/stencils/riem_solver3.py                                74     45     30      1    31%
pyFV3/stencils/riem_solver_c.py                               57     31     28      0    33%
pyFV3/stencils/saturation_adjustment.py                      525    400    200      1    30%
pyFV3/stencils/sim1_solver.py                                 77     64     58      0    11%
pyFV3/stencils/temperature_adjust.py                          18     13     10      0    18%
pyFV3/stencils/tracer_2d_1l.py                                94     71     36      0    21%
pyFV3/stencils/updatedzc.py                                   72     36     28      1    45%
pyFV3/stencils/updatedzd.py                                   84     31     24      1    54%
pyFV3/stencils/xppm.py                                       169    122     78      3    32%
pyFV3/stencils/xtp_u.py                                       35     27     16      0    24%
pyFV3/stencils/yppm.py                                       170    122     78      3    33%
pyFV3/stencils/ytp_v.py                                       35     27     16      0    24%
pyFV3/testing/map_single.py                                   15     15      2      0     0%
pyFV3/testing/translate_dyncore.py                            55      3     16      1    94%
pyFV3/testing/translate_fvdynamics.py                         81     52     24      0    30%
pyFV3/testing/validation.py                                   89     62     20      0    27%
pyFV3/utils/functional_validation.py                          31     25      4      0    17%
pyFV3/version.py                                               2      0      0      0   100%
pyFV3/wrappers/geos_wrapper.py                               275    275     32      0     0%
--------------------------------------------------------------------------------------------
TOTAL                                                       6493   4053   2563     98    33%

Describe the solution you'd like
Additional tests to increase coverage

@fmalatino fmalatino added enhancement New feature or request help wanted Extra attention is needed labels May 31, 2024
@FlorianDeconinck
Copy link
Collaborator

Those results are surprising, considering that they were generated with Acoustics which touches upon most of the code. The next steps are:

  • Run FVDynamics which is exactly all the dycore
  • Dive in some of the more central test and understand why the discrepancy (could be configuration, lack of data, etc.)

Ultimately, we should tie a coverage to a dataset so we now how to reach global coverage.

@FlorianDeconinck
Copy link
Collaborator

Thinking more about this, it actually makes sense.

The way the code works we never "execute" the stencil code declared under gt4py frontend. Instead we code generate then we jump over to the generate code in the .gt_cache_X folders. This means the coverage will always see those code as "uncovered".

Partial fix is to include in the coverage the cache directory. Partial solution, because we still want the coverage to look at what code is being touched overall in-between stencils. Also __INLINED and other code culling system within the stack will chop away code and will remove use from being able to have coverage look at them. Maybe this can only be fixed by custom implementation of coverage tool from with gt4py and/or orchestration.

This issue will be worse under orchestration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants