Can CF offer better support for the representation of climate anomalies?

[TomLav]

Topic for discussion

We continue the discussion started as issue #252.

A summary of the discussion so far is in this post.

Tagging @JonathanGregory, @davidhassell and @sethmcg.

[sethmcg]

Thanks for the great summary and for starting this discussion, @TomLav!

In response to your questions:

Do we have a shared understanding of this thread so far (my partial summary)?
Do we agree that we have identified a shortcoming in the way CF currently handles anomalies and their link to climatologies?
Would it be of interest to try and propose a solution for that particular issue?
Would it be best to continue the discussion in this thread, or start in another one?

I think your summary is an excellent starting point, I agree that we have identified a shortcoming, and I think that we should try to propose a solution for it here in this discussion.

@sethmcg do you want to move your questions forward in parallel? Or do we first try and "fix" the first issue at hand?

I would like to move those questions forward as part of this discussion, because I fear that if we don't keep them in mind in trying to address the first issue, we may inadvertently cut ourselves off from a solution to the full set of issues.

So, to summarize my last two posts: currently, the way we use cell_methods and climatology records the periodicity and coverage of the data you started with and how you aggregated it, but it doesn't say what the periodicity and coverage of the result is; that's recorded implicitly in the values of the time coordinate. This creates ambiguity and makes it hard to interpret the results if you aggregate more than once. This should be easy to fix by also explicitly recording what the result was, instead of just leaving it implicit. However, is there one more ambiguity, which is that if you have both cell_methods and climatology, you don't know which order they were applied in. So we need to record that, too.

(And once we start considering how to do that, it starts to look like maybe we shouldn't be thinking of climatology as something different and separate from a cell_method; they're both just aggregations.)

[TomLav]

@davidhassell : you mentioned (here) you had some comments you would to make on this topic?

[TomLav]

Hello. This discussion has not progressed, probably by lack of time before the summer and the summer break.

From experience with past CF discussions, what is the best way to move such things forward? Ultimately we want to propose improvements to the CF convention (a new section?) to help better describe anomalies. Should this be done through online meetings? I am tagging here the people who showed interest so far: @davidhassell @sethmcg @JonathanGregory

[sethmcg]

My suggestion would be to float an idea. People often need something to react to to get things going.

So: if we were starting from scratch, I think the way to do this would be to have an attribute named derivation that listed, in order, the transformations that were applied to the data. So you could have something like:
tmax80reganom@derivation = "interpolate from native grid to 0.25 degree; daily maximum from hourly; monthly average from daily; 80th percentile over region; monthly climatology from monthly across years 1980-2010; monthly anomaly from monthly climatology"

That's an extreme example, but would something like that capture all the information we want? How would it relate to @cell_methods and @climatology? Is it extensible? Does it invalidate old data? What do people think, is this the general direction to try to go in?

[sethmcg]

A quick addendum to this suggestion, prompted by discussion in #341:

It would probably make things significantly more human-interpretable if, rather than using a single string with a delimiter as in the example I provided, we used a string array to record a sequence of transformations in a derivation attribute. I.e.:

tmax80reganom@derivation = "interpolate from native grid to 0.25 degree",  
                           "daily maximum from hourly",
                           "monthly average from daily",  
                           "80th percentile over region",  
                           "monthly climatology from monthly across years 1980-2010",  
                           "monthly anomaly from monthly climatology"

So assuming that we don't end up with a total ban on string arrays, I think we should consider that as a possible element of the solution.

[TomLav]

Thanks for moving this forward. I can see that derivation gives a useful human-readable description of what was done with the data.

But what about the machine-readability? Would you aim at machine readability (parsing, grammar, etc...) as well ?

Also, if a variable holds two climatology maps (dims: lat, lon, nb_clim=2): one against 1981-2010 and the other against 1991-2020). Then I am not sure how to adapt the derivation idea.

[sethmcg]

What do we mean by machine-readability? It would be nice to have, but I'm not sure it's possible. For example, one of the transformations we might want to record is bias-correcting the data, which can be a very complex operation like using the MBCn algorithm (which does multi-variable trend-preserving quantile mapping in an iterative loop that randomly rotates and un-rotates the variables before and after). What does it mean to record that operation in a machine-readable way?

Good point about climatology maps over different periods. Definitely something we need to discuss.

I think what's happening in general terms is that we're talking about a variable that has a dimension that corresponds to variations on the chain of derivation. It would be similar to a variable that had a quantile dimension. One of the issues that led to this discussion is the fact that the existing mechanisms don't fully capture the way that the transformations affect the dimensions of the data.

So I think that means we need to flesh out the idea I've floated a bit more. Currently it describes what the transformations do in human-readable terms, but it doesn't formally record the changes to dimensionality. That example is pseudo-code, basically. If we developed a notation for how the dimensions change (time dimension at daily frequency => climatological time dimension over different periods), do you think that would cover it?

[larsbarring]

Hello @TomLav
Thanks for resurrecting this issue, and thanks @sethmcg for initial suggestion. This is very timely and highly relevant in relation to the CF2024 workshop in less than a month. In fact there is a specific session where this topic fits perfectly:

Tuesday 17 September 16:00 - 17:30 CEST (UTC+2): Metadata for describing statistical processing

It would be excellent if you were able to join this session (as well as the whole workshop of course). Moreover, if you have more concrete ideas that you would like to work on hackathon-style (maybe as an outcome of the conversation during the session) do not hesitate to suggest a hackathon theme in this discussion topic.

[sethmcg]

I can't make the rest of the workshop, but I plan to (virtually) attend that session!

[larsbarring]

Great! Just friendly nudge to register here even if you plan to attend virtually for just one session. This will help us allocate "connection space" or whatever our IT/web support call it.

Thanks /Lars

[sethmcg]

Already done!

[DanHollis]

I have some interest in this topic. From scanning the discussions so far it seems like this could get pretty complicated! My initial thoughts are that there are three things we need to capture:

• The aggregation process that was used to compute the variable (captured by the existing cell_methods, I think)
• The aggregation process that was used to compute the climatology (this could be captured by a second 'cell_methods' that is part of a dummy climatology variable)
• The method by which the data variable and the climatology are combined to obtain the anomaly e.g. difference or ratio (this could be captured by the suggested derivation attribute)

[DanHollis]

Here's a concrete (and fairly simple) use case, in case it helps with the discussion.

I work with gridded datasets derived from in situ station observations. One example of a parameter we routinely generate is the monthly mean of daily maximum temperature. Our grids for individual months (e.g. July 2024) have cell_methods="time: maximum within days time: mean over days". The corresponding climatology (in our case, the 1991-2020 climate normal for July) has cell_methods="time: maximum within days time: mean over days time: mean over years". For the anomaly we calculate the difference between the individual month and the climate normal. For other variables, e.g. precipitation, we express the monthly value as a percentage of the climate normal.

A couple of things occured to me while writing this:

The fact that these data are computed from daily maximum temperatures is obviously a key part of the description. However it is not actually a processing step that data providers such as myself would ever undertake. It is either a characteristic of the observation itself (in the case of liquid-in-glass thermometers) or it is a function of the data logger (in the case of electrical resistance thermometers).

Secondly, in our case both the gridding process and the calculation of the anomaly come after the aggregation steps described in the cell methods have been applied to the station data. However, other approaches are equally valid e.g. creating a separate grid for each day from the station maxima and then creating the monthly grid by averaging the daily grids. In this case 'gridding' comes between time: maximum within days and time: mean over days.

[larsbarring]

For the use case of maximum temperature anomaly, what standard name are you using for the anomaly (there is the air_temperature_anomaly that I think is applicable)? In in relation to this there is an open issue cf-convention/vocabularies#27 on specifying the reference_epochfor anomaly terms. But this, I gather, only covers the situation that the same dataset is used to calculate the reference climatology.

[DanHollis]

To be honest we have been fairly lazy in this regard. All of the data we save to disk are actuals rather than anomalies, so we don't have an issue there. The main time that we need grids of anomalies is when we are generating maps - this is done on-the-fly i.e. the output is an image, not a netCDF file. To achieve this we add our own bespoke attribute to the data in memory to label it as anomalies (the standard name does not change).

Clearly we could use air_temperature_anomaly. However, we process quite a few other variables as well (e.g. precipitation amount, sunshine duration, frost days, rainfall days, wind speed) and, as this thread is highlighting, we'd need more than new standard names to fully describe the data. I haven't had time to pursue this, plus I didn't really have a strong requirement (because we are not saving the anomalies to disk), so we've just improvised our own solution sufficient for our needs.

Having said all that, some months ago we did have a request from an internal user for netCDF files of anomaly values. If this discussion leads to a way to make these files CF compliant then that will be great from our point of view.

One thought regarding machine-readability - We use our bespoke attribute to distinguish actuals from anomalies and, therefore, to create appropriate titles and legends for our maps. I guess new standard names will be required anyway, however I wouldn't want to be parsing the standard name to work out if the file contained anomalies. If the eventual solution can provide a machine-readable way to identify the file contents as anomalies then that would allow us to drop our bespoke attribute.

[sethmcg]

@DanHollis -- Thanks for those comments! Those are very helpful examples and really clarified for me what it is we are trying to accomplish here.

The reason why we need to record the sequence of transformation applied to a variable is not to record the provenance in a way that lets you perfectly recreate the workflow that generated the file, but to record the chain of derivation that changes the meaning of the data.

So the point is to have something that will let you clearly distinguish between things like "climatological mean of the monthly maximum of tmin" and "climatological minimum of the monthly mean of tmax" that can be confusing or ambiguous under the current representation scheme that uses only standard_name, cell_bounds, and climatology attributes.

That's a very helpful reduction in scope, because it dramatically simplifies what we need to record, and I think makes it far more achievable to come up with something that is both human- and machine-readable. For something like a bias-correction transformation, we don't need to record all the details that would let you reproduce it, just that bias correction has happened. Ancillary information like (type of) algorithm, observational dataset, and reference period are important, but we don't need things like implementation details, library versions, parameters, etc.

Great! I think that's doable.

[DanHollis]

...to record the chain of derivation that changes the meaning of the data.

Yes, definitely! In the example I gave the resultant quantity is actually the same (monthly mean of daily maxima) however changing the order of operation ('averaging then gridding' vs 'gridding then averaging') will inevitably produce a different result. In your example the resultant quantities are different things, so to describe them unambiguously is essential. I don't know if this distinction is important but I thought I'd mention it - I imagine any flexible solution will handle both scenarios.

I agree that the aim should not be complete reproducibility. Instead we are just helping data users to understand whether they are comparing apples with apples or apples with pears (which fundamentally is what CF is all about).