Add flag for the derivative of thePDF

[scarlehoff]

This PR adds the derivative flag to validphys which can be used to get derivatives of the xplotting_grids (so it will be used automatically by any PDF plotting routine).

Example: https://vp.nnpdf.science/Tj8jTeWhSki5ObB17nV1BQ==/#pdf-bands-2

[RoyStegeman]

I think eps needs to be scaled as well in case of logs?

[Zaharid]

Not clear to me what do we want to plot here. Is it f(x)? xf(x)? Some logs somewhere?

[RoyStegeman]

Arc length on a log scale. Though in that case I guess we especially don't want to scale eps since that would leave the result unchanged

[scarlehoff]

Yes. This is what I think people has asked for.

But in all honesty it is not 100% clear to me whether I'm understanding it correctly, that's why I wanted some feedback before showing any plots...

[RoyStegeman]

Well what they really wanted is the derivative itself instead of the integral I think, in which cate we would need a plot per PDF

[scarlehoff]

That was yet a different thing, right?

And again, not super clear, they want the derivative of the PDF so basically a plot of d(xf(x))/dx ? or xd(f(x))/dx or d(xf(x))/dlogx?

(that I will just do a simple plot using lhapdf though, unless it turns out to be super important)

[ldd69]

Hi Juan What was in my mail is clear: |df/dx|^2 as a function of x You can add simply df/dx, so that the other quantities can easily be derived using the chain rule. Cheers, L. On 23 Sep 2022, at 12:23, Juan M. Cruz-Martinez ***@***.******@***.***>> wrote: This email was sent to you by someone outside the University. You should only click on links or attachments if you are certain that the email is genuine and the content is safe. That was yet a different thing, right? And again, not super clear, they want the derivative of the PDF so basically a plot of d(xf(x))/dx ? or xd(f(x))/dx or d(xf(x))/dlogx? (that I will just do a simple plot using lhapdf though, unless it turns out to be super important) — Reply to this email directly, view it on GitHub<#1605 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACQVU3UAWD6RFKPJBILOI4LV7WHM7ANCNFSM6AAAAAAQT3N5FQ>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***> The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.

[scarlehoff]

Hi @ldd69, could you resend that email? I've recently changed e-mails and I cannot find it now so maybe it slipped between the cracks. Thanks.

In any case, when you say f do you mean f(x) or xf(x) (we are always plotting xf(x))?

[ldd69]

Hi Juan The previous email is pasted below. The thing I would like to see is the derivative of the quantity that multiplies the FK tables in our parametrisation. So for DIS: F_I = FK_{Ij} f_j Where F_I is the observable. Can you remind me what our convention is? In any case we know that: d\dx (xf(x)) = f(x) + x df/dx Cheers, L. *************************** Previous email: Hello What we need is |df/dx|^2 as a function of x (plot in log scale in x would be better) for each PDF. Cheers, L. On 23 Sep 2022, at 13:28, Juan M. Cruz-Martinez ***@***.******@***.***>> wrote: This email was sent to you by someone outside the University. You should only click on links or attachments if you are certain that the email is genuine and the content is safe. Hi @ldd69<https://github.com/ldd69>, could you resend that email? I've recently changed e-mails and I cannot find it now so maybe it slipped between the cracks. Thanks. In any case, when you say f do you mean f(x) or xf(x) (we are always plotting xf(x))? — Reply to this email directly, view it on GitHub<#1605 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACQVU3WEL7I37LID4K5ZTITV7WO67ANCNFSM6AAAAAAQT3N5FQ>. You are receiving this because you were mentioned.Message ID: ***@***.***> The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.

[scarlehoff]

Our convention is that f_j is xf(x).

[scarlehoff]

I've done the derivative for f_j = xf(x), the result is:

(log scale in both x and y)

For representation purposes I've also done d/dlog(x) so that I could use a linear scale in y:

Not sure which is better. In any case, I'm a bit worried that the results are not very different between CT and NNPDF (below the gluon with a log scale as well for comparison)

I'll do it using vp so that people can audit the code to make sure I didn't made any mistakes. And to facilitate as well plotting the 1-sigma bands for NNPDF instead of raw replicas.

For completeness, the output of the arclength using an xgrid

with respect to using the normal one:

The arclength one I will remove (since as @RoyStegeman said above, it would be inconsistent) and I will add instead the possibility of plotting the derivative with vp as @ldd69 requested then we can play with it as we see fit.

[RoyStegeman]

The arclength one I will remove (since as @RoyStegeman said above, it would be inconsistent)

Not inconsistent per se I think; we're just not calculating the arclength of the PDFs themselves (according to its common definition), but rather of an object that combines the PDFs with a logarithmic scaling of x. It is still an arclength-like feature of the PDF so perhaps it's useful(?)

In particular the first plot is a bit spoiled by numerical issues, but could it be that the second derivative of the NNPDF replicas are pretty much flat, while that of the CT pdfs is not? On the other hand, after the bump between x=1e-5 and 1e-4, also the CT central value seems to have a rather constant second derivative.

[scarlehoff]

I've added a derivative: bool flag which gets read by xplotting_grid so any derived (heh) quantity will receive actually the derivative of the PDF (for instance the PDF plots).

Example report with the df / dlogx for NNPDF and CT with band plots and replica plots:
https://vp.nnpdf.science/jM9wSap9RlyRDRkgbr4Ezg==/

I'm using np.gradient with the plotting x-grid of the PDF and letting vp deal with the rest. I've opted for d/dlog(x) since it worked best with vp (otherwise I would have to deal with the y-scale, maybe differently per-flavour).

Things to-do: the y-axis label needs to depend not only on the PDF flavour but has to allow for some extra information. I'll see how to do that without too much hacking, but it is relatively minor point.

[scarlehoff]

Taking further derivatives is simply a question of changing the flag from a bool to an int:* https://vp.nnpdf.science/hrCzYH0cRJG2N0mT16Xo4w==
but let's first make sure that I didn't make any mistakes in the first one :P

(the changes to vp are actually very minimal at this point so it should be easy on whoever wants to review it!)

*in this report I also removed the central value for ct from the replicas which is misleading, but this needs atm a bit of ugly patching

[Zaharid]

@enocera @scarlehoff how does np.gradient compare with the thing in here #1581 (comment)?

[enocera]

@enocera @scarlehoff how does np.gradient compare with the thing in here #1581 (comment)?

@Zaharid That's an interesting question - I guess that your suggestion would be: if np.gradient does the same thing as in #1581 (comment), why not using np.gradient also in #1581?

[Zaharid]

Yes, that was my thought.

[enocera]

OK. So let me check if I can adapt np.gradient in #1581.

[Zaharid]

I think I'd prefer a separate provider for the derivative grid (and subsequent stuff). For one if you set derivative: True all the labels in all the plots (minus the one being looked at here) become wrong. That smells like it would be handled better by separate code paths.

We do something similar for the distances.

[scarlehoff]

minus the one being looked at here

What do you mean? They should all be wrong.

[Zaharid]

I had this bit in mind

    scl_txt = "" if xscale == "linear" else "log-"

but indeed that is something else.

[scarlehoff]

Ah, yes. I guess the title of the PR is now misleading. I haven't changed the labels, only added the possibility of generating plots with the derivatives.

[scarlehoff]

Since there are no complains, I think I will hack together something for the labels and then send an updated version of https://vp.nnpdf.science/hrCzYH0cRJG2N0mT16Xo4w==/ to the mailing list.

[RoyStegeman]

Maybe it's also worth including the log-scaled arclength? At least there you can see a bit of a difference between ct an nnpdf40 (though it's not so clear ot me if it's coming from outside the data region or instead just corresponds to the behavior we see between 1e-2 and 1e-4)

[scarlehoff]

it's also not clear to me and honestly woudn't know how to justify it at all... if some insight can be obtained from the derivatives I would prefer that since at least I understand it.

[RoyStegeman]

Fair enough

[scarlehoff]

It is actually not so hacky, kudos to the PDFPlotter + Xplottinggrids thing, it's much more flexible and accommodating than I thought.

https://vp.nnpdf.science/Tj8jTeWhSki5ObB17nV1BQ==/#pdf-bands-2

[scarlehoff]

I've removed the changes to arclength and left only the possibility of having the derivative of plots. Please review when possible.

[ldd69]

Hello What we need is |df/dx|^2 as a function of x (plot in log scale in x would be better) for each PDF. Cheers, L.

…

-- Prof Luigi Del Debbio Chair of Theoretical High-Energy Physics School of Physics and Astronomy The University of Edinburgh Peter Guthrie Tait Road Edinburgh EH9 3FD Ph: +44 (0)131 650 5212 From: Roy Stegeman ***@***.***> Date: Friday, 23 September 2022 at 12:03 To: NNPDF/nnpdf ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [NNPDF/nnpdf] [WIP] Kinetic arclength (PR #1605) This email was sent to you by someone outside the University. You should only click on links or attachments if you are certain that the email is genuine and the content is safe. Well what they really wanted is the derivative itself instead of the integral I think, in which cate we would need a plot per PDF — Reply to this email directly, view it on GitHub<#1605 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACQVU3QDSA23MXLZLN652R3V7WE5DANCNFSM6AAAAAAQT3N5FQ>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***> The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.

[ldd69]

Very good! Then we can work directly with xf(x), which is what we see in the plots anyway. So we want d\dx (xf(x)) and |d\dx (xf(x))|^2 as functions of x. Thanks for clarifying! L. On 23 Sep 2022, at 15:03, Juan M. Cruz-Martinez ***@***.******@***.***>> wrote: This email was sent to you by someone outside the University. You should only click on links or attachments if you are certain that the email is genuine and the content is safe. Our convention is that f_j is xf(x). — Reply to this email directly, view it on GitHub<#1605 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACQVU3SMQVNN6WQB2P65D6DV7W2BTANCNFSM6AAAAAAQT3N5FQ>. You are receiving this because you were mentioned.Message ID: ***@***.***> The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.

[RoyStegeman]

Logic seems fine, just needs the one change.

[scarlehoff]

I'll merge this once the test is done. Thanks for the review @RoyStegeman