N3LO fits with IHOU

[giacomomagni]

This PR is to introduce IHOU + MHOU theory covmat and allow for N3LO theories.

[giacomomagni]

@giacomomagni please remove the changes you did to filters.yaml as they are now in #1835

Thanks, done!

[RoyStegeman]

With #1835 approved, is this also ready to be merged (barring the updating of the theoryids)?

[giacomomagni]

With #1835 approved, is this also ready to be merged (barring the updating of the theoryids)?

Maybe before merging it would be good is someone else try to do one N3LO fit, so we are sure the machinery works also for others...

[RoyStegeman]

Are the theories those corresponding as this PR is right now? And are they on the server?

[giacomomagni]

Are the theories those corresponding as this PR is right now? And are they on the server?

They should

[andreab1997]

@giacomomagni can you post here the runcard you want someone to try before merging this?

[giacomomagni]

you can try:

231106-gm-n3lo-global-th-3pt:

#
# Configuration file for n3fit
#
###################################################################
description: NNPDF4.0 NNLO thcovmat fit with theory 439 (nFONLL), global, 3pt N3LO missing theory covmat 
############################################################
# frac: training fraction
# ewk: apply ewk k-factors
# sys: systematics treatment (see systypes)
dataset_inputs:
- {dataset: NMCPD_dw_ite, frac: 0.75}
- {dataset: NMC, frac: 0.75}
- {dataset: SLACP_dwsh, frac: 0.75}
- {dataset: SLACD_dw_ite, frac: 0.75}
- {dataset: BCDMSP_dwsh, frac: 0.75}
- {dataset: BCDMSD_dw_ite, frac: 0.75}
- {dataset: CHORUSNUPb_dw_ite, frac: 0.75}
- {dataset: CHORUSNBPb_dw_ite, frac: 0.75}
- {dataset: NTVNUDMNFe_dw_ite, cfac: [MAS], frac: 0.75}
- {dataset: NTVNBDMNFe_dw_ite, cfac: [MAS], frac: 0.75}
- {dataset: HERACOMBNCEM, frac: 0.75}
- {dataset: HERACOMBNCEP460, frac: 0.75}
- {dataset: HERACOMBNCEP575, frac: 0.75}
- {dataset: HERACOMBNCEP820, frac: 0.75}
- {dataset: HERACOMBNCEP920, frac: 0.75}
- {dataset: HERACOMBCCEM, frac: 0.75}
- {dataset: HERACOMBCCEP, frac: 0.75}
- {dataset: HERACOMB_SIGMARED_C, frac: 0.75}
- {dataset: HERACOMB_SIGMARED_B, frac: 0.75}
- {dataset: DYE886R_dw_ite, frac: 0.75}
- {dataset: DYE886P, frac: 0.75, cfac: []}
#- {dataset: DYE605_dw_ite, frac: 0.75, cfac: []}
#- {dataset: DYE906R_dw_ite, frac: 0.75, cfac: [ACC]}
- {dataset: CDFZRAP_NEW, frac: 0.75, cfac: []}
- {dataset: D0ZRAP_40, frac: 0.75, cfac: []}
- {dataset: D0WMASY, frac: 0.75, cfac: []}
- {dataset: ATLASWRAP36PB, frac: 0.75, cfac: []}
- {dataset: ATLASZRAP36PB, frac: 0.75, cfac: []}
- {dataset: ATLASZHIGHMASS49FB, frac: 0.75, cfac: []}
- {dataset: ATLASLOMASSDY11EXT, frac: 0.75, cfac: []}
- {dataset: ATLASWRAP11CC, frac: 0.75, cfac: []}
- {dataset: ATLASZRAP11CC, frac: 0.75, cfac: []}
- {dataset: ATLASWZRAP11CF, frac: 0.75, cfac: []}
- {dataset: ATLASDY2D8TEV, frac: 0.75, cfac: []}
- {dataset: ATLAS_DY_2D_8TEV_LOWMASS, frac: 0.75, cfac: []}
- {dataset: ATLAS_Z_TOT_13TEV, frac: 0.75, cfac: [NRM,]}
- {dataset: ATLAS_W_TOT_13TEV, frac: 0.75, cfac: [NRM,]}
- {dataset: ATLAS_WP_JET_8TEV_PT, frac: 0.75, cfac: []}
- {dataset: ATLAS_WM_JET_8TEV_PT, frac: 0.75, cfac: []}
#- {dataset: ATLAS_WCHARM_WP_DIFF_7TEV, frac: 0.75, cfac: []}
#- {dataset: ATLAS_WCHARM_WM_DIFF_7TEV, frac: 0.75, cfac: []}
- {dataset: ATLASZPT8TEVMDIST, frac: 0.75, cfac: []}
- {dataset: ATLASZPT8TEVYDIST, frac: 0.75, cfac: []}
- {dataset: ATLASTTBARTOT7TEV, frac: 0.75, cfac: []}
- {dataset: ATLASTTBARTOT8TEV, frac: 0.75, cfac: []}
- {dataset: ATLAS_TTBARTOT_13TEV_FULLLUMI, frac: 0.75, cfac: []}
- {dataset: ATLAS_TTB_DIFF_8TEV_LJ_TRAPNORM, frac: 0.75, cfac: []}
- {dataset: ATLAS_TTB_DIFF_8TEV_LJ_TTRAPNORM, frac: 0.75, cfac: []}
- {dataset: ATLAS_TOPDIFF_DILEPT_8TEV_TTRAPNORM, frac: 0.75, cfac: []}
- {dataset: ATLAS_1JET_8TEV_R06_DEC, frac: 0.75, cfac: []}
- {dataset: ATLAS_2JET_7TEV_R06, frac: 0.75, cfac: []}
- {dataset: ATLASPHT15_SF, frac: 0.75, cfac: [EWK]}
- {dataset: ATLAS_SINGLETOP_TCH_R_7TEV, frac: 0.75, cfac: []}
- {dataset: ATLAS_SINGLETOP_TCH_R_13TEV, frac: 0.75, cfac: []}
- {dataset: ATLAS_SINGLETOP_TCH_DIFF_7TEV_T_RAP_NORM, frac: 0.75, cfac: []}
- {dataset: ATLAS_SINGLETOP_TCH_DIFF_7TEV_TBAR_RAP_NORM, frac: 0.75, cfac: []}
- {dataset: ATLAS_SINGLETOP_TCH_DIFF_8TEV_T_RAP_NORM, frac: 0.75, cfac: []}
- {dataset: ATLAS_SINGLETOP_TCH_DIFF_8TEV_TBAR_RAP_NORM, frac: 0.75, cfac: []}
- {dataset: CMSWEASY840PB, frac: 0.75, cfac: []}
- {dataset: CMSWMASY47FB, frac: 0.75, cfac: []}
- {dataset: CMSDY2D11, frac: 0.75, cfac: []}
- {dataset: CMSWMU8TEV, frac: 0.75, cfac: []}
- {dataset: CMSZDIFF12, frac: 0.75, cfac: [NRM]}
- {dataset: CMS_2JET_7TEV, frac: 0.75, cfac: []}
- {dataset: CMS_1JET_8TEV, frac: 0.75, cfac: []}
- {dataset: CMSTTBARTOT7TEV, frac: 0.75, cfac: []}
- {dataset: CMSTTBARTOT8TEV, frac: 0.75, cfac: []}
- {dataset: CMSTTBARTOT13TEV, frac: 0.75, cfac: []}
- {dataset: CMSTOPDIFF8TEVTTRAPNORM, frac: 0.75, cfac: []}
- {dataset: CMSTTBARTOT5TEV, frac: 0.75, cfac: []}
- {dataset: CMS_TTBAR_2D_DIFF_MTT_TRAP_NORM, frac: 0.75, cfac: []}
- {dataset: CMS_TTB_DIFF_13TEV_2016_2L_TRAP, frac: 0.75, cfac: []}
- {dataset: CMS_TTB_DIFF_13TEV_2016_LJ_TRAP, frac: 0.75, cfac: []}
- {dataset: CMS_SINGLETOP_TCH_TOT_7TEV, frac: 0.75, cfac: []}
- {dataset: CMS_SINGLETOP_TCH_R_8TEV, frac: 0.75, cfac: []}
- {dataset: CMS_SINGLETOP_TCH_R_13TEV, frac: 0.75, cfac: []}
#- {dataset: CMSWCHARMTOT, frac: 0.75}
#- {dataset: CMSWCHARMRAT, frac: 0.75}
#- {dataset: CMS_WCHARM_DIFF_UNNORM_13TEV, frac: 0.75}
- {dataset: LHCBZ940PB, frac: 0.75, cfac: []}
- {dataset: LHCBZEE2FB_40, frac: 0.75, cfac: []}
- {dataset: LHCBWMU7TEV, frac: 0.75, cfac: [NRM,]}
- {dataset: LHCBZMU7TEV, frac: 0.75, cfac: [NRM,]}
- {dataset: LHCBWMU8TEV, frac: 0.75, cfac: [NRM,]}
- {dataset: LHCBZMU8TEV, frac: 0.75, cfac: [NRM,]}
- {dataset: LHCB_Z_13TEV_DIMUON, frac: 0.75, cfac: []}
- {dataset: LHCB_Z_13TEV_DIELECTRON, frac: 0.75, cfac: []}

############################################################
datacuts:  
  t0pdfset: 210715-n3fit-1000-001     # PDF set to generate t0 covmat
  q2min: 13.96                       # Q2 minimum
  w2min: 12.5
  maxTau: 0.020
  use_cuts: internal
############################################################
theory:
  theoryid: 439
theorycovmatconfig:
  point_prescription: "n3lo 3pt hadronic"
  theoryids:
    from_: scale_variation_theories
  pdf: 210715-n3fit-1000-001
  use_thcovmat_in_fitting: true
  use_thcovmat_in_sampling: true
sampling:
  separate_multiplicative: false
     # database id
############################################################
trvlseed: 376191634
nnseed: 2080989803
mcseed: 75955222
save: false
genrep: true      # true = generate MC replicas, false = use real data

parameters: # This defines the parameter dictionary that is passed to the Model Trainer
  nodes_per_layer: [25, 20, 8]
  activation_per_layer: [tanh, tanh, linear]
  initializer: glorot_normal
  optimizer:
    clipnorm: 6.073e-6
    learning_rate: 2.621e-3
    optimizer_name: Nadam
  epochs: 17000
  positivity:
    initial: 184.8
    multiplier:
  integrability:
    initial: 10
    multiplier:
  stopping_patience: 0.1
  layer_type: dense
  dropout: 0.0
  threshold_chi2: 3.5

fitting:
  fitbasis: EVOL  # EVOL (7), EVOLQED (8), etc.
  basis:
  - {fl: sng, trainable: false, smallx: [1.091, 1.119], largex: [1.471, 3.021]}
  - {fl: g, trainable: false, smallx: [0.7795, 1.095], largex: [2.742, 5.547]}
  - {fl: v, trainable: false, smallx: [0.472, 0.7576], largex: [1.571, 3.559]}
  - {fl: v3, trainable: false, smallx: [0.07483, 0.4501], largex: [1.714, 3.467]}
  - {fl: v8, trainable: false, smallx: [0.5731, 0.779], largex: [1.555, 3.465]}
  - {fl: t3, trainable: false, smallx: [-0.5498, 1.0], largex: [1.778, 3.5]}
  - {fl: t8, trainable: false, smallx: [0.5469, 0.857], largex: [1.555, 3.391]}
  - {fl: t15, trainable: false, smallx: [1.081, 1.142], largex: [1.491, 3.092]}

############################################################
positivity:
  posdatasets:
  - {dataset: POSF2U, maxlambda: 1e6}        # Positivity Lagrange Multiplier
  - {dataset: POSF2DW, maxlambda: 1e6}
  - {dataset: POSF2S, maxlambda: 1e6}
  - {dataset: POSFLL, maxlambda: 1e6}
  - {dataset: POSDYU, maxlambda: 1e10}
  - {dataset: POSDYD, maxlambda: 1e10}
  - {dataset: POSDYS, maxlambda: 1e10}
  - {dataset: POSF2C, maxlambda: 1e6}
  - {dataset: POSXUQ, maxlambda: 1e6}        # Positivity of MSbar PDFs
  - {dataset: POSXUB, maxlambda: 1e6}
  - {dataset: POSXDQ, maxlambda: 1e6}
  - {dataset: POSXDB, maxlambda: 1e6}
  - {dataset: POSXSQ, maxlambda: 1e6}
  - {dataset: POSXSB, maxlambda: 1e6}
  - {dataset: POSXGL, maxlambda: 1e6}

############################################################
integrability:
  integdatasets:
  - {dataset: INTEGXT8, maxlambda: 1e2}
  - {dataset: INTEGXT3, maxlambda: 1e2}

############################################################
debug: false
maxcores: 8

[andreab1997]

What is missing here?

[RoyStegeman]

Maybe from 784 onward the "Comments" key in the theory.db should be updated?

[andreab1997]

Just change the comments in the theory.db and I am happy to merge this

[andreab1997]

Ok good we can merge this