Skip to content

Commit b9dac91

Browse files
SvenKlaassenSvenKlaassen
committed
rerun did multi sim
1 parent fbf8975 commit b9dac91

File tree

6 files changed

+213
-194
lines changed

6 files changed

+213
-194
lines changed

‎doc/did/did_multi.qmd

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,13 @@ def make_pretty(df, level, n_rep):
6262

6363
## ATTE Coverage
6464

65-
The simulations are based on the the [make_did_SZ2020](https://docs.doubleml.org/stable/api/generated/doubleml.datasets.make_did_SZ2020.html)-DGP with $1000$ observations. Learners are only set to boosting, due to time constraints (and the nonlinearity of some of the DGPs).
65+
The simulations are based on the the [make_did_CS2021](https://docs.doubleml.org/dev/api/generated/doubleml.did.datasets.make_did_CS2021.html)-DGP with $2000$ observations. Learners are both set to either boosting or a linear (logistic) model. Due to time constraints we only consider the following DGPs:
66+
67+
- Type 1: Linear outcome model and treatment assignment
68+
- Type 4: Nonlinear outcome model and treatment assignment
69+
- Type 6: Randomized treatment assignment and nonlinear outcome model
70+
71+
The non-uniform results (coverage, ci length and bias) refer to averaged values over all $ATTs$ (point-wise confidende intervals).
6672

6773
::: {.callout-note title="Metadata" collapse="true"}
6874

@@ -110,6 +116,8 @@ make_pretty(df_ate_9, level, n_rep)
110116

111117
### Experimental Score
112118

119+
The results are only valid for the DGP 6, as the experimental score assumes a randomized treatment assignment.
120+
113121
```{python}
114122
#| echo: false
115123
score = "experimental"
@@ -130,6 +138,10 @@ make_pretty(df_ate_9, level, n_rep)
130138

131139
## Aggregated Effects
132140

141+
These simulations test different types of aggregation, as described in [DiD User Guide](https://docs.doubleml.org/dev/guide/models.html#difference-in-differences-models-did).
142+
143+
The non-uniform results (coverage, ci length and bias) refer to averaged values over all $ATTs$ (point-wise confidende intervals).
144+
133145
### Group Effects
134146

135147
```{python}
@@ -166,6 +178,8 @@ make_pretty(df_ate_9, level, n_rep)
166178

167179
#### Experimental Score
168180

181+
The results are only valid for the DGP 6, as the experimental score assumes a randomized treatment assignment.
182+
169183
```{python}
170184
#| echo: false
171185
score = "experimental"
@@ -220,6 +234,8 @@ make_pretty(df_ate_9, level, n_rep)
220234

221235
#### Experimental Score
222236

237+
The results are only valid for the DGP 6, as the experimental score assumes a randomized treatment assignment.
238+
223239
```{python}
224240
#| echo: false
225241
score = "experimental"
@@ -274,6 +290,9 @@ make_pretty(df_ate_9, level, n_rep)
274290

275291
#### Experimental Score
276292

293+
The results are only valid for the DGP 6, as the experimental score assumes a randomized treatment assignment.
294+
295+
277296
```{python}
278297
#| echo: false
279298
score = "experimental"

‎results/did/did_multi_detailed.csv

Lines changed: 48 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -1,49 +1,49 @@
11
Learner g,Learner m,Score,In-sample-norm.,DGP,level,Coverage,CI Length,Bias,Uniform Coverage,Uniform CI Length,repetition
2-
LGBM,LGBM,experimental,False,1,0.9,0.40532081377151796,0.6677253898222073,0.45051314831034145,0.08450704225352113,1.0005469053604612,213
3-
LGBM,LGBM,experimental,False,1,0.95,0.5007824726134585,0.7956438762517957,0.45051314831034145,0.15492957746478872,1.1095167204679879,213
4-
LGBM,LGBM,experimental,False,4,0.9,0.548904538341158,0.5814196300250507,0.3237627211845255,0.24413145539906103,0.8948945048168284,213
5-
LGBM,LGBM,experimental,False,4,0.95,0.6381064162754303,0.6928042204373742,0.3237627211845255,0.3333333333333333,0.9852933278894489,213
6-
LGBM,LGBM,experimental,False,6,0.9,0.8974960876369327,0.5798375844479007,0.14088912748925403,0.8685446009389671,0.8915452790934116,213
7-
LGBM,LGBM,experimental,False,6,0.95,0.9464006259780908,0.6909190968602316,0.14088912748925403,0.9436619718309859,0.9825229360947381,213
8-
LGBM,LGBM,experimental,True,1,0.9,0.4123630672926447,0.6676885234807931,0.4519262895879074,0.06103286384976526,1.0019752546929692,213
9-
LGBM,LGBM,experimental,True,1,0.95,0.5066510172143975,0.7955999472965198,0.4519262895879074,0.1596244131455399,1.1110236525840373,213
10-
LGBM,LGBM,experimental,True,4,0.9,0.548904538341158,0.5813589039575363,0.3235983019273521,0.22065727699530516,0.8953221615501692,213
11-
LGBM,LGBM,experimental,True,4,0.95,0.63302034428795,0.6927318608648866,0.3235983019273521,0.3333333333333333,0.9865207873010946,213
12-
LGBM,LGBM,experimental,True,6,0.9,0.8959311424100157,0.5799660631056092,0.14000216987912564,0.892018779342723,0.892440611190863,213
13-
LGBM,LGBM,experimental,True,6,0.95,0.9471830985915493,0.6910721886233918,0.14000216987912564,0.9389671361502347,0.9816012226010797,213
14-
LGBM,LGBM,observational,False,1,0.9,0.9080594679186228,2.7888919178439227,0.7035607423224498,0.9483568075117371,4.334054524286509,213
15-
LGBM,LGBM,observational,False,1,0.95,0.9636150234741784,3.3231696888918334,0.7035607423224498,0.971830985915493,4.758567521024827,213
16-
LGBM,LGBM,observational,False,4,0.9,0.8986697965571204,3.4956058606970917,0.9669737969666872,0.9624413145539906,5.377062497216683,213
17-
LGBM,LGBM,observational,False,4,0.95,0.9643974960876369,4.165271291532112,0.9669737969666872,0.9859154929577465,5.921434465221159,213
18-
LGBM,LGBM,observational,False,6,0.9,0.9225352112676056,2.190208929252288,0.5167519307047893,0.9671361502347418,3.4154051265476033,213
19-
LGBM,LGBM,observational,False,6,0.95,0.969092331768388,2.6097949079569784,0.5167519307047893,0.9859154929577465,3.7483564185778286,213
20-
LGBM,LGBM,observational,True,1,0.9,0.9064945226917058,1.1368726933892221,0.2783026371207721,0.9389671361502347,1.7717588062983114,213
21-
LGBM,LGBM,observational,True,1,0.95,0.9620500782472613,1.3546673682932289,0.2783026371207721,0.9671361502347418,1.9429676563761076,213
22-
LGBM,LGBM,observational,True,4,0.9,0.9158841940532082,1.4254308754317884,0.33866283883400894,0.9014084507042254,2.202675700642339,213
23-
LGBM,LGBM,observational,True,4,0.95,0.9616588419405321,1.6985056496945854,0.33866283883400894,0.9530516431924883,2.421762282833961,213
24-
LGBM,LGBM,observational,True,6,0.9,0.9151017214397495,1.0463068305990617,0.24826243692564173,0.92018779342723,1.636975223567781,213
25-
LGBM,LGBM,observational,True,6,0.95,0.9604851330203443,1.246751486667643,0.24826243692564173,0.9765258215962441,1.7956888077887063,213
26-
Linear,Linear,experimental,False,1,0.9,0.8615023474178404,0.29480782482194634,0.0786421880740863,0.812206572769953,0.45940764438954385,213
27-
Linear,Linear,experimental,False,1,0.95,0.926056338028169,0.3512851900886226,0.0786421880740863,0.8826291079812206,0.5045899270443839,213
28-
Linear,Linear,experimental,False,4,0.9,0.28482003129890454,0.975894733280616,0.8330978501200553,0.018779342723004695,1.412718271492504,213
29-
Linear,Linear,experimental,False,4,0.95,0.3658059467918623,1.1628502977965955,0.8330978501200553,0.07511737089201878,1.575365705490123,213
30-
Linear,Linear,experimental,False,6,0.9,0.9049295774647887,0.9845361877515872,0.23869850791621902,0.8967136150234741,1.4224342348752608,213
31-
Linear,Linear,experimental,False,6,0.95,0.9483568075117371,1.17314722589988,0.23869850791621902,0.9530516431924883,1.5861522128436303,213
32-
Linear,Linear,experimental,True,1,0.9,0.8630672926447575,0.2947851759266125,0.07861158483452843,0.812206572769953,0.4597746703234241,213
33-
Linear,Linear,experimental,True,1,0.95,0.9264475743348983,0.35125820226525856,0.07861158483452843,0.892018779342723,0.5049192522960325,213
34-
Linear,Linear,experimental,True,4,0.9,0.2895148669796557,0.9759084550161325,0.8326923121552289,0.023474178403755867,1.4121969723711594,213
35-
Linear,Linear,experimental,True,4,0.95,0.365414710485133,1.1628666482529382,0.8326923121552289,0.07042253521126761,1.5761780385299982,213
36-
Linear,Linear,experimental,True,6,0.9,0.8986697965571204,0.9845444925351251,0.23872704179656282,0.892018779342723,1.4218091878908978,213
37-
Linear,Linear,experimental,True,6,0.95,0.9483568075117371,1.1731571216598229,0.23872704179656282,0.9483568075117371,1.5857684237069227,213
38-
Linear,Linear,observational,False,1,0.9,0.9119718309859155,0.31913264481306997,0.07504846382409917,0.9530516431924883,0.4966577096852585,213
39-
Linear,Linear,observational,False,1,0.95,0.9581377151799687,0.3802700008534462,0.07504846382409917,0.971830985915493,0.5457965923475415,213
40-
Linear,Linear,observational,False,4,0.9,0.4061032863849765,1.2274527144295637,0.8093030821575489,0.17370892018779344,1.7566478372130891,213
41-
Linear,Linear,observational,False,4,0.95,0.5093896713615024,1.4626001205144619,0.8093030821575489,0.27699530516431925,1.9631223954139048,213
42-
Linear,Linear,observational,False,6,0.9,0.9021909233176838,1.0316221276603512,0.24917990085463324,0.9061032863849765,1.488738970418394,213
43-
Linear,Linear,observational,False,6,0.95,0.954225352112676,1.2292535838683007,0.24917990085463324,0.9342723004694836,1.6622175036443294,213
44-
Linear,Linear,observational,True,1,0.9,0.9092331768388106,0.3171256867784898,0.07481042183312375,0.9248826291079812,0.493585500375777,213
45-
Linear,Linear,observational,True,1,0.95,0.9585289514866979,0.377878562854461,0.07481042183312375,0.9671361502347418,0.5423776539443926,213
46-
Linear,Linear,observational,True,4,0.9,0.3974960876369327,1.2240896932129774,0.8120340183488837,0.16901408450704225,1.7523471284937546,213
47-
Linear,Linear,observational,True,4,0.95,0.5003912363067292,1.4585928335706568,0.8120340183488837,0.27230046948356806,1.9580510608715704,213
48-
Linear,Linear,observational,True,6,0.9,0.9025821596244131,1.0267883142861962,0.25013308990604005,0.8967136150234741,1.4811458752504516,213
49-
Linear,Linear,observational,True,6,0.95,0.9503129890453834,1.223493739973321,0.25013308990604005,0.9483568075117371,1.6532118883170046,213
2+
LGBM,LGBM,experimental,False,1,0.9,0.4075,0.6680449671616123,0.4536971478292392,0.106,1.0014645995766545,1000
3+
LGBM,LGBM,experimental,False,1,0.95,0.4969166666666667,0.796024676138938,0.4536971478292392,0.149,1.1101910211253438,1000
4+
LGBM,LGBM,experimental,False,4,0.9,0.5299166666666666,0.5832956540561086,0.33011501262366927,0.221,0.8979294468476701,1000
5+
LGBM,LGBM,experimental,False,4,0.95,0.6176666666666666,0.695039640948207,0.33011501262366927,0.302,0.9884075405086947,1000
6+
LGBM,LGBM,experimental,False,6,0.9,0.89775,0.5802322315200278,0.1421029922499711,0.891,0.8927725918322601,1000
7+
LGBM,LGBM,experimental,False,6,0.95,0.9469166666666666,0.69138934785114,0.1421029922499711,0.948,0.9827083539024966,1000
8+
LGBM,LGBM,experimental,True,1,0.9,0.4046666666666667,0.6678639885704151,0.45277355982980805,0.085,1.0010411944316,1000
9+
LGBM,LGBM,experimental,True,1,0.95,0.4968333333333333,0.7958090268465587,0.45277355982980805,0.143,1.1099170693556415,1000
10+
LGBM,LGBM,experimental,True,4,0.9,0.5294166666666666,0.5833160278609535,0.32982215913570107,0.212,0.8981881991531053,1000
11+
LGBM,LGBM,experimental,True,4,0.95,0.6166666666666666,0.6950639178340466,0.32982215913570107,0.297,0.9891633208675981,1000
12+
LGBM,LGBM,experimental,True,6,0.9,0.89975,0.580193622985627,0.14218113073558258,0.901,0.8924214431904512,1000
13+
LGBM,LGBM,experimental,True,6,0.95,0.9490833333333334,0.691343342944879,0.14218113073558258,0.941,0.9821856487700115,1000
14+
LGBM,LGBM,observational,False,1,0.9,0.9114166666666667,2.745442581507895,0.7067512182439725,0.956,4.267995889646912,1000
15+
LGBM,LGBM,observational,False,1,0.95,0.9645833333333333,3.271396611351424,0.7067512182439725,0.987,4.685018272931908,1000
16+
LGBM,LGBM,observational,False,4,0.9,0.90725,3.522756602113769,0.964041513740829,0.962,5.420041663733542,1000
17+
LGBM,LGBM,observational,False,4,0.95,0.9638333333333333,4.197623395365736,0.964041513740829,0.992,5.964308172387913,1000
18+
LGBM,LGBM,observational,False,6,0.9,0.9243333333333333,2.1873521785120444,0.5149849766795307,0.961,3.4127245573450833,1000
19+
LGBM,LGBM,observational,False,6,0.95,0.96975,2.6063908794939334,0.5149849766795307,0.983,3.744841077573908,1000
20+
LGBM,LGBM,observational,True,1,0.9,0.9086666666666666,1.1201454805349786,0.27275770285583456,0.928,1.746888265672863,1000
21+
LGBM,LGBM,observational,True,1,0.95,0.9575833333333333,1.334735664815871,0.27275770285583456,0.973,1.916447134945182,1000
22+
LGBM,LGBM,observational,True,4,0.9,0.9208333333333334,1.415823117191095,0.3279182075028663,0.928,2.1893552276764523,1000
23+
LGBM,LGBM,observational,True,4,0.95,0.9615833333333333,1.6870572996314692,0.3279182075028663,0.968,2.408008384167633,1000
24+
LGBM,LGBM,observational,True,6,0.9,0.9035833333333334,1.0342009611787182,0.25272266117158226,0.925,1.6169600815502123,1000
25+
LGBM,LGBM,observational,True,6,0.95,0.9551666666666666,1.2323264535360365,0.25272266117158226,0.979,1.7730857540946194,1000
26+
Linear,Linear,experimental,False,1,0.9,0.8538333333333333,0.29474516889634966,0.08084244811337131,0.777,0.45978521833536623,1000
27+
Linear,Linear,experimental,False,1,0.95,0.9159166666666666,0.3512105309483955,0.08084244811337131,0.87,0.5045503638584217,1000
28+
Linear,Linear,experimental,False,4,0.9,0.3103333333333333,0.9755748479295066,0.8059915763717771,0.042,1.4116936904642545,1000
29+
Linear,Linear,experimental,False,4,0.95,0.3874166666666667,1.16246913089087,0.8059915763717771,0.078,1.5738462873560166,1000
30+
Linear,Linear,experimental,False,6,0.9,0.905,0.9844757965076787,0.23586020264982355,0.904,1.421142814988169,1000
31+
Linear,Linear,experimental,False,6,0.95,0.9513333333333334,1.1730752652943266,0.23586020264982355,0.957,1.5854008375388589,1000
32+
Linear,Linear,experimental,True,1,0.9,0.8529166666666667,0.2947409209064057,0.08090529833241722,0.773,0.4595811381839562,1000
33+
Linear,Linear,experimental,True,1,0.95,0.9155,0.3512054691561725,0.08090529833241722,0.873,0.5044085624001035,1000
34+
Linear,Linear,experimental,True,4,0.9,0.30975,0.9755981107617252,0.8057657027853717,0.045,1.4127941326439093,1000
35+
Linear,Linear,experimental,True,4,0.95,0.38825,1.1624968502651534,0.8057657027853717,0.082,1.5745211647752397,1000
36+
Linear,Linear,experimental,True,6,0.9,0.90425,0.9845527716230539,0.23562373204232834,0.902,1.4213288951702785,1000
37+
Linear,Linear,experimental,True,6,0.95,0.9529166666666666,1.1731669868015593,0.23562373204232834,0.954,1.5855821588385934,1000
38+
Linear,Linear,observational,False,1,0.9,0.9013333333333333,0.3180945635163335,0.07688429889402026,0.893,0.4947430553763157,1000
39+
Linear,Linear,observational,False,1,0.95,0.9528333333333334,0.379033050694909,0.07688429889402026,0.939,0.543298878262958,1000
40+
Linear,Linear,observational,False,4,0.9,0.41125,1.2376859429116744,0.7929800095241654,0.179,1.7688886396548633,1000
41+
Linear,Linear,observational,False,4,0.95,0.517,1.4747937643389757,0.7929800095241654,0.271,1.9773822754488526,1000
42+
Linear,Linear,observational,False,6,0.9,0.9018333333333334,1.0288570593763655,0.2510929759491061,0.907,1.4841286685672213,1000
43+
Linear,Linear,observational,False,6,0.95,0.951,1.225958801790062,0.2510929759491061,0.952,1.6571596022563468,1000
44+
Linear,Linear,observational,True,1,0.9,0.9031666666666667,0.31609423050346813,0.07696990405262741,0.896,0.4919409315123034,1000
45+
Linear,Linear,observational,True,1,0.95,0.951,0.3766495068962009,0.07696990405262741,0.938,0.540169288119527,1000
46+
Linear,Linear,observational,True,4,0.9,0.4071666666666667,1.2320142323935404,0.7928492798685264,0.177,1.7614082912050193,1000
47+
Linear,Linear,observational,True,4,0.95,0.5160833333333333,1.4680355044159439,0.7928492798685264,0.27,1.9699921213258018,1000
48+
Linear,Linear,observational,True,6,0.9,0.8995833333333334,1.0226446037500223,0.25004589301052516,0.908,1.4775056202619612,1000
49+
Linear,Linear,observational,True,6,0.95,0.9493333333333334,1.2185562043286982,0.25004589301052516,0.958,1.6496021973448636,1000

0 commit comments

Comments
 (0)