From 24a9091c1387bcf58928fcdfed2befbde0851b3a Mon Sep 17 00:00:00 2001 From: "Amir H. Jadidinejad" Date: Sun, 23 May 2021 14:37:06 +0100 Subject: [PATCH 1/8] added a new tutorial to introduce PSE --- tutorials/README.md | 5 ++ tutorials/closed_loop_diagram.jpg | Bin 0 -> 60334 bytes tutorials/propensity_stratified_evaluation.md | 68 ++++++++++++++++++ 3 files changed, 73 insertions(+) create mode 100644 tutorials/closed_loop_diagram.jpg create mode 100644 tutorials/propensity_stratified_evaluation.md diff --git a/tutorials/README.md b/tutorials/README.md index 20ada5454..17c74947c 100644 --- a/tutorials/README.md +++ b/tutorials/README.md @@ -22,3 +22,8 @@ An important focus of Cornac is making it convenient to work with auxiliary data - [Working with auxiliary data](./working_with_auxiliary_data.md) - [Text to Graph transformation](./text_to_graph.ipynb) - [Visual recommendation algorithm with text data](./vbpr_text.ipynb) + +## Stratified Evaluation + Most datasets in the field of recommendation systems (MovieLens, Netflix,...) suffer from [closed loop feedback](https://dl.acm.org/doi/10.1145/3397271.3401230). In this situation, [Jadidinejad et al.](https://arxiv.org/abs/2104.08912) proposed a stratified evaluation method to better estimate the performance of different recommendation models. + + - [Stratified Evaluation](./propensity_stratified_evaluation.md) \ No newline at end of file diff --git a/tutorials/closed_loop_diagram.jpg b/tutorials/closed_loop_diagram.jpg new file mode 100644 index 0000000000000000000000000000000000000000..7d8117084a242e83987debf56038e197cc96639e GIT binary patch literal 60334 zcmeFZcU%)sv^TsV^xmX5DIy)|Js{FV1VK88^d`LpM1mk7MNkk>EC?uw2%#4#p@X0z zMUYN{f=V%=glH0Y7k}m6_dfsJ&+~cT=bz_}?(R2d&&-^2W_I>FGpk3Rk5&NI^TuYz z00aU6rr;mo2n~1{!To&zz`_EM1^|E-poCBZP!K@?3h>1KHwH66PXC8I%^#rrqXPhX z;1vKUo>BZGf$l$p{(~Vuj&B~J=}!7ZMuuuBC0yS)0$SaD1 zxx>ADw5-n=|D6~7r7QZkkufnb@-fQtA>qCXXEZf66%>^el$7K^4Y`Qe;7E74TyTWg zKP8;=j_?fk4~_H>2^Kjn(cL2?DpFTeQ~|6(;qNMr;lH-(iTqFcp9uU<1pX%i{}X}# ziNOCv;Qzh|{0H0d4hAVs49I1Gqb)$g$}c1`B*HHwR76=`5zshmWT|%Jv8(h1uFmq1xJQko17JK zaC8!(`3e#^0e~4`1gP9SBSH)glu4eU;gJA9eGZh1#YBc4V?PjchJyxzIQ|&>{)O)zV~@Y^^FKP5 ztqnn)R*M$L^aaad{sLmfU@!k*0AS$+v7V=&yBCNrgIG2wDC8I? zg7`G39}41^AlC5sPi6Z4C+zO-@o)L=?!N!xzx)Dof<~P4j|jcy9{cCV|CgtrYf)hA z{-jgjnb9xY)Ec~kK0!VQhL|5?Mi6fXdt9^xF*}F}{-Ae%VG4S`C{tSygZdN#o)Ly% zTmdc+D|)$~GY7E%h>e4{z8{RBhMb!b(|L)dhO410_s!JdS5d+mWzUzFCyCJ@4S>ck>Qt*<)EFE7eXVg zj$;VgNqIHE{k#>3L3=2pfeU~ka1IawqQJug2mt~B{|}ojo8Yg%?id5^KsewFcmr~O zOa94W|L0CHc(wuJfGr>x)QR{v-`PKRy@6;@PX1T=S6(^5=g-~fKRE({HL!*tz#O;+ z{*D5%H<;(2+U$TYpv3>*-~ZI^3EFxMJdW@EuQCJBs|#SQ$68VUlq#lfwT?BrJdxu$I3f zpl_ia&=x=hih{O6-$6T%@!$0u|7ki0q5a|t%1p6&u7!n#A?(gdtDPjP=1-wPff<5JBMU)hk&H%vi zJLOmh0Cu_lyelAFBmW@_>jnVL3~+2Z9?nIMwE)0f3;^2J0DuGkA2P8W@bS0{0G%H_ zqr#*A(xW`SP=TWX3;0Cw0z!Zoa2oVT2~Y#H0e!#-I1g9?mjOrM3fSxX!2TQ#!~pTY zO&}G>1nvU)KoL+3lmRb+YM>ry0Xl#lU=SDuJ_4VBd9eR{1vUUIfCq>W2!sa01mS@2 zLWCg_5E+OPL<6D+F@juxSVJ5kS0TQTU`P}s9+C{ngycesASIAWNFAgN(gS%58Hdb5 z(2(zt9msDAC?0 zC5<~xBuy&KLz>q#D4GeHuQUW&23lcSC0b)zC)!}zB-#SnSF~NU6SV8JL^@VF2|6u0 zOFA#QSh^g#=X7uA#^}D%5$W0JCFynPFVhFm-=u#)Uqe4Yzd*msz`!8NpviEF!Ji?K z;Q>P}!!QGy;TI!2qco!-qYGmcV>V+2V>jb0<1P~u(CMTvyrfjB4raq=6CIT}D zvplmIvllah`4Mw7^91uY3muCDivi0OmN=FImIjtFmQ7YV)>Ev8tnRD{tdCgRSZ7%G z**MtFuvxMNv1PHnVtdQ>jh%-56uS|-7yB*tGWI_9RSqbJ7>6N;Cr2{JbB+OyubkAJ zlALCo0i2nf)tnzVu_xG1sGP7n0YCBRMCXYWE-2S2E;FucTsd6zT+>{?xdpimxV^d4 zxofy5xDR;vdGvX_c`|sAJfC=e^NR2q@donV<89$xk0b{ z-xo#+{}4GLVjyx&2CreI#I7t+h6txvi5=DwGiqVT{i1~=!7wZ+n ziVKTdh{uapiO)&UNoYv;Nfb&9NgSM#JY|0>?NrODZ<0KcW|DD|Rgw#*nNGt_hn+4x zJtajgr6Cn4^+f8UG=;RPw4d}N=`k4!88sPynI|&ivQ)B~vca;YvNLjwat3nIa#eC@ zc`o@2@=5Y7^4kjH3Qh|56owQI6;&0lDLz-6Kf`|J{F$UPZ_ex}Nh^6OJyM!dW>Pj* zMku!`<5Xl+yj7m6d{*U9wNOn{?NueKsjG#n)u??}7gxWk{#boRgHyvwBTHjglS=cf zCPK4A^Ou&IR-{(F)=zCY?Lh5U+TU~}b-Z<+>#XXE>bmQe=&tCA=v~!&rndq+33G>) z!Z7+``d<3w`s)T#1_1`I4YtoJo((_SboRhd%kYL__c`b}lXDs8J{YkZ*%}oYEg6d% z`x;jnV@*^|uA6k5Qkj~Y-ZPys6EyQQt2D!&S3iH_e7`xPxs7>|Ip%`&h42e+EGR6_ zTjW_RT$H#Ne6ht6ur#yGvs|>2v>e8c2Yu0D1(j;&5KPIgY^PW#SA&iT$?T~u9C zU8Y?nUE!{8ukc;DcBRve#m(KV;VRWt`>U_6lH4ubOWpT9%sd`@U_A{z3q3cy^u6wT zefQS&&h=jR(e=so`R1$Zo9Fx859U|kx9M-_U*wMqFbyaUI0&>1tOz{1W`7MCL>qK9 zs4bX1I54-6WA(Y})*88*5+oNgvlgh0s{K)SwQjoJyuPK zq&c=3+v3-<((2ec-gdsN_l@?Orgr)EijI>Vk2+6u=5*0_-Re3*#i91QL%TP7e0x@V z-FoNy9QvmEFZF*IFdrBmG#>05G8jS)>kfCk)q2}DqA}7ksy5pEPW4^Wd)4<%AJjfH zkExHfe$@Qe4# z_m4lEe|+5t-5_qJZn14W-aftE_*4JqJFEltD=r*&w3D^VyIZlRve&zBwZDW9!V?cN z2)u-szchXg{kH$TPK+VZkRFhw$ZrnK4;PMtj*ddz!`=UI0VsH;@Bud{b}9gX$r0oZ zAHivY^IvnmKN^s~=BXft{CymY{|o+WUU+;1FiC0C)->5+G&-ug6sV zvOYketo-i*N{&;6kFwGq0AUOVsR`leXjc#bXx;z-IsE8|_~ht_{2ZJs%mP4n*k5za zV{K|y0C>iGjNZ+F@V}(L=AoeQA0Pkg74%$LS>-s_|Lc0x4Y1Kden9u35K(}F4FY9@ z9Q6ajpwCnwp#jrB^*|_~lvLC-v~=_gph6QXKmmb5DJY>-RL2VokQ?BAfRc@h{p1+~ zYK}|pG@@agN;mJ7(2AX{>p5XPjult-2v4G;=i=t!<&!uid0I+ZMO95*LsQG}oRP7K zso8m(%eHp*AeHm<^7ird^ACuKjEatd$Hpb!N=Z$-ot}}KcR#9Hl#=GyF9<~pctF`GsZO4uW;eJ*;~vH#s&tc<^X$Enx*j?) zWozsSkMMDNE^!sK1n$_iKc4;f983D&^6X#7{==_%ka$!4Nl*%~m7q|tr6|EfMMHI* zXlOw?1}56S6aAkA(kBK6kQx1*j=(}7U>RU$YVa=;9Sz+-xBmOrqeXD9PXA~QV1$A@ zer!-S00xjrr5UGz|BDak(Z~NYF?bm)FVdJ~URBA?<8GX5CVuth?K0VSLx}*dIF`!7 zE%^w*+Oy2r3^WAJv23Pc2Qd!YQG~6UctgCsl-zjd-2v|kcG_N3lSk5fq2r8O3$!$I zSI!&(Nks`CiWD-wVqtxwZXEHpGa~Z!&IIA}T8{(9_xq9JQqqVPNWSjI{aR4BI$>;D zNF=RK_(X^DDl#>uK5?MKqU1rTtt612H{jfoRReWHjHf)==awBV6mspJ+^)Hm zoZKU5Fsz|EZof-MR3KfzEt5D_@e`k5zASt{d=QiYb6>RVN>5Q;orhPEpH!Wr>zB1Q z%_4~NWXgEo<3;$vR!W80;~lXLcoW=MW;_IGhCTw=o#YFe3++B+M%FF-h%T6nb-pR` z;Dvje@z)=;B76Wb7$t!SLd^ff-y|38&)?A+TG@J?Al&*!Elwe0-{QieV}QJ4Vc{ePwe=O(8T6Dx? z-@IkrjS<93({*9I8#b#dbu33Vx^_LeJis^kn;&$~67NgM!N4+?fer?o_CW2N!E1P- zam$qaYMn{;tXHArL7>?1$eHA%p^Yo+Zl@9kwNi`A4hN3_dR0NYAN(?=I%4G=4GX4q zMtr{v3!+b6zMki1LFJ{~hxe}nxkDXhouy)Hg1sEv@6g!AgrCsgm*j-RWqkF)%_PZnsMBQC z=or2*dgdx>M*ZrbRxYbjWJJk@d*^}TH$&FzcH}$c>`GtevT_%r82|80S>^5xS!*%I zR1<64``Nby_<$DM5ajE|QK@~o^B5U(bBCw?+mcb)r%SWAVq!c_*j{Eg6M?*~vLzh> zM?lZH1XThNpGkhM10&Lq#882{Y8GclCf$mZdgPlJ5>L6GF)VDH+#1gR7S#uQBel@;(!Pm#n;2*IuPsCPjKPfF*1{Ptc5u4}wq zj+wBxF1_w(D#!TtaTQWG+dw4Inc#{`M&}Hon1#7W(m27K^vL@U=HG^SeeJ01m3i>O zm>y@!_9;WF3NRQ_{eI$?!Vy3@NYKVX@o_lIp^#)f`N@6CiAFb8|4F|$O#{#GItAQ) zFF3-xaxbl5^J@0u^UFS;U8IoH3{~yU{Qe;d<=IP1ClqAsYfZ|gcc(P3Z3SQKYW%oV3X#3?R!@kh4ig3necEGb^W^MLP(feSf~dkhRx^%E=fuAIX?H! zRu!k%gouP|xASiZUi=_vXwjOnJ&mTkI|hAy&-BiC{#5?e>#SjJ83?R13!g<0Ho3ct zwzk7f#u~?Y5c6?JWWnN29oy?ffvxvF*VvNub@iVybt3TZ$(=ej*uYEVibGD4GB))Q zc4BmkJ9@fka&c*+Ma#9}{`Ki>hD<+F{@lBs&>q`|F(S;%zK4CqP4UG|>%9y)LO=Le zKdJ^v+T@u`1)-x0Cd^hdq;55D_eBqy6`t;|R}{%r9GZanB%=e}X6Mmqsj|V>%ClkA ze$hvOucH`3@kWy1uM1D++vJ8cHIaTFb=1*Fn(yXi1UK`#ag|~7;pS#8+??~th_=rr z?5`*+#aHv*K4Th@xixusnxyop=~9V*g8WCMs?xmCldcy*ay=oIQeFzh*X&~^^Yp1s z+!W5#kyF!o;%j-M|MYwm+gr~>YW_-Hne#`0^<-nK(O^Xs)AI{G&@s{e7ulvj9aJ1T z0JxSm$Tn3ATi@-gaGQ5Et?GErA)dvp@s{3f@cQJwtK0dHjlcGT!tu)&>zb{1T^2{~ z8nWkLVg*tzZCe0-mmF#dd3`N>j0w^$pJZQ*`?NT>?u)nHdA+pQ?Vaz-^!34KRzvDJ z6U|pb{J62Cc<=QbT6psL8NK&6)$u`_>Irsmeo|pj+rcyD^1ELs^xy8icrAz+(Br|r zsF)|&RDJfF?$jZlU6XM*0)DnR+V4MO=~u{c!(LXH%R$@cyw>9nI0C3ukiZ7_+MGk# zOvsn%tmcLh5B7e%&%`Ol;f=fRAe2Hz39Er`*7DGIST5mCb%qzj`Q-WQ?r*(&ILp*j z&nt1SmabPcHbM) zGm#dO{d1>E(4BF!3X$)frmyqQ`zzO954?Y7l>cgS(|E>Xq0~gZNb+OUHI^h;6AZuU zM%N_Mo|(hkaTY)K`_tB`oeFu;hfpt_;7e&V;VU7w!(&q?a7Vxv{YK5id2YR)`#>{zJ8Gs}BYX?=E}(fUyxO`t(>Pg`IH0NcU2I z++Crcm)bHW7R|mf=GH$On@&{sdws{bpO5)wd2>U}54)nMq?M*m3%DL&aZ*WJdUyV5 z04gos5go!k+gyg$^|hSdVry3U9W`YiM!nJ4abbj6+b2=|TWRZ}Cq|*~Crf}-*~X#d zc;oypX7=e_tCpl_U3j+HiJXZ^`%utB!5;_?9edx(Wq&8Lj0Z& zx2~io)R+mRILw$@Z-VrUTOc=ZUkmjhWZtd!%;5m4R!w=hSodw4FC(n8H5=Z>>yiy|2a?_a{oE zq@P~!<<+^G%@n^*G)b5*ra^flS-&p{5lVb9f^$!DmknaezoY!?=_ky%3nqEl&OIA6 zGsxwv+?M&V#qBt$V^rn4os46*{^YtPsDIQOyUKFV6I+&hJxxsh!x)n{$;n)$-a?p*Oa~?7Q zNmHK;g3UakxqitkEVevWEzE>jNj%^CM%HX^6164o`y1_7xQXiVjM$mYb!&s>!mh5 z>{VdgElHbU)-6dkzzs;T82*!V z4Omjl_;+ON+J*E}^4?}_IxlBm^jJIljK4kc%blnM5~F%5nwGY8{w zzj!(kR{G;~O^6Gw@Rp9UirS(0jLPCE=i+^-8!=1`W``^HrdrVYf)XbLl#RmJEkPzR zTr&PK)y|HWlC6g_Qf0KN3gk-)3O?COw3j<_M|#*eqhvw46mu$3vb43D8)M&xWOoH$ zw?^@c-FqZ9d1CG(#E8@C_@Fr&|35P$MSg9^3{1i^4;g5@ z$*2O;LotHSBG;vR^&cI@8L}Tad#ujr?UPMsc29x~*@_T?8M}D|G%ckQ&EU~edIK2E z$WD)mpYA(B`<=9gyovRzRJlvMsitymxPyxTto*mkXaYmLD>i4yEmezV07kh`J@~L@ zPMip(zV1K7;cT9r+aux?_CrZSq}ciy%h%OI24#2O7YWmR%oXMLPAncI?afyu6(hzn z@Q30NKfm@TBHGKf!Kx~0Tu7{67^`o5j8%c$E`tfXl%C3gph}W^~E?n z?TmT(!*17-t0Fewj$l)txaVl9mq7~8Ft6&S8`d#My3dfD+rd)me<_yB32`TK)Y*s(2{o2$+i*{9Otcwj2+V>kwWUV|gccHDhEp{^d4OaKgxc zo0D6c``ujM!K>DEJ^Se{Zg6O)LG+mG3Bx_nHC6CA=7vhY)RR%CzGvqlT^{jQ`Oi|w zmeAfUk@@85!5|-afWQlh91gqD4r?9(88#P%?|YzxP%CAOWry^xSifAwTDZ2}GEV1> z&PSS!<*jPsdFR7XVTdZ?_Q(E0zOPg1&;7V|7zcm9n9w$!V9Xk#H{R7a6nPZR0CmfA zs|Vi)2v3FdL&m(-n5`b1wmz<+znG_SV$2%}zvbL~&#zu3?5s#CJi~OP9hc&epD%WK z&iwieiEGqsDp-#KF6@vT)=-c0OI?B76~c_H^bdVxkxRYdK))CAnFcMnS%E|0ZL#+0 z$U|O&vi*$8hVS_P+wDDV-t7-19s|4;_ey2QWD{KGALNdTNEQL4pX7oBX}{uzG!mzL zNb)jeheNzM7BR^8F3yF`Fcx1W1#7IHaP}9wcHjT?w(C`;oG0UgX6rexU@Nu44LK0pu%NlhYKmDGzfUYKj77g3*B5VpSbnC5Usho=;( z;1hU>yaMxwd|!Eu|eN47*fcZ~qA9M9Z!e24*I6+45G%+_%SXM=IS1B9fp`K3=BoDF3Qrj=cS z&)uoXk55+8_=RD1H5JLKN+HpMA<=+mC{QL$3aJr%F_D2xp7tNto3nXS#kt>Dj5JQK zw}$cN-`>mmZACLv++0E8uvyv8#HKkAkVMVz+(Jw^L{fFJqeJ9I`$Lbe=6W|JTL#&h zk7EJ>Ug>CQ$HG8rS>78ghjb{S*#@j>rW?i;FHNXJgvDTB%+5?g2i8f(N%&pUPa?Ue ze$@l#?)%Jw)HU0e$F@_xY3=vd4{Z z@@HC2SOjT!4z;5>ZqhP@=Hj6T@h zfB7DPd}4lHPoNFQnDiauvE)`Zc!faJ5qk&7W)-) zsm!y)$I@-7r@FfrTlYP$NPyiSg#0jpHJw;EP4EyX^0k*2py(55bXux|+^85s|UfzQJE6}0P+h8GL==1A&uYP=C zai{78ZiEn}4ttgx6mCW$QkMlt+UTm=@mGtup1hLyQuO3YUG?IA#;=K3rZbtvs@NCN zSy&Ao!aDvgp<)#_55v#alW)h%Ag_FbF-2al3?f~grZiG3lYhjq;1-*6kmWZbS9<4W z&Ij^_45=OA4$_7LM?EM>J?e}4tw>RA@KZ$OdLQ0*Mx1cGcol>*ez=+xw5m9o3Dtz#N(dRB(Z@7mWLYU2|cLx&S5<^MUP&*<1$A)+dA-3TyG6E^3T$8r6?tCHS13${%9CzFXjCPCkg(=k{`w#@Few-r$Tqk5&T2#!SjYQH9@2JziEZWC;7VFcf zl6lxUxjYj!#Xr*QUIznaH%V1I6q4@uMWx`1pmR`PS4i1g(glysszU}Aq9)vWxkn72 z+c32shGDo9Xfd(9d8%FMyU${^SGp5_v~b(~SXwjVx!Lf`b;COdf|xA!Esluk8nW9} zv5oBc@RbPv5|?eV?iLL8Fb`AVdiJC^Le+WbA$j2=&1i9b$Y#iD@jP_J{${)x(U>q} z-9+#&{eA>o#TMRDp)-0}AS3lW1*!1V=-!3wAL!8Xvk#{8jNxYMZIrSY?-&9pz+uro>qe#9%w(BjV@JW&SFNfe6i4+tqr@rQeH}H(T7~;h=$c)=QDQfh(}j4R|FiVv84x5uUq_Z&fd5*Ku8))WKK2 zTKsU=;X7^w+t02cB(ZpE;v3C2=FUCOb0T?FV8^bR__>YnMNt3v7_8QOI0_2&gOW(|)4n_a#XA!i`g3S$9ti+?qH$vij2F6>`j~ zdr#ye&C1Z$7PzGVBR_xzu+aE{v`U8?pvl^fF2_)bg3OrQRUwnZoUJ3W+-B--S}=B$&bW zd%i)j$U|CR3e};(N4a`lmJHAIK15j~_Sml2fBy^&;EbZGmach;Y z^(y^eEl^_Cr^Akb7+@=$No#=pGsog85mIBVayDmEeq-b|c)_q!AOI=d&Uq z(7Y`?q;JqY3q9I`jpFn=%fk-RNDTgNy^1bp43t-J{n00F8un1FOrkzz2a~SX1 z`IS=0_$)_?T!30BhBf&1yUlv`E{PHzq1SLM86DTl%G)a8sn^?%fX3Wa8yh7auU%S0 z`Zw8^)4b(GuHCx=4)s)C9}fisQ1wF9F9|!bTgb>K)+4KZIoDoI=+zK0rVq{l{aBqP zUrk%r73I@5=;8*G{^j}Q6}KKa4m5QoQD+hUX7xfX1A5UzH+p0a5~%g78F!`pJFug^ zHTIBgu2kt&cN;)};du!su|orh#)ARjS>@HVwCZ?g-$0=<_rO=MN6aPOG3d-AK>S7l zMv^6j^V`^(^kcOtoBUzeq+1e+AB{|V%08jPurkj%8L3PCfuDiloKF~22R$2iL_v&( zK*JdyO<6PXnYFP^D3JqBTq(}daVDN6#kMy+!l>UF80oxoKT}1G&22$$akk)`kh6=q z7LtuCTg4jMwcE0de(TI|mM$+(@}?hjSNo9e3~uz0 zt73DA=H|Y6j=B>eB!YZL#IrC+&C0tr;@hi7TT^icE-YzzhkU3*`j9F(*c=liHlmgT zlim!WB(Gnop;A1OxlkyKQm8rX_srLrUWLpn$7D1(Ve$w#p~H{hoT)Xa*x(btq&PAb zCJ+=+vZNKZGrVNCN;-3KoifXQ8{}PXD+BAjh}lPGGx4T1OHzb%Z9941^^TKESXSVaW-n$xl?;QF9922bMVH{^ph#zQc;j0B(gUg9x3S z2VP9;oM6Wr4SHd=c<;1ltMH2>Ao%Cp=pnE03I^V8R-1Vlr`8rrE-aZ_*d{7{@bnkL zguZYrP2rZ{Oxhf2vodG7r$>?S){PYvn3LwDFo>Gx#`tEa5<|G=Ybq-UhUYsE-;JYd z+kQP$Hln!YQ=r7pChL9qrLWHiK*#ftP8g>!sK-GV!^z;Sr=xs7?FrH|5Xq`f&*GdH zzQ^EZE%8?!lZ+xI*-!Agl%q$_1g{@5K!_RyYqvDz!wIh6Pivlvv`jU=*SgLt|E|b+ z-t-P26+B9;G@y?W0}`%PinHzhByqs{=G1S2yRgeAIwa8F^hRI!y+LH`ZP2QBJhiPZ zZwTZx>oWEYdigBaM>Z_iYz+mnx|N!Kmd}&6l&f0!Y#P*03m_rt@PlxjW&h9lWW47@ zhtcXX{}`Gq+1tG1hhUEUhGBOIYHp40Z12dCD1Lvxk^bAr%1|!7XjwwyBBDp9QBWDL z&_EDgwdC1QFUq+kw_oiuV>J=XH!{D~5-8q#ss*8F9*_l`{w!-=d)f^;4O;MEz;aU+ z+g(Gu6P5k6%)V=E!_A#^_GCN+W9ONua(&0_2&iGv7fwN74K>(m^i&q^?TW%(A^3<_ zc}KvrjDwJQSU0ROXXxkLIvN}n%dvP=7*!OJ1pitHMW z2aLg4)bTUmc6kq@=@F3mS$T|PohD3BN7|8J!LC`}IRaQw0j@B-1o?p~PrMvL^-Ekr z9mdzF;LC-pMBCxyPNnX1h72@7LT+;WQYNjOV*eRr6At z`*d$q4)}uJOQ}i8rCh$K>5FZ%5G3qL4-Rg^HrBxvu*vF0Ai-cmn0~uU z6%2rVhZ}Z6m@OcIE7@Tt>VDT$Wb6g*-Vt3bGkKBpA?OxcWlAQsRgIY))Sn2&4}uTB zOS}-)vYYIN@!dS2eo+E{)WdbBYLsKyGP)?`#e-dWX7BKi3nqLIJ^eB&r5b!3lz!3l zvi!2p1BWov$f!BWD|I&EyCTeFm=t_*F(#1QJbq)L8 zm*om&-{V*|!Tn3A9NZW=h2(q$lzJnrLBmfHXdEX=ayWI2>mCc&lWN!RaTkllD5{jd zk4DJ7b-|95uSpEAhPqGj?Vx7gA5Vu?vrV*yG82E__8ummhVFSD(wh-o0uc4fJsakh zl;=<4hVDodjT*O}u4mwpOHy528lrM6b7T{vWb4xJXMJJWzf5_^3q6 zA7diSW9&0GCokcst1ICzx@LW2qHn%q-E8B(vd}yo!?^v!>{n>!FqkQH?}-t(<5}zy zG}s#ib8Nh*XZw^l$aUvBmm@c{Yw}y*-q;FvH|q1js$`EF73uth9tWK3hhU6xV{3j| z+TPCFNHMwyj(aS>hb!z&)_+PpV;MyeBq|1joV(qcSTc?PZ?bZKF1!p)-YXjU#=+Y$ ziqJO8Uy>PhA9)@V{c`ii1IF{}*xgA)k4=KaLV#P7eeV%)6UE+?xy6s>Fev*`v}c#A z5*Ws`R;HpJqjZfDl}E{w#k!^>c;CAg;TO+K)FeR&QOE%d5)!Y4MWr=8U^m5}CRH9e zJ3pAPl}1Zst9mm&uk7Ka%)T`<>elGn^K-Tkzk);E23MAfws-n4>V3}U=BO?H)1<0L z8mfNNUbg<615r>YZ-J6hO6s(+hadE-@Z7;6T{ZR*hNWjwatKxzk|sOfZ&yFh@7H+8 z_R5gsq_;M2|BO8~RQ6mt*Cpu!D|BY_dsO3_6UqKgr;5wfowXfAbk-l-RdDM?6}sJC zW`YF@6C~WSRS@8!&_J0T)~7DsT1v#9+kEQz?chLH6U!J;siz|6>l*~y&W!LnhwKet zkweia&IDe#ImR}>-N^t`&0#t1*%HxK;!KPz9+iYjsBcNOaYB~HjW~fzMx2nNiT^_@ z%x4Ha*ckNBAp|5qnCVc`->k~7D0SLzl;>-z->+=Agf*Y0r{q^}T{>&}0-NtsRy3G2 z@I$Dp$a!wOInEYeMHGp@IKJ=3_(PeuCLV@q%;hW;5Xse8E#Q@)r+A17^rz+NzCwe9 zH=uu#I1g64QV;2HIq7;rB&m;4v~$Zg9oW{cwofl^4ho9xUa=}kBkexr8Z-!Jmf1JX z0DhU^ZOAzZoQH4qIE#(2vewHRPDO)JcjES>I=bMyDUbw*t7pwj3vWKIiyjpk!j}VI>{v?r;BtfpyqmI|jL;DWiu<4oWN_*!2Jp}FF=&vB&Zn7-K^Ha!A?o)~W2lgAB;n0=?Tmca^E>-vbo1p8dBwcR)0v*d#8LPw4VJ4 z(5{ocXc=&r`0^P`UUdQ^NuPiwY6qZbQP(`51!y(bdnK){_g;w%6~Clp78u6iYE18H zBKlL@SZo;TZ*4$o!Ln?!5N{;BeX8|Im1Q<<$ceh_SwJwCDe|l0i>cF=1tGE(sit;* zJs4AOS%`=tqa7;2?-|Sclp70LQJhWvK6N?3xzMT#dBa2hHey22-R5&lZM^IOV ziL0T_IwKp~&a(U7FCDZsopa!V3cm*TVZI)0nh{|M^QeaNer^mM5l8}!1Hs;1j#S_0 zEk~LgK+f6u=gSG1oBK7ZuBL^_Z_4N`h05$xgCA5PpCJ50W)bL+Et{1~E5+${^;L7j zzLToXqW6CAMXp1Ncm-XfhXF@1Z?8~lA%+|q>WW=NbY`15U@<0xnbduq&(mKz6)mU;d)*6*fyCh7g!>RWcGmsR* z+ZD>Nx*d}Ng=~4d#(5W@eQofiLPMlzSxa9=pYaM1e!ARbN#Y1`!{rPNqx=uT3D;LF z83LpW<_@b%Zq(c`w`!yeG0JL8Pd`&=6c|dXyqaZ2eu6E=N7a)b=mBmdMeiY%PY!p$7ILpG(Ci#YpLFP!VNc-|ll-A|8v2WXD<*Lu@rSteAvn1ICl>;K zB(;ir&6l5A86?Z?l;J!-T7G+=UR=`LduQCk$~H5Ycfp}fa-Rjml}Wr7xZJRP?-?9s zjpcfTv)ljLjN>Xf*~NEGEnKcQUFGMZ`RP{P#5IQz@ZaHW&wRdbMYA%t*+K*o7QtQ_ zlf8~hOVqi9U4lnBc+`D*{Zl?l`*uZ>qutrFcl5Z}%b5q+6|9*JRHqL=qw0jW#)+=* z4>CEd@I3TlVG2n*_GJ#!=YgzcRnb)KPDzLsQ&5N>1`d%Xz9N(CbM{+$nPTQ?-_gwP($K zrSF8yBfw>n^23SW2G8NxCBP>Y!wQ&v)SP>ja`BVY z?-(N>!F+*Wa!5afS%$Rqx(3=2B1(rMg2=_Ot32Lsf}G5KTywQ<*`@nL4lgy2H=9ad z>I&PB*+dY9;845)R&^#VUO7y@u4Y4l^Y>C)7~g>0!re-?A%OtXj3v1bo;eY$u9?al zLo=7HXx7w8oXU9GPe(u?j9@|v!EQLzk9P#3TZQ`N!LPX{e!qRbdx_7_+CJ4(Tz>o> zKlLtcP%EQ`CG+8X1oje3W|vR{A%tie?=l&$y8Fm3~SFFhv6uXKEZduip!BoYoG&Xf~G)(x8L#9+^$g-mk4ks84 zg40E1uU#T7KMD7=eiisFXC8sKB{zZN@}mMQbvnip@|mPM8B!gxH;k0%YW@^lbFyit z$b2p114ijw?uAPp(ht1F!NH`u(TZlha8nKo!`G3Y=$uaAAh}HI@V$vgI)@+RE?4|Z~enlf2hht35>Yb@nj`(2Q$!=;e z%;Se-$()dF?|O&@bG6`MILTYpO-QbdxU^eM1$PoA%Ut~nTXMxe^Y`1#_Q)!kw4Zb= zQWK-PXVwi6s$9v9;N~%^ULjp&bZidUTLA9Krzq4Ibri}3`tqGe#TLvFFNo4^t5Hh{ zd@~0UF475icCq1%@2%$0ed|)el2Tn2+$O74&82v*b~bIAt6b2Rq%2FGn)%{8#KKOp z1E*Pczx{UWSx#P}{6vz()~{b`o{B;W!JEf6uf(gpK40@N;N6w05&5CvTem`_V$P7m zv1uuL#ZZFfIjmjc9j-z&S!>d~-dOu#(CbnM4@z7v8*nO7OZqfO=T5<{p$L-T?0id@ zc!2~XRF$bd781VeJWKd0aeqLkpeVld9Q?ABlDQ7!^snLBdgV#+gt|UPJM4?jH?o(` zm9Vi=+1m0P|MwW7K;`LYEZ_n_2y9kp72iQH_eEPU*tU#!m^=%B+Drs1oOu5{%hT^g zQUs6hxqA+RKVIA3OR%Ox-s@r|QGiZ9BY2|Ur*-poM1U^=dbs(;EQdafhR!HCdh+6z zShaJ;8PQVnymw7R8L9H4YhXSggIeDOr|wt_aPd`rg=P!#iZG9l#?>SX-r2N|E;u|n zQ?9(E`eb`LZ_93Q`sxAwiiS61)(@V0xlMaCTfRi8gk@M`VcN9>&IQ6Gnmm-O0)Dc6 zquIN2GD_F!aQeO2)3qIqpy1Lh-BAlsBL-=}UL>D~oD4=y0p(#iOT;gdi)_vkTvwkU zy0wOS+v!5S7Vt}jtWC=$Of+1m^xJWAHmXAkXs5@xJK}e5x)~o%KV>Js(uqaW#`6*vmrTFjml0-NB4jPfO@%IqhxOs9TT&zKq$gwVh)e2>a!+VReE;2wW$!+zReU+f zajp^Ci2?}=!cA$3P3DL>8jM0(quFM4fg20lvd>HHU*o%^lKiv3dU%s7 zt$5v#WN@P66EYvTBKEC|gK%e6n8kmYNkyuOKMy_6Q)tzRD%f|XpE*VQj!pdidZnXd zrnjS7%joa7po0Q%&~tKi8M%@aLbN4G;1tqSn8IP~ZxQHG?e7x#stmlVXWq&4ihAGH zx!`%GK6f_cp<>O_!_~3bV*Dl(8C$f@2lU%@MY((G=#}5m5;Ly^KVgK z7E#)^-nnmll2qvJlF&J>FRU2|W6}x2=B>yT_9@H?P4>O9t1K}L@_#;r@eG}d>lLS7 zGb{ScdnFbUBWtiI!xdbP5+R)c-&-4GR0k9XawLB_?!jW_uejI0^YN7tasx}H_yhJI zXAQ!0&#Lsh$s6Slev)aq^TL!8VW46~11>*+a}u8@@+*{Yr#iY~=$Gv44!gVaZswJQ zDA(%Os;y;S{y|({VsnCH>Kntac3d;ItsVXK2z<+dn;uCI5=IK+(o*$k;bXl;X(>O~ z{W~%@QgcURc{s>LYxD0~?!Sce%Nkw{R%kLM{{UZ<^yH`Dazsc*oMe9q@od7>6+MI? z$tdH4Gn4e>#l@f!zk(TtX7A=(;-V%|BL!zfdL;6fnRS#`EjO!(SH8P7pQh zyLI;m7nf7~8{?LU$UE05)oOcG?9Z44i=9#K&2A4qantsy0S|II<1`e9!c=S)68#e9 ziq_Lq#^$cuVbgAQ(4jYwO z;Ax7DNfY6Eb1}s|sN6xHPE^KOqkzXoGOvl1vH1K_Y2yFG-kZlm`S<(dBc$wP$vT7- zS+XSSC?TmxA;c(ql4P&RMA^3xN=2a*G4>_PWZ%g;A;y?W_8BpTS$yB!=l-1YIiEV` zasSSJfA8P9zmLx!(=}aY=DM!e^}1f~*YbQlU#!4f{dmE)hyFQ{{{PJmvW3Q?Tm4OX zd_yQZi=j1pt?{J<+ZmCIIS;b}EjhL@28Iy6bN}-fa(nn+OGKy$;17toT^x0$Jx5lo z6nQr^6T904tk52hzpg4=u~SmqyOB~V-8mi`E_9TCLk6*j!m{SoamQ4 z=c<359RoS=4}5Kp{EJC34C%NHS=4<8MCH&$FeFWoqA$bsQ+IjO_ubX2`SXK+isxTbDe^yJZLbC#*-E`%pj-IKL`Nl0 zUi#(%m&=P6!sJH&n^h7ncQva@a4{u&KJeC&0-|X%sj6~Mx2G>tN^#Ju@E9%SiII!k zcUmELtPK$${NY%jh;Bm$YzVV-CXH#$l9Fs{OGu~3hNZ(`WQR;$20CNnO@JUPquzKX4XUcYG|=b`bxtIT72C( z9GdwXx*Q#&1JAd0f2a41h;1vg$HIHkNCYE+mg~XNk{8@UQ#z-8E%lCse8?)+Xos;W zhJ$R(7IWIm%LnQ{tS^Y92)xOjKg>N~KX!JH-%Y9fWQOr+FUMo3K9DrgZb37Pw1%i) zV4jJwQvxI91#t2CDcONFnd<1ZUN^gji^=vOUw7S;H?okSh}FH{Ym@65fA-VekS)Yf zate+sR&P%wu{~7rRl|Ui!-a}>8G|!DXC-e&oE-dm^x4c6m1O1v_*7|YfB3${i6PQa zbN-M+K{puQ@4ywa(1@l#tCc?V(g0iv*?f!M4qLNA04p8Ju?=ZC_va^n|M2}6QnIc_ zD1+i#JKiYPcSb2ZM5=^%?#ZcwZASJSP2!Qj*4Bp$2wr?Og6bQ;T^zeA-HUy#o_D=?=Suce2=_at1CH&&A zP(l$MeDp&}4yb7{qqiX^3AG<76n_ryI72N8{|Nux-c-C*y6%dp0Cp!}zC@TPVp+bQ5n3@WK9c%p5vD zws{+}C7_0+LaMeQc>j+B#$}0)wMSQeIwUwY>k2@P(Ab1wbO=BT5Z^7j=Zv)%+dmEb z>6pJ3$hk%QN5wwvKOWTY=PRxiK0xwW6m%2&`Jk{rUa@^V-|eHMKgIm-diT#4Gh%|` zNS126g?%BCX{Z$>$gc?baczM`Jvax)dM zWi<8zgzNzFZ+b}JcYrE?`SUUP(;EH1ZH+MCSz%hOs3?I#uPwd+92?NscjLx}Lr9!SJM&f4QXE9DvjmSCqVo?3k^zpEl5s6QuM?&;on;<{KC2e% z5XHP2m5r`=rrMk-I)X4PF89iL6y6`71nADBzwMd`piFG2nGNZm*0k@cqaZJ z-(dqV!ZtYyy#WH!&8gdv&WiSJh!Zj#crSQR5kxMcZm5{cqGS|n@@EGbb@&x|2&+91Jeo1m)lzEPacDDdP&fWgF3?tX9Zi2EXXx{? zPUeP}Z?V!oQ`nPE&Nn4vj$UEUz3yke3-Cg*EMXS06l;VyE#J?}=_2{eoj?r|M~l@~ zcm=J@_S11U_761{Hm6SUkRG(BO58QPBLi2b_hywQHo?ZsPK=hr!jEj^XV7M*;Fqpy zKs~CF2TwI~J4P{`+v#=hddL-4}ahwfA#t$HWC#LAngE@ZQPvoabm{krD%iw zC@OD%==GXr&5>1?Zx?+U+4*MUxVavpr<1Z+mvnYh4%i0u*;#@kWv zXXcQ?`##utF(qcn{olF$P<>0cdgeXU<^FrZa5h>~Sn z;brK=^+L}R#Wf>py#luv&Kz69Yuq^Fs?*WqyYyXfc}e03YjU1M(xs0z!{M@8R_`zt z*(m{+gDJuE&LgsWs0QWUTX+{Fj2=^YbvZEyLZ@jYJ6&{ipKUiQR5&*)etd25vQ7t(iKDc95{&R=vNxT zb)=zjhrVVGwNXLdiQvMI<+0oWQt>^v{7^3!KIK?m$P-Q)E1-S#viS(si4VtqkUwb7 zpaseB_wV@oqw8Gq$KBsqW-MnclUyQX)+U zE5E@Cf&0*nhTde|_fk_0vb4gAZ+eO$lMXw{T*MvgOLc57FU^(IzxvK+EhtEg9(!D^ zoi`~~eC^B5fCo0wG_Mi8J&41jb&}hVU7UG1ul-2QD7~HxkMd|Qd?JKFI6v`F9LL*7 zE)v|wAedB9%UkQR!UCrh$6qbe&)D4g`t*&=O(clj_cVx8Ee9sxWd9j9F7AtMGny`? zk4|2GaJNy$3@x%Gv2-I(te_SZ3Uh*vEwmxaQ1KxOWLNsA8QB1}uAql+792F=i}&48 z*!Vz2>M=jwWS_Cy3HGj2pRKX}WA_UzUdqUolvli43)p8O`}~|C|DeRL%!Xg@EUwp%_+h`M_V*715dalktQx_SlyR#!(|jctfYPSMxupLd~UwZs1E zWpT}J(AknRO46qnUz!Z@BTl@ydsI+*?sJ5$iaAHk+O15w-4lvhdLHj?3rO0#y%KdkpG#eEZy4$1fGg4;nkmH5L~L{iLVl+e*z@ zF#;*3F0+M`FDIXdiA;!;TUlmK8Cjd->>oxN8!<(kZ_|VPNM|s4(lV$AXHHi1^Ly&; zBl;KCH8|VaFMgP?J;~5v(_3e9=x!jJwByF!nY%E3z)P1n#Shr=gaU~y@3&k_T%Nb* z+k1=(-l}+AVQ-;r@qEfwk>pw#AY=JdD#FOh`~v2kRX|eCoE~(rP)(P$4|{g#t)!_Z zpy5l~D-j{VFO{IiaE@P}K_9|8@E>02FBZ%#96aeqmnJnTL=7M&qi6bQidVP9fg6kn zNnBGYBxE^&+qp3-@v~VDR%-K(3Xvnb;0{nta9m9DA-c619!&3CAI_)qydUUT(qpah zw=z7F|1#QV<-F!=!!A1s-K#OwtPkkvH%A-}U<#@7xc>X&q`bn(F2{Rj6?G=prIU^w z%2vNyd0~fL;=>(hA?jO(z*uL3hKjI7oij68><&TN7PYFo{cdma+Qs^a7Dd@SV4$>` zPdjpph|i{TB)*99I+sc}qsBZG{WrotKfp6;QI(R)2V%*0u4k{dVY7hf$JYs?+KV z+PL6+`>kd8gSf9z_;06BC1z7gZuDavOTG(Q2eNB-{}iYHwch{l`&X67va?%&9E645 zy|ZQ;k~`|ZQ1KZ~Fza6jdd5Xd;{f@3leY_B$a!S$bsERF>-_`52eKPP=YIPM7#n)J zQtwFhlV&7a1LvGeUd^b#UGG&p_syH5Nrf*_V$`6p1(!ZyaXaCUm=WJ~%A$|X$RKEyuxc&p(u$fM30WnrwDH;zOX4}f`IsB+ov0uqG@H$?oH^gc(D3J7U2<@ipWP`b0zJLtVN{mJ5 zYJ{)hy!_p5XU^D8rY)STtPQ^IFrY=)1QelnYcLS{7ZL zoM%K%&%ZPVziL|-7*IShW_Hs#+VO;}pZmG9(?&v}h1i5F{yGpOFz+E+B@u2<`~kYD|o9fC2#F zhR54AdAMxf4gT2-V-L5-cc+D10}-5$+pki`M(Jvx)4Q4W!XFj}+;*yy&onZ^f!}8T zNJXKXcj>&=gV0ey9Xo&J6qk|5J9h_Mf_Qz2 z4VK?0dL+ogNZE6=C;yacPIwNXyANaP;(v#Ix++C4l9o^9LWQ&!lRamu-UA9D$)IKR z!+G<7VTIN?Mw3VI+ScDP)6 zL`gF7tNaa?G!oI7+FGQ_7p#+9)8^UvguZ)#vQIpd{c%V}+e_N@f`0UMcw~lbv)RYM za|Dg%38Nt@?8W3yXA>H{@`#?`x$%bwuPEmi-j)%!%YAYSb)n(1 zzr0i%oe7+)+tPYqA=Qf9J@d@%7(pR^VQyR_f~xB`dSvxfsQtK6b8xKiyEjRpfdLXK za@TZnLqusWK-dXTn{4_7mS*`C0njl=uMN(Fja4w}y>Rzi$m6k(hKirIuzj{D!@&Cb z_@+hR9bMoUG-K3yO-WAoOv{4kBe|Zp;sm4eX1U}Kee2PRYnOnmY2S$ti5duhZ`2_E z?7Cau;-`08x0qhK4s3=}feN9;`AiB;m~bpM(PD3OB_`R=W!iYj@Vt$nRAu!sb!zMmg@5$Lhn0S_oword&-Hn zFg}2J)gkxdViR{GP7D;Tn`$PF9u$LAR|M?ko#>4xo~(atrpsw`?A@5}(Dt;6mO<~QOv13rW1 z@&yx2gBX2W{2ydRTuW0bb@*CiGCxaIhQ@I-mj@W_*&bz13#&IGOOwb)YxnD-(Jt|_ z`o&~H+>?GldTB$nPbjV}#IqxS_Ztb?k%|tq@ z6tT8FixI&y4G+h1R>V#oSomuG{wH?jWCO)bXLIP6y;44Jk+Ikk z_4ddR_HLrcv~eQUt$^Howlwy82Ue3+J*l{oOK5E>YU{@72>Ti=P^L7F2WILwbm; z=KxUbR-*o=CbB9)GHo@e2o_3b7T3JJsdIds`QE%&z7WU8cqYFGbNe$&zuIBR4eI-Z zV3Z^309lL(KSjn73YxKeezyb5?p>-1yzL=!`10lJ~F zUq~!Mc^i`KiBqC5%%Qi=hy|)|_D-*GAR7&22oaG=kvblP1`e6E38Uhem6OAnIEB-4 zmu+lZSO=NjU)j04Pd@h-jt~fs66@VjVQ8EnFz?`qABY67m@`X6%Xqm@& zd~GpvATDF-tMdM;lvKa#dsD9=elDd665B#Sj~cq-NQxH$+mgv#PtMC8Z)%VksZ-5^ z+sut>oj|J{Y1o$*EFI=^VDotucemej@5FBNy%gx;MOi2p!cp;5dGib)MQOTwUx_r@ zX>KCn_FNkCfzVIEuOVNgdf%sj#zhobbH}e7?e{BV$NR=%a7k5RO#N(nNoXhOs{h0ys??h=@V$^JS|IxBA@(rFt4 z{fw8-z5U7|eOZ~Ca*PtIwgKO@)WaNo3iw#ApW zKKtkBBW7cp&o<&f&lYTK0k>Qs8>s`w_lmcl52w}?o@ficj9l(2ew%5>i6bT1_up+$ zpemDJ@(m%hiDy%0l**Vrqt+{`a@RI?go#`7fa9TxOGC)J=DFAMB9y&*7E<&;la4-fW zspov_&HH9C_ayuKIoSvkMAXCf&#Ki{va^=$y1@%>rr1;s5#yX&wcg%Cw;#Uo^4)MV zA1+_o*JW>N^)8~TZ!)mk$7f$FpXbn_<>Rk6_ScmG;g+*B!UcjpAan_+xV2}V@FS~} zchb*R3truKPyDi~;XCf`%nJ>R&iH$9g^=cVh3NK`G1j3bG@k;-p|rHRhCu7GPxCGx zA6L8iUaWNY>c9X4OE%{ws_P|LH7Kh}po$=fJMzK@R|q>d;=PurCFquhG0K>HRTppH zt$k(qWNc08r=8bLv}SxyJqVX8&k5?As=CAUyy7&CNWcI>D?q8D>K)l2Kx|0N54tPF zf32{HvAmF5W>l83JN4l~X{p1IJ#mZV~Vt=Z^CM8f)5mrzoTKqK68q(Ql=nZ3XPJ`UU{*{=9!gx%2-bp2v9N zpkwLAv;u!9FV#1Fnkxz+m@~&lJb%gVW1Z>cz`z?DA!pu9irBq-o!C9D9Q=vv>SoA6 zI+HHa;Fd@kDdulC%BV;!v;RNLGHUM!JqZx3AEzE8wu4A*iF%&UrC`={%jn75OAZZ* zukU2ISE%54m1VItN5sY;x6E1Qzwi^R|3+&Ky)^)5{)HJg{>AqhHVXgt9~PgR0RW?9 zG7G*cyg@kVU7HA$JqUX`O@Rv@=!lvz^)R`kaw$Y6V~D+UYK#B9CH%!faFubou;IP=U^qgSG5|9-3EY-yHO zmq~()=U&iH(}*#s z8nV#@e-iB)bS3)uzM$HsuKXMb!|Ntc7TcAB0n%TtbpG$nb5**~6AYL#<+OSDQMT zcj_+oD4Q@Im0HP{XFtQ`b!O=6Ra)WJE49Vt5G8uY6ogg@a~T~ew}>XRsYT8C9+^JZ;Bvr3PT5$KwwIQNbgO3}C6O-&PTx6oqICGVPjJq>jE0Em zFw;Rt&BG^ib7QpLbZn;jtQqq6Ms~j$%qA?=7_J4-cn|hg`L1F~+U2xDy&dxc0H&@d zO=i90)U2Q|9O=Ig~G zm&BGsoI(DA8~LN!#jjRDxUaWkKu^rnFN_V#XK(u5?jY`SWu?bgA4#E!)srk&^4RqT zW2Z--y{7Vbhh=JyszU|rCg)T<-X#>gG5w0Q*Qrs^+n6%Qjr6lQ=8?m3Qtt{s`Tv(f3NYua9Dt9`hIHF&}w~M(fQX{gH^6p^bX`Tu6xMx{<0 zEel<&UX$KsZ*7TD_)=2oitk;`ILiJcSn@>VgQM3Y*A!!_T|Ol^bmb;rCk>%I^Dg^h z{Sap;5+R*#Bt*ps)d_#*lr9o@XC_`&be(c@qg5e?z4lGDo5`2IB!#Uwq7Sg8QCwrl zd87ndO@k1y+kT_dv>4`38q9JxwOl0yj12E!E^S%~ ztKsC{*gdOSog;_1Tu7Zv?1hgP;4n*2FNz7E<63!-WKZNqMfjN&V^4T1INA&pDY94g ziVhh|MaDRV#Z1^SA-XTQwR@t*7f2-pXZS9^m3!^)wOrOm>b5H16wVXQ##Am#8^-N3 zq8Nl<=@gB7?|48+eXG{c@^rWhUkPQ@<0S{;u0Qj`%w-?j z*X`F?-0$Nf->(yTGoZj!%Ingzg3y#WOwoX0;L!cc{JR#*C5y9T4thMFu-PEa%zEfa4CNfQ@eGhWJ;x(c_xN1I*o#lCVNX?(Pn9lT zqhfJfV{D{qvhS-_oHrx=L!$TdiM&)MYc1<>;a!cmE2h!1tHS}i_l172j*NH-XVGj@ z{(3La>gqgAY+T*z)c6xXcapU?k$geetTBFAf(>M5}H6)2+h9dusJZ`b=-@7D&HW6F7A~n04o+!=04pn1=`< zT6{TIWW!OI0N}yTI!{i;(`qG0)ib5a_e#-(cdR~i+-<2K#dQAu?!G(ycvMq0{Rl#q z$kjCG4_yq$Jdc4^M(GWGe)KBAamxC!0NtrgBcX<*tNaqa@hGD3$lEkh#ZpK{rF@~F z0d6HWi(rNfgNv6RZIGctLGs|f!FkM*)QCcS^>?*cbcIB!&{Dhd3cJb&Wy?-IJMka< z-%KCKFNpG|!LTswSXG;b1>9pwytM9JRe}v@d0nIw>Me_yyoflR^+_{b(zjgYi~iKu z2Al;DBve#2!&p(xi4baVWfh_26wxkti`_fWQ}lisw{~6M)0Fbv#r4)6sN1y1Naud$ zkluh>sY~JO{ZWW>MNSm6OZE22U7=N*K5atO>Tc>%T42DJN4Vnzi^elE9+*6{Bvh=W z+vjBD=kv*5JZiHZU>OmXZWI`@QD1$5OB`Bq(P1HDG}_lUaP^SS{2sNmbWXcOMI-J? z{9T@n*&XN?DS2}%-FC;v*5l3AP=FQeD*U4-4z$?I1Z2uZY?-kKH5?rj7?F%z;<0bu z+a%M&jQ?bQqf}m=q4Cp>P`sWX4b;Hhqx{_K*k(}UY&LKfW9V3x*$}UPhUsZ9A5R6Za$MS%TSn9!#E_&q{LO!Y?d_+SZ9@^*AS6nTVc5gnUHtWPt&B?dbez2mv#zpq#FSzTD`30}S*oA3lpgRB) zF$)<)|EwvbkWu62PTXPj%A!N~;jh-;&n?TO?w8x2=4TP!$$az3v|K2`1yq)#+B3Wp zSX#Q360FV1sHM`Yui4pZz_Y2w%lqI~M^iG}9nP{=u(;c6tw|lb6 z=X1t#!Fa6K{*kz_m;?F$4hs3r4*mV$BM2(2L3kU|We}o@{FsO{qjBD%j5e^6JpmCO z2e}s%S770=HO+6JIQF|*rH-*3S;=L_J7Juw4aq&fbBk;Dw?N^5@a)cA${**ghxYit zIrO9^!_+@v(x^Yl;6Y1O_tG#xqx>gF^{<_$f48H$4%1(Isj^-pMC-+j2t^b zbfc%Mhg4gCMlQN-L-s-AVT;*8poC_c(GVGd#6olag7?NeJZE@kv*TBi`2Vu+*CB2P zVEfHl0FOPCa!7cVo??UPCYli>7q&z`OlD=Poijve4*qr&5TG@~H&1%Ec~f|(2SH-4 zAASqd2f*~@xIr^)C2@crVcD=}p$otdsGom>AN-HEe?Rhp zl`M}#QFAr$8nh@nA4p}-qUt@TQ4#3X2iQ0)Ns0`-XoJijn1KI!bS~p>y~}^U&Y<;> z)34HxkdS3Ob+t*7VO#=1t|rb8!cUwS-gl_S3<6Ox`6j<6T&!b?Gh-tUow~!)rllKE z`|4E3ip!hv*5GUXLR}>>E`Bm*5=o77XBB&uVY+A>fI7HwZOMuM7}qfQZ%KM#BOAlPR&ji$<$gGQ{w##Gh7MS!t8QPqefn}J}FdZi(ccD0Zi zOH?MY)qvt36##q|!3u6@&;%uG1Iqx?;>~7&(s%8GjwE0suLVhQxFfPzBLp-Zu9+f0 z!Qeg#oIx9aU_@M}Jpv>yxU?j_hZB<>u&!io;&J`~&cCxRvRA9&h{>-klyN-er! z%LpWBDTGmuA_vR~k#x>Q&}Oy-B@BU5P9%+=YZr2QxqlmyfcXd|^kU1OM))2CaSy1UP-qI5@|>IsYbTF2~v|D5$djL8!(* zOTL1{^7H*lB)fv~r1WU-Fi^k91eZx}g)Ufne}+fhX8uG5H~A+;fc-3YRPnDa{g*e@ zwg9=0YVc8UwjCVrk5V1@S@d^8e|71 zCByhG;uF+aqoS1a^OYeGM~3(O>WI$b|I)RXVOgvvmQAmO>Zs%G8QU5(2=sezP%T`1 zUDv>f=&4a3(e&Pmx*BJz@sHE^ooMfcb^thX%et$v;S0xt!M%5SdwUYqhR!_yXd<0_ z(p0rpjj8Ogbn;n~&zJysLK-isBBmj2zt2}IJpcy&#~GCk*k7#59;H8R_g~zIKW+Da zI8c9jB7b@!zz6%&6ZzliS6hG(NmD&uw&e(1tfH57mBZjHnq2YrlkMbqM`@#uitSix z(4>i#6^lgJG5&S1#{4lrss+IeS)@^KM{&9-0RS>koHk~MtJ5Y-N+a5Y_RHlvJBRYv zbmhmlxIs9Z>OOTJ%~WCKjj>YX+dTUUU~)u1{FeFwu<}1dp$+Rl=iUD6QtS>;8|C_q zz~k?aOn+Z#{a1Ydb9!=N#rCJ*{|EZ)zah%|t*!b`asFe(`Pb1De~R-TBhJ6_fBq@X ze~dW)$`$&jIR7!?{D+lp{uP&>-=Fh8Mx6h!I>ql#?O*ZzKSrE?r62t#J?!$GbTz6M z7-azq;`~0N(JE%6R=96C>Vs2H^}A}vQ+5+AdLFKK@>q{T1g_-2ea|?00M&tHqJP9z z_Ai}LXse(cLdbsb<6<&C^q_G@N$k>DtIsc_4$G7me+gw$j=p|wFTY!_6XGN-|3}Ls ztjysl!k(CeQ@XEXi4(C18GB8(wR@@jeEz%1>?bvzBrq;Rx?&d>(=6m}NXG!2F9eiu zE6!vs6%ydB85>Q*wFJSa#gKuVc6I6v0IeR68-1~~JfOn+^}u8!+rC%dMGl&4!jj6U zCNvaP1fY(%39$F@ShS$JH?Q4TP{9cgbeS`~d#HlVv^8e>P-Q>?yNeCfscpXdfxfik z1FpIKKdIe8&XW?NP+N$t{AN0>vd))soytKTz-2{*Xpq0;FO2F*#`K@7sQr*_#>ps} zeumNT(2;#QDr&)$<+pFOKXc9fV8lkX{g^(Tl6SL7D7}Lvd0Mk0cX{ z3OE+g z3}iZvU$NQioUb~4STqzK2?s;or*pF^VT+wJv1Ypwm&s;0ZMJ$N%u?EU+o?~N>@L0E zo1=EqSUavDPUQ$jlvlayB+GIu|EB75`XG{U-g{5bc{5^WEJd4q^NzMhob}Y5x)hG~=O{v}G6Ph*h}t#8B^{Gxbj=7T+aoUSCSx zuYY3sLJJ`{Vb}R;FBh+ReVE;U@`#IWx4sKO7X(?`3{nr~$Fteh4J|((@;9n7EpZB# zK3#liuk_}A?hZ?8nJUzyMy^DxhpiZCO>YXaqAcr+uqGYNA=# z2<_tUq5vvGu1{*&IusrcjvUcf9+PWQ_R)ua_)m-*zr#5E9`Ep<-Xr)47`lZB@d>>J z$%_2ui}?Y~{X@N5A3&Oa&2{pF|AgV68xEqMz{ZX3hLM1K@t{B?McX9!0Kxcvg@>PY zn4k+>>9kSx=&NOcVBy^g0~Dsj24C8K>+g&dHaH+>08#k4^(Bp#K3F+QJxloF^$^ut zh5l0C=bL23KjRA_$TPy6_p-@$CrSv|$U}}peuXMi+IHI77hBAaF`eRwaPfiQg);|ArEBBefX{rOv&&$8#LY}-owIpV|PR4z?k`lQv$&fW5I z%0w*6@`SnRVF`$e=VltMu5RCt;r0_|O;{oN?v@O(v@%f#iNv#Iu3?`p861u^FCVXC z^UQ7ZgC5{5&Cd_#r!at(gte5Y(PH+|t!)F!Ie7cAclna)1$VEAxmQC@?<#9*7)5bj zBhTJbI^Jsb?PAd=1Zr~u=3FngMN)v7W7a2OXCE2_7R=eIw~Sv?JiF5sdFVlVxMQvc zAz9gM1i1t0-hUrno)vMOYD2hZbLUF4S&cwogsEw)rbNww(%@@VJnhwsjyIpAXI5

o#^%+Y#7Vi$>E~XX ztks_xr|6pWQNV(Vb~9O0TZz$=P3X5C7WB_dbYrU8dN}>`paALHV$AIQ;3HDp`qE+Y z{UJYU`})AraK%k2#o|bzGurV|EgLGkyvhqhy1b4Af6v z-4D_1e1$kk+*pz)ir~x=8x(^kE;aCsFE|JdaK7<;S@$Bh#(biwS+(FqRKBVxD}O)% zw+pQSVMt^oWfA-Dh3uosli7&cO_>rOBr-XS&!GvE!y1D7G$oVY=h)>&`q%C*?l~qh z%*g_Q4z(goiKcYcJrpIxL89(Tm{Ogg{|uC8ZlVG6D(HIC9gCB!ch-ZioVzZjY>@Y) zW>+)Bvjy!9A3xuYWdVpXT{eeGo3%ty#PNzDHP47-Rd4@1(>qr%!uRfPo<6^?jmQ}< zQhhnB0t4g#CTRto$3-p{;=BbeUOcX2h>vw0_4hJ%RP9Lt*e;jI;LZEz^!M!IocuB& zz20@8Llf&8+2DZ`+V@4%L7Qe6XU}}b$HDV z)O7|XO8`NEKf#)oPz=C04O@5h zMronv&V%NMv)mA_$Qhkme`V`vgT{XDpk%B=d4+$%vbuHH{Gc&Ye!sU$Zpm`v zxJ^LAZVP>sbK1q}_^*#gkK!%*7WFO7RZL{>Lgqa;!y78INV|ZEd}{M8WJl^ERjlah zxZjz)w?&d(TrJEz(+>`7w>~ZE<(xTt`S@91cpwmgGyNz)EOOB~$euV@q@wUZgXARW z_4DWzNqsQ)&O9`eQj`Ixh^vq-mUbxN0(4Bi@mAZ593TZ$4Ch0-{W>Hn<_>aQ$pxHQ`}@@95T+~0FnJ&$TPgWRqa4|Tb0WE#;$~cPI~O_6vz01)Mrmm z>;(=W<7z*ZKe1_wfoQ!LgXVuRa1ClJoDl8fR6xr-j8Ir!ZLoQ&^vKEm^A|S$4i+JE zLvJuJda;wxFf`2-8KSo^e(e?*l1)8bMKth#ll`eE-No&;&($=NiG)iO?BSsHYj)Gr zzNuX-qc8~IvqO(PA;n=ZP2!A1zRFC=RGaBR|5(?Oni@+XZ;29j7e@ve-s5EE%4uG` zKGat%opp%7*oEgZ6!vJJNFhj>=9w2+=R%9(16OZ@IYw5lP_}?5Kg}5u8{`-_<6tnls^i9@Ab{%$z z1?yNJ*&h$Pi!Mu#MI0uYo~j#AEJq40q?9?>b+e3Lo35&WS2W33dPqEI`KrOl&cesA zzS6cbu036~xRBjggm5Gp7Tx2=^OaZ0%d~zOQiAUvXHIsroL0U6u}0*PvvuDOB9*W- zF!fcHXjyenmdL*m;jfK3Q6Vlh2D20j zVBPm-vYKIg5$B(nMF!l<4YE6>RcFiNal}T*=@XB@E}oVXvwp2F+aGCgIfH}TqnZ$v zmI9}-92dz3V8C}y4mqoNE9ca^SB1F8Lt589bd=dKvUbh-O_|;eIaj=)a~Ek;v1$!Q z$|6ZVWS*JB(^*16$)^2$sa&-1Y}+$$uti4tNsd+tg7V*)QU_wF@0~I>8y0}FJRv~o zyAyK}7DTW3uUUASc5~%Bg*xHZvgSas?|>82&CX0mCBMY^zCwE+EV%0F^6YmCefI>h zz?3XA9q&COw$Z2*jposDnOQpf)h+DQ}yP)F+;9O@`=?)eu@L znTO&<2d=L}nNm2Q*LwuuW`?YF@j2iNAK@y-$zRnTAMY*Raq&c72g4xZ30` zZqC?zG7xFAu=AFa{e)Gvnrzgh^wZfyDEdK>I80L(7ea(kzT_#}qEV++PE3{-cF6j? zT*c(H!rVK` zJBeo@3qDjt}4i~4jGisrv% zt(H0}vCfp>xsgROARoj5$Z8n?6;h$Nf?ZxkZiClWDqTkYD+K<*8$Eib0{G#WqL^kd$PEes1&g-SW$6|kIgrb&rCI2ksr~ivIt@H z-7|F;W(cBG8u-okHH6^_?!ela^*!mct#tQ^mq{JgD_|4 zuAl}?bB3>HrhM$Vbemz9DwFHpOo1Eb(%w>GQx+#;3idYnAN|yGl4THHwy_iorV^Dz1gzw?*VtKMx`g+jg5fg?eY1WMuesWc_I5S~sBNpNI zV{U}vW1fsv`j%zb6kmss+Z9Ns1jBGL_}BWo3?+!HsPr+BG)BO*dWV%LMti(&^_q#b zV>{jK&C%2^EjSwUeA@1l94|5P;h~AIo#29@_wA9(D9P%HYa!9)uf%Lv*uA`=E^op9LNcP{b|n=-y+*@ z*6^_xjZ!q{Fb&PdwvU~9sT&&2$mkFza{V-g3yL^0T3E5fVqr15k(Rq8b5=;~B1^&v z9i8j>^Fb2AWa}Ah8|vdMPHAygc5#eo*1pA=eXULk8gX5Xrl~h^EX6 zRl3z@qooF=A2kY;Z}E2weequ?DftSdE$*-w(9cg0T#BAqG9ZB%B`h{biDW(EmZZah z^K$L#x_vVGod2N*GRrxtAn<*=daTNb4teT=^x*CZ+g*7P4@`J z)KuX>^R)=4pmGiBiEN*{~Su zb>d2^Lcm1xmP|!^ypx88NOZF6sFO>~fe&iQU5yv>_my#Tvg`!IjrG)jj8iUiTinE_ z10EqWc-`Q7c}3V-tN7zUe`t!&v->9OXY;hr-U!;lSQ36v-3n38U_EV-jZ0S+EYtO4 zrabYw;k_BzpVBW9hT1}-4_7f}P`?lao`EZ2zCZ?@8+-^)z9TTE1_5lRQ#RQr1GZ%E zu;rX!okndt{h)E*u5JmhTn1s@CL4K@?{Na@9?naVgR5D~>s3mg?dy{Yha62kgHnwk zC-Ih+5Ozma2_RlRiqaR_(2WA_gQP47$r86Fs3y33?X&hYk zBTE$vUdJUIM3lI5g42(DpZ?62ZlKH&p5&agGe8+SlD-~ z!Q_^k%|jIxUYQ#}h#jm?mDxE}_U=&DS7X3umANe)Bw$IIWM3D2KLmT?I{Uezg50_o zZ?)P|y&$*56^3f4;%Tp>5Eos2k`h0;wrPs_gOZLVUe~tYU|@~hQ)P|${<6OAly+GS z#O=A5$+RyRy9?mxyx!B&AqSqc!FSi0;b9TB4%DQa$AD_dK(J6rb}P zVl^5HY(X7a)2GOmd=C$cLwFOfdhW+EqhkCcGdk@wJ5M%sO_sj85(gOBX71|Fl5WuQ zmXRKbJV9kv1h;TSuMi>k!NRoIneMUEN<~v`Etl?O_U5ox&b3Lq^##f8Eo~>SP>#*N zLm&w@u-&zNpi~Q#`fBwTD~zA$&12=HOl6;34}z?Ok~^lzrc~ChLe)*2z{1k?drM zEXk6xM9iISvL%W}CWEqX;YNtDMz*nxr7UBt$yQ-73NtE^WrpICF}+vsdEbAY`+1&o zp7*@xIp=+kKjxq7T-Q0*{I2i%`TahhuLfh(v8GZfRt+wJGHIJw%v}|j8?9K{xY4#V z`t*HMjyqyhCjoK(E9(Gkz6s6UxaZ#qb~zo2UU$s&^2iCTdrG!)rEO6YdLzX5O`4|k z!QjNY1ypKHpgyb+UyJWshoFM?vgt>$OZEX4FTVctsEMm_lJ+$BGqH-3e4?Fy?NY@F z%amg_a_ldf?{<^Q%f}TX|GUOXRw|-HWGV7~l4{Pvx(3BamnHX%VjpL<`KJSSa&4Oj#DalqU(M*B_ww+rkVs5JGB#vE zk?Xu+&#cV_Ql;=OCSvTItIUN9mRq))Vq!YA*b9509+D@>(+?ZR^QDus zdUV=t2F+8hJi2=QUR+`HC#&1_Qb}WkJGrJA_3DT+=@t%9w;oOxOxf}Ursszd6#t0h zII2|~U!FSFzsifJZs};N>$%F0PD*RKFH96wC81wtducd2c;KInnodFVD@`qKB>*h4 zEfWd~stz$x1sNk#jl37Qr$vY--M~hrn(P<8Y8>pUcNf!20d3=^V+s!>dJ-lPCTBj+ zUnEGzuWAynSq#cJ))$rYOQH~e99B){T4fQ5AYutx~ zkA1j2r&_PfHyM`cO7VrU?>& zo;=L_8BVo_^J(;~PYZU#1>3~1;jX8I#q^mAN1fUOx7p# zgD?z1F-!&Uryy2}$#|~mS>vE}mXaLx7w+x{Zl!wM$K@hsH%cS^ly_r)H!g5(@~b+Y zya(xZ6yJ_Uncwt(v80{Xmf4(Xf2FF{imlkE?`vnJJZmIxn4LG2YJv$rMHvWM=Ro57 z@`c9MhKd#~#;^tb>8A!Fg+E1{}sTba;5m`)aYVfN} zIp9UydNnw$nWN)V(vgM;fA!hXVN7(wjAS_$%?4T0=?XhXO-WI`g}OMO=iTk{eoe?x zzFqr^&Qk}KXsE7lx90Kr&EYU4u)yFn+f_og>q4G;HD%?Cy`xKInTPs6ujg*R_u<%6 zwbf7Y8<}>A8(O5#R|lB`wo(Y_aM(dS0t>kTA{D=wVv49+d7+vwUmE3v#^ihGM|?`7 zn7O;*n74no#JzB^*`zv|e{*24AO$an9KP`Z*=8l?o-{0W!O14_viw#g zUP(_1O*qv7L0zZu_Q!$1q8X4^<7}?F(bb$Yk@v>2WbH;T2T64&M>9f`i;syV{QNqD z^9LGH4l4L_dDdm0H2r?|_R3@B>wn}pd@@Xqa=w&06~Ac($RSqqZtg0h&g5RA8rOO1 zo%9JbkVthmvbVWDv(Bsbxd+ms;9`RdScmE+Dj$88iXHJ8kwHxKDxiI-h=T zrzG2++3w*l*uWz&AdypHCMIyv)c|x+@4Xx;qTi2~ZT1U;ONA}TpNU(bLP<8_Y^`7Y zq34w0_ITDmuAedoLNhXsmw>wNNwzM zUT<)*Jmz5u3_d&+(vx>&#^r60Ecf}+H_vXn*uEQ&0So7n*f+}|Z1#(mAk@i*qb^2x zj4N~;qI&^Z^A9;}1GjVRvMQ>s45TI=iId`u*oY1ao*|rk7LHQfwOnd~<0k1sh%UVY zR9&}g&D5cvveE6TP}P9m;*o7q9O5g_*_V$${bH(i#)Pu8cQd#V)K)l=4go7sVisHa z((JKT{QDm^@lLOuudb~6$-6yH8EU>YD4{VRvM9s)(`{Fi@fx&pm`D2w$|yNAL)!cP zx8^I8L8sV>l@By-&&ZIk*y)C&TGBKJnBz0QwD0!hi-7)+c#j)Y7DHk!n)Vn%Ba!aM-0%Hi02a1bVz{33UKB$E8M*-WaP#fxORe#KRwiSL_bdXZMOB=gr`IQrF63!s07Gnh3kq_!0dg9eh}pBvy#2NpWT?grTu)Wtu0N; zzU%UpyBl%tf4*4M!WQBY{!@?U%T^XDXOV*!QmknjNkK1ou^d!AI<_?Z2uj5#b@nq5;exJh|_a%HkJ zCB`3|DBc{w5Wf6Pfg$`>bx)mQv|E3}sWagmc2`l(^?VXiqZ2X(n@6ndQ9?1Xjt`de zm1*rCQsjZ$UVHiS<+E)Sc81&s7fh!Kk|iXM;~6SKGWH}@Ci_hg5@7G{XiV?JaH;mB zS`3M}H0Y*AYAD-vENPR6l ze;d?TH`Nn>J1%3m+Js*?fIyQofHuz;=8v{dHg}kII7DC*qdD8h-b11c3%nn&6XcJS zAp$%<1##>83B%Iu2cZpCD*qMXU(H_St`v=YiDhm02?Kt(f zGJ*8g`^?+dqqVTuudF*a!7uJdcW&BChk1`4qALM!Xh86s^Yu2h^3mWf&8nbK(ZtU8 zuPrB{o|;K)ZhW~Ic!9;|=L4EO$U~n;^SS_oy+0eDrxY&Lisn{d*szVCm5ZSrbxRIp zi5L|44Gr|eaNBgjv3ZVFlL~A+-P}KfJbE~a^Gsu*s`YSi1(U&ju~aj$-NQF;L_Og- zvsWNTgE6BCG7JtNY}+cG!jYhRec|Idv!GB5$kld-d!=$=)^EECU$Z7vN}UpB;}6gm zN3RldOjcpLEXedeIL9KIe`%E$8~c6QHTsqaW#)40fu+Y+O9JUnWQK&E9GU6V65dH* z9Gu%V25iKOw8ouxrasO7Q6+F5wnKYp-4_CMdptqQ=c6AsYP!&=Fmc6nkYFyvdqC7bC_=C zRdBK5V`)a@C7DdhGUdWy8RZLU7ALrQd#x*2xcRlPeJOzAc6g9ao;*3GXGZ6Kx@B$d zwRirDVpTRD51zzvj3J{J^0TdV;&%r@6-B|ac{IWQm=nr$!e0#^tEW_Vnh0(+)MRe1 zoN23^=_)U;Da*LODGZUjHaP`9#1KVFQx+eoDS;aShwkM*^f#C&+Nkk6!%QH&d8I5_ zoOt&79<&EqcX6b;V9v?gPD4WSJ&(P5x(xL->uZ&Iu3@+yWqSygtblR2g1nLucIax7)F9Sl@#)aFiXbYM;KO|s81)oOq1W)SZ5 ztsB|aem3j+*<^YlRZ?2_gHqOZtPn_LqdMS8f)r?Xo*)5ymd=3=C|t@vg==dgo~ioK ztv1}=T5^S?5O1hplMmdDDB`pB&4WH&9E3NvGt91;4Cs|EduN2Y-OP5LRaYCP8Jd?}d zc$#KRGpo}mmi%mBQ9Zp~VDp^RHwP1f$g(_7)=_^|X8PB) z(t%{H>oMtvjZVF(h;~cg0slOkc573N3qbH!h)+9@iTPyf%6v*Ji=6WIsC8IT#}p6( z-hu^SeEYNXFS-qdBPse;rbu@(Ish4CK`1P zb?H|4<|p0d4zKm?ZkTM^4f3M8s|%=?vHT%|>^Sr~D{(C=;l zi*&1#={#pqccL~m?6DyI2jew^8GzG3;QtiuQ5XEs_lK~a9KHsPB$wIQA4hCe)XS^V z_2(XFWlDXPO^mTNv(cVhMPnY5xVH-2O{X)!lgqHpB z#F>VMBTrjAQ{y~k(yV&*z!QIwFcd5GcB>a9g%gynr`V-dIShOuJVaDFIE{*mMbF4M zdYiUJ94AK`NQJ}B`HxZ1AX2l{vRM^2QAO9KL55=cIjg>UT3YAxdmoXl6-`vf zH+HmpE>@2pcro8DO@V~}kZbhgc4{)O)G4<#ln$~Jw=w#|&d66q^K>Ww#So2Eja?rX}6y1*B72%wvJSOP?5R-Z9vln*U8N* ztqXE=C+fF8hW5+wi`o`N%MVH$mNa;^eK%#Bnj4z=mI%ftBbitAp&|d|Y}H@pOWFUe z@W{h&aCVg1FQz}E>0sSd2kpZA6#EX!O8%3xU(jFw7Z?(deheVTNZEBpq2B+1gu}QP ze8Zt+9p_8k)_SKIll3FdrJ4mo0!VMDI2Nr>azi7Z6GNB{s@H*&m0^?k1G;Kp5j02{ zC1l?Nxl>aXFWSNsMTS$=58V(=Ad3-GRfpAg%g$EH#NkX;N=}V}>Meg+>wiQb{-0j| z-}!d`l#T(TAKoxZ*n$EH3Nc)1KuFwoR_C#S0!W7(HVagoX#B%Ma$cMm#9c zXhK)VucUxV8=;`mhI%vTW#z2PT=HO15BQ;{p^W%#P<6r&+~MfG0#@eOU1?BE!Ua^g z-~~4vfC{Fck?}idhA4c?1XTEF2UR~roj^&8Yv^vD-_6%|``^Q$9{Hddr^EFMiaOvR z8acTK#@Dad-^vvSz@tcXTPT$T<`meT&^oA+L7D@*(gH~N{@d=a?Y<@Kd)axLRg6o8h!VaOhX(n6i;5a5s%!AdGSX?mGKWde@3nKQ-Uald{ z6|eE9ja28f|uA0GWDFaY^+rSJamGZ`@y=zcG`2 zquBSE{Ydej5p&-a_FZA$75=@hpf-j?khjCzj;d};@KK2=h30`3=^fcO*>f*d1hPK8 i!p&rHh#)5?`DE literal 0 HcmV?d00001 diff --git a/tutorials/propensity_stratified_evaluation.md b/tutorials/propensity_stratified_evaluation.md new file mode 100644 index 000000000..59c40b21f --- /dev/null +++ b/tutorials/propensity_stratified_evaluation.md @@ -0,0 +1,68 @@ +# Propensity-based Stratified Evaluation Method + +![results](closed_loop_diagram.jpg) +Recommendation systems are often evaluated based on user’s interactions that were collected from an existing deployed system (see the above figure [Jadidinejad et al.](https://doi.org/10.1145/3397271.3401230)). Users only provide feedback (r) on those items they were exposed to by the deployed system (e). Hence, the collected feedback dataset used to evaluate a new model is influenced by the deployed system (RecSys), as a form of closed loop feedback. In this situation, [Jadidinejad et al.](https://arxiv.org/abs/2104.08912) revealed that the typical offline evaluation of recommenders suffers from the so-called Simpson’s paradox, which is a phenomenon observed when a significant trend appears in several different sub-populations of observational data but that disappears or reverses when these sub-populations are combined together. In addition, they proposed a novel evaluation methodology (Propensity-based Stratified Evaluation) that takes into account the confounder, i.e. the deployed system’s characteristics. + +See [the paper](https://arxiv.org/abs/2104.08912) for more details. + +Using the proposed propensity-based stratified evaluation method is as simple as using the classic evaluation in Cornac: + +```python +import cornac +from cornac.models import MF, PMF, BPR +from cornac.metrics import MAE, RMSE, Precision, Recall, NDCG, AUC, MAP + +from cornac.eval_methods import PropensityStratifiedEvaluation +from cornac.experiment import Experiment + + +stra_eval_method = PropensityStratifiedEvaluation(data=ml_100k, + n_strata=2, + rating_threshold=4.0, + verbose=True) + + +models = [ + MF(k=10), + BPR(k=10), +] + +metrics = [MAE(), RMSE(), Precision(k=10), + Recall(k=10), NDCG(), AUC(), MAP()] + +exp_stra = Experiment(eval_method=stra_eval_method, + models=models, metrics=metrics) + +exp_stra.run() + +``` + +Compared to [the classic evaluation](https://github.com/PreferredAI/cornac#getting-started-your-first-cornac-experiment), you can simply use `PropensityStratifiedEvaluation` instead of `RatioSplit`. The output is based on the defined strata (Q1, Q2,...) and the "Unbiased" row represents the performance prediction based on Stratified Evaluation method: + +``` +[MF] + | MAE | RMSE | AUC | MAP | NDCG@-1 | Precision@10 | Recall@10 | SIZE +-------- + ------ + ------ + ------ + ------ + ------- + ------------ + --------- + ---------- +Closed | 0.7450 | 0.8988 | 0.7422 | 0.0396 | 0.3038 | 0.0450 | 0.0346 | 19956.0000 +-------- + ------ + ------ + ------ + ------ + ------- + ------------ + --------- + ---------- +IPS | 0.7450 | 0.8988 | 0.7422 | 0.0396 | 0.1907 | 0.0011 | 0.0286 | 19956.0000 +-------- + ------ + ------ + ------ + ------ + ------- + ------------ + --------- + ---------- +Q1 | 0.7464 | 0.9001 | 0.7395 | 0.0377 | 0.2992 | 0.0404 | 0.0299 | 19843.0000 +Q2 | 0.5403 | 0.5403 | 0.9915 | 0.1254 | 0.2971 | 0.0417 | 0.4175 | 113.0000 +-------- + ------ + ------ + ------ + ------ + ------- + ------------ + --------- + ---------- +Unbiased | 0.7453 | 0.8981 | 0.7409 | 0.0382 | 0.2991 | 0.0404 | 0.0321 | 19956.0000 +``` + +`SIZE` column represents the number of feedback in each stratum (`Q1` or `Q2`). `Unbiased` row represents the estimated propensity-based evaluation per each metric while `Closed` row represents the classical evaluation. + +## How to cite? +Use the corresponding bibtex entry to cite the paper: + +``` +@InProceedings{simpson_recsys20, + author = {Amir H. Jadidinejad and Craig Macdonald and Iadh Ounis}, + title = {The Simpson's Paradox in the Offline Evaluation of Recommendation Systems}, + journal = {ACM Transactions on Information Systems (to appear)}, + year = {2021}, +} +``` \ No newline at end of file From d2a74f01add622565ac7196b37a7d1b191b04361 Mon Sep 17 00:00:00 2001 From: "Amir H. Jadidinejad" Date: Sun, 23 May 2021 15:09:06 +0100 Subject: [PATCH 2/8] update the example to be compatible with the paper --- tutorials/propensity_stratified_evaluation.md | 47 ++++++++++++------- 1 file changed, 30 insertions(+), 17 deletions(-) diff --git a/tutorials/propensity_stratified_evaluation.md b/tutorials/propensity_stratified_evaluation.md index 59c40b21f..749d2e570 100644 --- a/tutorials/propensity_stratified_evaluation.md +++ b/tutorials/propensity_stratified_evaluation.md @@ -9,22 +9,24 @@ Using the proposed propensity-based stratified evaluation method is as simple as ```python import cornac -from cornac.models import MF, PMF, BPR +from cornac.models import WMF, BPR from cornac.metrics import MAE, RMSE, Precision, Recall, NDCG, AUC, MAP from cornac.eval_methods import PropensityStratifiedEvaluation from cornac.experiment import Experiment -stra_eval_method = PropensityStratifiedEvaluation(data=ml_100k, +ml_dataset = cornac.datasets.movielens.load_feedback(variant="1M") + +stra_eval_method = PropensityStratifiedEvaluation(data=ml_dataset, n_strata=2, rating_threshold=4.0, verbose=True) models = [ - MF(k=10), - BPR(k=10), + WMF(k=10, seed=123), + BPR(k=10, seed=123), ] metrics = [MAE(), RMSE(), Precision(k=10), @@ -34,26 +36,37 @@ exp_stra = Experiment(eval_method=stra_eval_method, models=models, metrics=metrics) exp_stra.run() - ``` Compared to [the classic evaluation](https://github.com/PreferredAI/cornac#getting-started-your-first-cornac-experiment), you can simply use `PropensityStratifiedEvaluation` instead of `RatioSplit`. The output is based on the defined strata (Q1, Q2,...) and the "Unbiased" row represents the performance prediction based on Stratified Evaluation method: ``` -[MF] - | MAE | RMSE | AUC | MAP | NDCG@-1 | Precision@10 | Recall@10 | SIZE --------- + ------ + ------ + ------ + ------ + ------- + ------------ + --------- + ---------- -Closed | 0.7450 | 0.8988 | 0.7422 | 0.0396 | 0.3038 | 0.0450 | 0.0346 | 19956.0000 --------- + ------ + ------ + ------ + ------ + ------- + ------------ + --------- + ---------- -IPS | 0.7450 | 0.8988 | 0.7422 | 0.0396 | 0.1907 | 0.0011 | 0.0286 | 19956.0000 --------- + ------ + ------ + ------ + ------ + ------- + ------------ + --------- + ---------- -Q1 | 0.7464 | 0.9001 | 0.7395 | 0.0377 | 0.2992 | 0.0404 | 0.0299 | 19843.0000 -Q2 | 0.5403 | 0.5403 | 0.9915 | 0.1254 | 0.2971 | 0.0417 | 0.4175 | 113.0000 --------- + ------ + ------ + ------ + ------ + ------- + ------------ + --------- + ---------- -Unbiased | 0.7453 | 0.8981 | 0.7409 | 0.0382 | 0.2991 | 0.0404 | 0.0321 | 19956.0000 +[WMF] + | MAE | RMSE | AUC | MAP | NDCG@-1 | Precision@10 | Recall@10 | SIZE +-------- + ------ + ------ + ------ + ------ + ------- + ------------ + --------- + ----------- +Closed | 1.0864 | 1.2743 | 0.9009 | 0.0658 | 0.3791 | 0.0799 | 0.0625 | 200008.0000 +-------- + ------ + ------ + ------ + ------ + ------- + ------------ + --------- + ----------- +IPS | 1.0864 | 1.2743 | 0.9009 | 0.0658 | 0.2003 | 0.0049 | 0.0459 | 200008.0000 +-------- + ------ + ------ + ------ + ------ + ------- + ------------ + --------- + ----------- +Q1 | 1.0911 | 1.2785 | 0.8987 | 0.0572 | 0.3637 | 0.0612 | 0.0479 | 197556.0000 +Q2 | 0.7544 | 0.7658 | 0.9935 | 0.1985 | 0.3680 | 0.0653 | 0.5584 | 2452.0000 +-------- + ------ + ------ + ------ + ------ + ------- + ------------ + --------- + ----------- +Unbiased | 1.0869 | 1.2722 | 0.8999 | 0.0589 | 0.3638 | 0.0612 | 0.0542 | 200008.0000 + +[BPR] + | MAE | RMSE | AUC | MAP | NDCG@-1 | Precision@10 | Recall@10 | SIZE +-------- + ------ + ------ + ------ + ------ + ------- + ------------ + --------- + ----------- +Closed | 2.0692 | 2.2798 | 0.8758 | 0.0623 | 0.3720 | 0.0723 | 0.0547 | 200008.0000 +-------- + ------ + ------ + ------ + ------ + ------- + ------------ + --------- + ----------- +IPS | 2.0692 | 2.2798 | 0.8758 | 0.0623 | 0.2026 | 0.0077 | 0.0547 | 200008.0000 +-------- + ------ + ------ + ------ + ------ + ------- + ------------ + --------- + ----------- +Q1 | 2.0840 | 2.2929 | 0.8730 | 0.0428 | 0.3390 | 0.0387 | 0.0292 | 197556.0000 +Q2 | 1.3782 | 1.3908 | 0.9998 | 0.5807 | 0.6892 | 0.1170 | 1.0000 | 2452.0000 +-------- + ------ + ------ + ------ + ------ + ------- + ------------ + --------- + ----------- +Unbiased | 2.0754 | 2.2818 | 0.8746 | 0.0494 | 0.3432 | 0.0396 | 0.0411 | 200008.0000 ``` -`SIZE` column represents the number of feedback in each stratum (`Q1` or `Q2`). `Unbiased` row represents the estimated propensity-based evaluation per each metric while `Closed` row represents the classical evaluation. +`SIZE` column represents the number of feedback in each stratum (`Q1` or `Q2`). `Unbiased` row represents the estimated propensity-based evaluation per each metric while `Closed` row represents the classical evaluation. It reproduces Table 1-b (or Table 2) in [Jadidinejad et al.](https://arxiv.org/abs/2104.08912). Due to the random splitting, the above numbers are slightly different with the paper but the insight is the same! ## How to cite? Use the corresponding bibtex entry to cite the paper: From ec2e4b5d28ab1ad3c69db93b17aa39fc44c4cf78 Mon Sep 17 00:00:00 2001 From: "Amir H. Jadidinejad" Date: Sun, 23 May 2021 16:42:33 +0100 Subject: [PATCH 3/8] clean up --- tutorials/propensity_stratified_evaluation.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tutorials/propensity_stratified_evaluation.md b/tutorials/propensity_stratified_evaluation.md index 749d2e570..029525463 100644 --- a/tutorials/propensity_stratified_evaluation.md +++ b/tutorials/propensity_stratified_evaluation.md @@ -1,9 +1,9 @@ # Propensity-based Stratified Evaluation Method ![results](closed_loop_diagram.jpg) -Recommendation systems are often evaluated based on user’s interactions that were collected from an existing deployed system (see the above figure [Jadidinejad et al.](https://doi.org/10.1145/3397271.3401230)). Users only provide feedback (r) on those items they were exposed to by the deployed system (e). Hence, the collected feedback dataset used to evaluate a new model is influenced by the deployed system (RecSys), as a form of closed loop feedback. In this situation, [Jadidinejad et al.](https://arxiv.org/abs/2104.08912) revealed that the typical offline evaluation of recommenders suffers from the so-called Simpson’s paradox, which is a phenomenon observed when a significant trend appears in several different sub-populations of observational data but that disappears or reverses when these sub-populations are combined together. In addition, they proposed a novel evaluation methodology (Propensity-based Stratified Evaluation) that takes into account the confounder, i.e. the deployed system’s characteristics. +Recommendation systems are often evaluated based on user’s interactions that were collected from an existing deployed system (see the above figure/[source](https://doi.org/10.1145/3397271.3401230)). Users only provide feedback (r) on those items they were exposed to by the deployed system (e). Hence, the collected feedback dataset used to evaluate a new model is influenced by the deployed system (RecSys), as a form of closed loop feedback. In this situation, [Jadidinejad et al.](https://arxiv.org/abs/2104.08912) revealed that the typical offline evaluation of recommenders suffers from the so-called Simpson’s paradox, which is a phenomenon observed when a significant trend appears in several different sub-populations of observational data but that disappears or reverses when these sub-populations are combined together. In addition, they proposed a novel evaluation methodology (Propensity-based Stratified Evaluation) that takes into account the confounder, i.e. the deployed system’s characteristics. -See [the paper](https://arxiv.org/abs/2104.08912) for more details. +See [the original paper](https://arxiv.org/abs/2104.08912) for more details. Using the proposed propensity-based stratified evaluation method is as simple as using the classic evaluation in Cornac: @@ -66,7 +66,7 @@ Q2 | 1.3782 | 1.3908 | 0.9998 | 0.5807 | 0.6892 | 0.1170 | 1.000 Unbiased | 2.0754 | 2.2818 | 0.8746 | 0.0494 | 0.3432 | 0.0396 | 0.0411 | 200008.0000 ``` -`SIZE` column represents the number of feedback in each stratum (`Q1` or `Q2`). `Unbiased` row represents the estimated propensity-based evaluation per each metric while `Closed` row represents the classical evaluation. It reproduces Table 1-b (or Table 2) in [Jadidinejad et al.](https://arxiv.org/abs/2104.08912). Due to the random splitting, the above numbers are slightly different with the paper but the insight is the same! +`SIZE` column represents the number of feedback in each stratum (`Q1` or `Q2`). `Unbiased` row represents the estimated propensity-based evaluation per each metric while `Closed` row represents the classical evaluation. It reproduces Table 1-b (or Table 2) in [the original paper](https://arxiv.org/abs/2104.08912). Due to the random splitting, the above numbers are slightly different with the paper but the insight is the same! ## How to cite? Use the corresponding bibtex entry to cite the paper: From 075f528cbdf2a5d873fcef45620a5ffa6a8ffa39 Mon Sep 17 00:00:00 2001 From: "Amir H. Jadidinejad" Date: Sun, 23 May 2021 17:35:22 +0100 Subject: [PATCH 4/8] clarify the name as propensity stratified evaluation --- tutorials/README.md | 4 ++-- tutorials/propensity_stratified_evaluation.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/tutorials/README.md b/tutorials/README.md index 17c74947c..915b65ab9 100644 --- a/tutorials/README.md +++ b/tutorials/README.md @@ -23,7 +23,7 @@ An important focus of Cornac is making it convenient to work with auxiliary data - [Text to Graph transformation](./text_to_graph.ipynb) - [Visual recommendation algorithm with text data](./vbpr_text.ipynb) -## Stratified Evaluation +## Propensity Stratified Evaluation Most datasets in the field of recommendation systems (MovieLens, Netflix,...) suffer from [closed loop feedback](https://dl.acm.org/doi/10.1145/3397271.3401230). In this situation, [Jadidinejad et al.](https://arxiv.org/abs/2104.08912) proposed a stratified evaluation method to better estimate the performance of different recommendation models. - - [Stratified Evaluation](./propensity_stratified_evaluation.md) \ No newline at end of file + - [Propensity Stratified Evaluation](./propensity_stratified_evaluation.md) \ No newline at end of file diff --git a/tutorials/propensity_stratified_evaluation.md b/tutorials/propensity_stratified_evaluation.md index 029525463..9ab7113ec 100644 --- a/tutorials/propensity_stratified_evaluation.md +++ b/tutorials/propensity_stratified_evaluation.md @@ -38,7 +38,7 @@ exp_stra = Experiment(eval_method=stra_eval_method, exp_stra.run() ``` -Compared to [the classic evaluation](https://github.com/PreferredAI/cornac#getting-started-your-first-cornac-experiment), you can simply use `PropensityStratifiedEvaluation` instead of `RatioSplit`. The output is based on the defined strata (Q1, Q2,...) and the "Unbiased" row represents the performance prediction based on Stratified Evaluation method: +Compared to [the classic evaluation](https://github.com/PreferredAI/cornac#getting-started-your-first-cornac-experiment), you can simply use `PropensityStratifiedEvaluation` instead of `RatioSplit`. The output is based on the defined strata (Q1, Q2,...) and the "Unbiased" row represents the performance prediction based on Propensity Stratified Evaluation method: ``` [WMF] From c6ba3f51b21b64c55cda82bf4651448bd8a6f22f Mon Sep 17 00:00:00 2001 From: "Amir H. Jadidinejad" Date: Sat, 5 Jun 2021 11:19:22 +0100 Subject: [PATCH 5/8] add a usage example of propensity stratified evaluation --- examples/README.md | 2 + ...ropensity_stratified_evaluation_example.py | 56 +++++++++++++++++++ 2 files changed, 58 insertions(+) create mode 100644 examples/propensity_stratified_evaluation_example.py diff --git a/examples/README.md b/examples/README.md index e049092d3..6ed0cfb84 100644 --- a/examples/README.md +++ b/examples/README.md @@ -8,6 +8,8 @@ [given_data.py](given_data.py) - Evaluate the models with your own data splits. +[propensity_stratified_evaluation_example.py](propensity_stratified_evaluation_example.py) - Evaluate the models with [Propensity Stratified Evaluation](https://arxiv.org/abs/2104.08912) + [vbpr_tradesy.py](vbpr_tradesy.py) - Image features associate with items/users. [c2pf_example.py](c2pf_example.py) - Items/users networks as graph modules. diff --git a/examples/propensity_stratified_evaluation_example.py b/examples/propensity_stratified_evaluation_example.py new file mode 100644 index 000000000..56ab67986 --- /dev/null +++ b/examples/propensity_stratified_evaluation_example.py @@ -0,0 +1,56 @@ +# Copyright 2018 The Cornac Authors. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================ + +""" +Propensity-based Stratified Evaluation Method + +Reference: +--------- +Amir H. Jadidinejad, Craig Macdonald and Iadh Ounis, +The Simpson's Paradox in the Offline Evaluation of Recommendation Systems, +ACM Transactions on Information Systems (to appear) +https://arxiv.org/abs/2104.08912 +""" +import cornac +from cornac.models import WMF, BPR +from cornac.metrics import MAE, RMSE, Precision, Recall, NDCG, AUC, MAP + +from cornac.eval_methods import PropensityStratifiedEvaluation +from cornac.experiment import Experiment + +# Load the MovieLens 1M dataset +ml_dataset = cornac.datasets.movielens.load_feedback(variant="1M") + +# Instantiate an instance of PropensityStratifiedEvaluation method +stra_eval_method = PropensityStratifiedEvaluation(data=ml_dataset, + n_strata=2, # number of strata + rating_threshold=4.0, + verbose=True) + +# define the examined models +models = [ + WMF(k=10, seed=123), + BPR(k=10, seed=123), +] + +# define the metrics +metrics = [MAE(), RMSE(), Precision(k=10), + Recall(k=10), NDCG(), AUC(), MAP()] + +# run an experiment +exp_stra = Experiment(eval_method=stra_eval_method, + models=models, metrics=metrics) + +exp_stra.run() From c2ec5c3bf23cb4644235003ccc0b64a913c9e060 Mon Sep 17 00:00:00 2001 From: "Amir H. Jadidinejad" Date: Sat, 5 Jun 2021 11:22:40 +0100 Subject: [PATCH 6/8] remove propensity based stratified evaluation tutorial --- tutorials/README.md | 7 +- tutorials/closed_loop_diagram.jpg | Bin 60334 -> 0 bytes tutorials/propensity_stratified_evaluation.md | 81 ------------------ 3 files changed, 1 insertion(+), 87 deletions(-) delete mode 100644 tutorials/closed_loop_diagram.jpg delete mode 100644 tutorials/propensity_stratified_evaluation.md diff --git a/tutorials/README.md b/tutorials/README.md index 915b65ab9..2b5c74144 100644 --- a/tutorials/README.md +++ b/tutorials/README.md @@ -21,9 +21,4 @@ An important focus of Cornac is making it convenient to work with auxiliary data - [Working with auxiliary data](./working_with_auxiliary_data.md) - [Text to Graph transformation](./text_to_graph.ipynb) -- [Visual recommendation algorithm with text data](./vbpr_text.ipynb) - -## Propensity Stratified Evaluation - Most datasets in the field of recommendation systems (MovieLens, Netflix,...) suffer from [closed loop feedback](https://dl.acm.org/doi/10.1145/3397271.3401230). In this situation, [Jadidinejad et al.](https://arxiv.org/abs/2104.08912) proposed a stratified evaluation method to better estimate the performance of different recommendation models. - - - [Propensity Stratified Evaluation](./propensity_stratified_evaluation.md) \ No newline at end of file +- [Visual recommendation algorithm with text data](./vbpr_text.ipynb) \ No newline at end of file diff --git a/tutorials/closed_loop_diagram.jpg b/tutorials/closed_loop_diagram.jpg deleted file mode 100644 index 7d8117084a242e83987debf56038e197cc96639e..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 60334 zcmeFZcU%)sv^TsV^xmX5DIy)|Js{FV1VK88^d`LpM1mk7MNkk>EC?uw2%#4#p@X0z zMUYN{f=V%=glH0Y7k}m6_dfsJ&+~cT=bz_}?(R2d&&-^2W_I>FGpk3Rk5&NI^TuYz z00aU6rr;mo2n~1{!To&zz`_EM1^|E-poCBZP!K@?3h>1KHwH66PXC8I%^#rrqXPhX z;1vKUo>BZGf$l$p{(~Vuj&B~J=}!7ZMuuuBC0yS)0$SaD1 zxx>ADw5-n=|D6~7r7QZkkufnb@-fQtA>qCXXEZf66%>^el$7K^4Y`Qe;7E74TyTWg zKP8;=j_?fk4~_H>2^Kjn(cL2?DpFTeQ~|6(;qNMr;lH-(iTqFcp9uU<1pX%i{}X}# ziNOCv;Qzh|{0H0d4hAVs49I1Gqb)$g$}c1`B*HHwR76=`5zshmWT|%Jv8(h1uFmq1xJQko17JK zaC8!(`3e#^0e~4`1gP9SBSH)glu4eU;gJA9eGZh1#YBc4V?PjchJyxzIQ|&>{)O)zV~@Y^^FKP5 ztqnn)R*M$L^aaad{sLmfU@!k*0AS$+v7V=&yBCNrgIG2wDC8I? zg7`G39}41^AlC5sPi6Z4C+zO-@o)L=?!N!xzx)Dof<~P4j|jcy9{cCV|CgtrYf)hA z{-jgjnb9xY)Ec~kK0!VQhL|5?Mi6fXdt9^xF*}F}{-Ae%VG4S`C{tSygZdN#o)Ly% zTmdc+D|)$~GY7E%h>e4{z8{RBhMb!b(|L)dhO410_s!JdS5d+mWzUzFCyCJ@4S>ck>Qt*<)EFE7eXVg zj$;VgNqIHE{k#>3L3=2pfeU~ka1IawqQJug2mt~B{|}ojo8Yg%?id5^KsewFcmr~O zOa94W|L0CHc(wuJfGr>x)QR{v-`PKRy@6;@PX1T=S6(^5=g-~fKRE({HL!*tz#O;+ z{*D5%H<;(2+U$TYpv3>*-~ZI^3EFxMJdW@EuQCJBs|#SQ$68VUlq#lfwT?BrJdxu$I3f zpl_ia&=x=hih{O6-$6T%@!$0u|7ki0q5a|t%1p6&u7!n#A?(gdtDPjP=1-wPff<5JBMU)hk&H%vi zJLOmh0Cu_lyelAFBmW@_>jnVL3~+2Z9?nIMwE)0f3;^2J0DuGkA2P8W@bS0{0G%H_ zqr#*A(xW`SP=TWX3;0Cw0z!Zoa2oVT2~Y#H0e!#-I1g9?mjOrM3fSxX!2TQ#!~pTY zO&}G>1nvU)KoL+3lmRb+YM>ry0Xl#lU=SDuJ_4VBd9eR{1vUUIfCq>W2!sa01mS@2 zLWCg_5E+OPL<6D+F@juxSVJ5kS0TQTU`P}s9+C{ngycesASIAWNFAgN(gS%58Hdb5 z(2(zt9msDAC?0 zC5<~xBuy&KLz>q#D4GeHuQUW&23lcSC0b)zC)!}zB-#SnSF~NU6SV8JL^@VF2|6u0 zOFA#QSh^g#=X7uA#^}D%5$W0JCFynPFVhFm-=u#)Uqe4Yzd*msz`!8NpviEF!Ji?K z;Q>P}!!QGy;TI!2qco!-qYGmcV>V+2V>jb0<1P~u(CMTvyrfjB4raq=6CIT}D zvplmIvllah`4Mw7^91uY3muCDivi0OmN=FImIjtFmQ7YV)>Ev8tnRD{tdCgRSZ7%G z**MtFuvxMNv1PHnVtdQ>jh%-56uS|-7yB*tGWI_9RSqbJ7>6N;Cr2{JbB+OyubkAJ zlALCo0i2nf)tnzVu_xG1sGP7n0YCBRMCXYWE-2S2E;FucTsd6zT+>{?xdpimxV^d4 zxofy5xDR;vdGvX_c`|sAJfC=e^NR2q@donV<89$xk0b{ z-xo#+{}4GLVjyx&2CreI#I7t+h6txvi5=DwGiqVT{i1~=!7wZ+n ziVKTdh{uapiO)&UNoYv;Nfb&9NgSM#JY|0>?NrODZ<0KcW|DD|Rgw#*nNGt_hn+4x zJtajgr6Cn4^+f8UG=;RPw4d}N=`k4!88sPynI|&ivQ)B~vca;YvNLjwat3nIa#eC@ zc`o@2@=5Y7^4kjH3Qh|56owQI6;&0lDLz-6Kf`|J{F$UPZ_ex}Nh^6OJyM!dW>Pj* zMku!`<5Xl+yj7m6d{*U9wNOn{?NueKsjG#n)u??}7gxWk{#boRgHyvwBTHjglS=cf zCPK4A^Ou&IR-{(F)=zCY?Lh5U+TU~}b-Z<+>#XXE>bmQe=&tCA=v~!&rndq+33G>) z!Z7+``d<3w`s)T#1_1`I4YtoJo((_SboRhd%kYL__c`b}lXDs8J{YkZ*%}oYEg6d% z`x;jnV@*^|uA6k5Qkj~Y-ZPys6EyQQt2D!&S3iH_e7`xPxs7>|Ip%`&h42e+EGR6_ zTjW_RT$H#Ne6ht6ur#yGvs|>2v>e8c2Yu0D1(j;&5KPIgY^PW#SA&iT$?T~u9C zU8Y?nUE!{8ukc;DcBRve#m(KV;VRWt`>U_6lH4ubOWpT9%sd`@U_A{z3q3cy^u6wT zefQS&&h=jR(e=so`R1$Zo9Fx859U|kx9M-_U*wMqFbyaUI0&>1tOz{1W`7MCL>qK9 zs4bX1I54-6WA(Y})*88*5+oNgvlgh0s{K)SwQjoJyuPK zq&c=3+v3-<((2ec-gdsN_l@?Orgr)EijI>Vk2+6u=5*0_-Re3*#i91QL%TP7e0x@V z-FoNy9QvmEFZF*IFdrBmG#>05G8jS)>kfCk)q2}DqA}7ksy5pEPW4^Wd)4<%AJjfH zkExHfe$@Qe4# z_m4lEe|+5t-5_qJZn14W-aftE_*4JqJFEltD=r*&w3D^VyIZlRve&zBwZDW9!V?cN z2)u-szchXg{kH$TPK+VZkRFhw$ZrnK4;PMtj*ddz!`=UI0VsH;@Bud{b}9gX$r0oZ zAHivY^IvnmKN^s~=BXft{CymY{|o+WUU+;1FiC0C)->5+G&-ug6sV zvOYketo-i*N{&;6kFwGq0AUOVsR`leXjc#bXx;z-IsE8|_~ht_{2ZJs%mP4n*k5za zV{K|y0C>iGjNZ+F@V}(L=AoeQA0Pkg74%$LS>-s_|Lc0x4Y1Kden9u35K(}F4FY9@ z9Q6ajpwCnwp#jrB^*|_~lvLC-v~=_gph6QXKmmb5DJY>-RL2VokQ?BAfRc@h{p1+~ zYK}|pG@@agN;mJ7(2AX{>p5XPjult-2v4G;=i=t!<&!uid0I+ZMO95*LsQG}oRP7K zso8m(%eHp*AeHm<^7ird^ACuKjEatd$Hpb!N=Z$-ot}}KcR#9Hl#=GyF9<~pctF`GsZO4uW;eJ*;~vH#s&tc<^X$Enx*j?) zWozsSkMMDNE^!sK1n$_iKc4;f983D&^6X#7{==_%ka$!4Nl*%~m7q|tr6|EfMMHI* zXlOw?1}56S6aAkA(kBK6kQx1*j=(}7U>RU$YVa=;9Sz+-xBmOrqeXD9PXA~QV1$A@ zer!-S00xjrr5UGz|BDak(Z~NYF?bm)FVdJ~URBA?<8GX5CVuth?K0VSLx}*dIF`!7 zE%^w*+Oy2r3^WAJv23Pc2Qd!YQG~6UctgCsl-zjd-2v|kcG_N3lSk5fq2r8O3$!$I zSI!&(Nks`CiWD-wVqtxwZXEHpGa~Z!&IIA}T8{(9_xq9JQqqVPNWSjI{aR4BI$>;D zNF=RK_(X^DDl#>uK5?MKqU1rTtt612H{jfoRReWHjHf)==awBV6mspJ+^)Hm zoZKU5Fsz|EZof-MR3KfzEt5D_@e`k5zASt{d=QiYb6>RVN>5Q;orhPEpH!Wr>zB1Q z%_4~NWXgEo<3;$vR!W80;~lXLcoW=MW;_IGhCTw=o#YFe3++B+M%FF-h%T6nb-pR` z;Dvje@z)=;B76Wb7$t!SLd^ff-y|38&)?A+TG@J?Al&*!Elwe0-{QieV}QJ4Vc{ePwe=O(8T6Dx? z-@IkrjS<93({*9I8#b#dbu33Vx^_LeJis^kn;&$~67NgM!N4+?fer?o_CW2N!E1P- zam$qaYMn{;tXHArL7>?1$eHA%p^Yo+Zl@9kwNi`A4hN3_dR0NYAN(?=I%4G=4GX4q zMtr{v3!+b6zMki1LFJ{~hxe}nxkDXhouy)Hg1sEv@6g!AgrCsgm*j-RWqkF)%_PZnsMBQC z=or2*dgdx>M*ZrbRxYbjWJJk@d*^}TH$&FzcH}$c>`GtevT_%r82|80S>^5xS!*%I zR1<64``Nby_<$DM5ajE|QK@~o^B5U(bBCw?+mcb)r%SWAVq!c_*j{Eg6M?*~vLzh> zM?lZH1XThNpGkhM10&Lq#882{Y8GclCf$mZdgPlJ5>L6GF)VDH+#1gR7S#uQBel@;(!Pm#n;2*IuPsCPjKPfF*1{Ptc5u4}wq zj+wBxF1_w(D#!TtaTQWG+dw4Inc#{`M&}Hon1#7W(m27K^vL@U=HG^SeeJ01m3i>O zm>y@!_9;WF3NRQ_{eI$?!Vy3@NYKVX@o_lIp^#)f`N@6CiAFb8|4F|$O#{#GItAQ) zFF3-xaxbl5^J@0u^UFS;U8IoH3{~yU{Qe;d<=IP1ClqAsYfZ|gcc(P3Z3SQKYW%oV3X#3?R!@kh4ig3necEGb^W^MLP(feSf~dkhRx^%E=fuAIX?H! zRu!k%gouP|xASiZUi=_vXwjOnJ&mTkI|hAy&-BiC{#5?e>#SjJ83?R13!g<0Ho3ct zwzk7f#u~?Y5c6?JWWnN29oy?ffvxvF*VvNub@iVybt3TZ$(=ej*uYEVibGD4GB))Q zc4BmkJ9@fka&c*+Ma#9}{`Ki>hD<+F{@lBs&>q`|F(S;%zK4CqP4UG|>%9y)LO=Le zKdJ^v+T@u`1)-x0Cd^hdq;55D_eBqy6`t;|R}{%r9GZanB%=e}X6Mmqsj|V>%ClkA ze$hvOucH`3@kWy1uM1D++vJ8cHIaTFb=1*Fn(yXi1UK`#ag|~7;pS#8+??~th_=rr z?5`*+#aHv*K4Th@xixusnxyop=~9V*g8WCMs?xmCldcy*ay=oIQeFzh*X&~^^Yp1s z+!W5#kyF!o;%j-M|MYwm+gr~>YW_-Hne#`0^<-nK(O^Xs)AI{G&@s{e7ulvj9aJ1T z0JxSm$Tn3ATi@-gaGQ5Et?GErA)dvp@s{3f@cQJwtK0dHjlcGT!tu)&>zb{1T^2{~ z8nWkLVg*tzZCe0-mmF#dd3`N>j0w^$pJZQ*`?NT>?u)nHdA+pQ?Vaz-^!34KRzvDJ z6U|pb{J62Cc<=QbT6psL8NK&6)$u`_>Irsmeo|pj+rcyD^1ELs^xy8icrAz+(Br|r zsF)|&RDJfF?$jZlU6XM*0)DnR+V4MO=~u{c!(LXH%R$@cyw>9nI0C3ukiZ7_+MGk# zOvsn%tmcLh5B7e%&%`Ol;f=fRAe2Hz39Er`*7DGIST5mCb%qzj`Q-WQ?r*(&ILp*j z&nt1SmabPcHbM) zGm#dO{d1>E(4BF!3X$)frmyqQ`zzO954?Y7l>cgS(|E>Xq0~gZNb+OUHI^h;6AZuU zM%N_Mo|(hkaTY)K`_tB`oeFu;hfpt_;7e&V;VU7w!(&q?a7Vxv{YK5id2YR)`#>{zJ8Gs}BYX?=E}(fUyxO`t(>Pg`IH0NcU2I z++Crcm)bHW7R|mf=GH$On@&{sdws{bpO5)wd2>U}54)nMq?M*m3%DL&aZ*WJdUyV5 z04gos5go!k+gyg$^|hSdVry3U9W`YiM!nJ4abbj6+b2=|TWRZ}Cq|*~Crf}-*~X#d zc;oypX7=e_tCpl_U3j+HiJXZ^`%utB!5;_?9edx(Wq&8Lj0Z& zx2~io)R+mRILw$@Z-VrUTOc=ZUkmjhWZtd!%;5m4R!w=hSodw4FC(n8H5=Z>>yiy|2a?_a{oE zq@P~!<<+^G%@n^*G)b5*ra^flS-&p{5lVb9f^$!DmknaezoY!?=_ky%3nqEl&OIA6 zGsxwv+?M&V#qBt$V^rn4os46*{^YtPsDIQOyUKFV6I+&hJxxsh!x)n{$;n)$-a?p*Oa~?7Q zNmHK;g3UakxqitkEVevWEzE>jNj%^CM%HX^6164o`y1_7xQXiVjM$mYb!&s>!mh5 z>{VdgElHbU)-6dkzzs;T82*!V z4Omjl_;+ON+J*E}^4?}_IxlBm^jJIljK4kc%blnM5~F%5nwGY8{w zzj!(kR{G;~O^6Gw@Rp9UirS(0jLPCE=i+^-8!=1`W``^HrdrVYf)XbLl#RmJEkPzR zTr&PK)y|HWlC6g_Qf0KN3gk-)3O?COw3j<_M|#*eqhvw46mu$3vb43D8)M&xWOoH$ zw?^@c-FqZ9d1CG(#E8@C_@Fr&|35P$MSg9^3{1i^4;g5@ z$*2O;LotHSBG;vR^&cI@8L}Tad#ujr?UPMsc29x~*@_T?8M}D|G%ckQ&EU~edIK2E z$WD)mpYA(B`<=9gyovRzRJlvMsitymxPyxTto*mkXaYmLD>i4yEmezV07kh`J@~L@ zPMip(zV1K7;cT9r+aux?_CrZSq}ciy%h%OI24#2O7YWmR%oXMLPAncI?afyu6(hzn z@Q30NKfm@TBHGKf!Kx~0Tu7{67^`o5j8%c$E`tfXl%C3gph}W^~E?n z?TmT(!*17-t0Fewj$l)txaVl9mq7~8Ft6&S8`d#My3dfD+rd)me<_yB32`TK)Y*s(2{o2$+i*{9Otcwj2+V>kwWUV|gccHDhEp{^d4OaKgxc zo0D6c``ujM!K>DEJ^Se{Zg6O)LG+mG3Bx_nHC6CA=7vhY)RR%CzGvqlT^{jQ`Oi|w zmeAfUk@@85!5|-afWQlh91gqD4r?9(88#P%?|YzxP%CAOWry^xSifAwTDZ2}GEV1> z&PSS!<*jPsdFR7XVTdZ?_Q(E0zOPg1&;7V|7zcm9n9w$!V9Xk#H{R7a6nPZR0CmfA zs|Vi)2v3FdL&m(-n5`b1wmz<+znG_SV$2%}zvbL~&#zu3?5s#CJi~OP9hc&epD%WK z&iwieiEGqsDp-#KF6@vT)=-c0OI?B76~c_H^bdVxkxRYdK))CAnFcMnS%E|0ZL#+0 z$U|O&vi*$8hVS_P+wDDV-t7-19s|4;_ey2QWD{KGALNdTNEQL4pX7oBX}{uzG!mzL zNb)jeheNzM7BR^8F3yF`Fcx1W1#7IHaP}9wcHjT?w(C`;oG0UgX6rexU@Nu44LK0pu%NlhYKmDGzfUYKj77g3*B5VpSbnC5Usho=;( z;1hU>yaMxwd|!Eu|eN47*fcZ~qA9M9Z!e24*I6+45G%+_%SXM=IS1B9fp`K3=BoDF3Qrj=cS z&)uoXk55+8_=RD1H5JLKN+HpMA<=+mC{QL$3aJr%F_D2xp7tNto3nXS#kt>Dj5JQK zw}$cN-`>mmZACLv++0E8uvyv8#HKkAkVMVz+(Jw^L{fFJqeJ9I`$Lbe=6W|JTL#&h zk7EJ>Ug>CQ$HG8rS>78ghjb{S*#@j>rW?i;FHNXJgvDTB%+5?g2i8f(N%&pUPa?Ue ze$@l#?)%Jw)HU0e$F@_xY3=vd4{Z z@@HC2SOjT!4z;5>ZqhP@=Hj6T@h zfB7DPd}4lHPoNFQnDiauvE)`Zc!faJ5qk&7W)-) zsm!y)$I@-7r@FfrTlYP$NPyiSg#0jpHJw;EP4EyX^0k*2py(55bXux|+^85s|UfzQJE6}0P+h8GL==1A&uYP=C zai{78ZiEn}4ttgx6mCW$QkMlt+UTm=@mGtup1hLyQuO3YUG?IA#;=K3rZbtvs@NCN zSy&Ao!aDvgp<)#_55v#alW)h%Ag_FbF-2al3?f~grZiG3lYhjq;1-*6kmWZbS9<4W z&Ij^_45=OA4$_7LM?EM>J?e}4tw>RA@KZ$OdLQ0*Mx1cGcol>*ez=+xw5m9o3Dtz#N(dRB(Z@7mWLYU2|cLx&S5<^MUP&*<1$A)+dA-3TyG6E^3T$8r6?tCHS13${%9CzFXjCPCkg(=k{`w#@Few-r$Tqk5&T2#!SjYQH9@2JziEZWC;7VFcf zl6lxUxjYj!#Xr*QUIznaH%V1I6q4@uMWx`1pmR`PS4i1g(glysszU}Aq9)vWxkn72 z+c32shGDo9Xfd(9d8%FMyU${^SGp5_v~b(~SXwjVx!Lf`b;COdf|xA!Esluk8nW9} zv5oBc@RbPv5|?eV?iLL8Fb`AVdiJC^Le+WbA$j2=&1i9b$Y#iD@jP_J{${)x(U>q} z-9+#&{eA>o#TMRDp)-0}AS3lW1*!1V=-!3wAL!8Xvk#{8jNxYMZIrSY?-&9pz+uro>qe#9%w(BjV@JW&SFNfe6i4+tqr@rQeH}H(T7~;h=$c)=QDQfh(}j4R|FiVv84x5uUq_Z&fd5*Ku8))WKK2 zTKsU=;X7^w+t02cB(ZpE;v3C2=FUCOb0T?FV8^bR__>YnMNt3v7_8QOI0_2&gOW(|)4n_a#XA!i`g3S$9ti+?qH$vij2F6>`j~ zdr#ye&C1Z$7PzGVBR_xzu+aE{v`U8?pvl^fF2_)bg3OrQRUwnZoUJ3W+-B--S}=B$&bW zd%i)j$U|CR3e};(N4a`lmJHAIK15j~_Sml2fBy^&;EbZGmach;Y z^(y^eEl^_Cr^Akb7+@=$No#=pGsog85mIBVayDmEeq-b|c)_q!AOI=d&Uq z(7Y`?q;JqY3q9I`jpFn=%fk-RNDTgNy^1bp43t-J{n00F8un1FOrkzz2a~SX1 z`IS=0_$)_?T!30BhBf&1yUlv`E{PHzq1SLM86DTl%G)a8sn^?%fX3Wa8yh7auU%S0 z`Zw8^)4b(GuHCx=4)s)C9}fisQ1wF9F9|!bTgb>K)+4KZIoDoI=+zK0rVq{l{aBqP zUrk%r73I@5=;8*G{^j}Q6}KKa4m5QoQD+hUX7xfX1A5UzH+p0a5~%g78F!`pJFug^ zHTIBgu2kt&cN;)};du!su|orh#)ARjS>@HVwCZ?g-$0=<_rO=MN6aPOG3d-AK>S7l zMv^6j^V`^(^kcOtoBUzeq+1e+AB{|V%08jPurkj%8L3PCfuDiloKF~22R$2iL_v&( zK*JdyO<6PXnYFP^D3JqBTq(}daVDN6#kMy+!l>UF80oxoKT}1G&22$$akk)`kh6=q z7LtuCTg4jMwcE0de(TI|mM$+(@}?hjSNo9e3~uz0 zt73DA=H|Y6j=B>eB!YZL#IrC+&C0tr;@hi7TT^icE-YzzhkU3*`j9F(*c=liHlmgT zlim!WB(Gnop;A1OxlkyKQm8rX_srLrUWLpn$7D1(Ve$w#p~H{hoT)Xa*x(btq&PAb zCJ+=+vZNKZGrVNCN;-3KoifXQ8{}PXD+BAjh}lPGGx4T1OHzb%Z9941^^TKESXSVaW-n$xl?;QF9922bMVH{^ph#zQc;j0B(gUg9x3S z2VP9;oM6Wr4SHd=c<;1ltMH2>Ao%Cp=pnE03I^V8R-1Vlr`8rrE-aZ_*d{7{@bnkL zguZYrP2rZ{Oxhf2vodG7r$>?S){PYvn3LwDFo>Gx#`tEa5<|G=Ybq-UhUYsE-;JYd z+kQP$Hln!YQ=r7pChL9qrLWHiK*#ftP8g>!sK-GV!^z;Sr=xs7?FrH|5Xq`f&*GdH zzQ^EZE%8?!lZ+xI*-!Agl%q$_1g{@5K!_RyYqvDz!wIh6Pivlvv`jU=*SgLt|E|b+ z-t-P26+B9;G@y?W0}`%PinHzhByqs{=G1S2yRgeAIwa8F^hRI!y+LH`ZP2QBJhiPZ zZwTZx>oWEYdigBaM>Z_iYz+mnx|N!Kmd}&6l&f0!Y#P*03m_rt@PlxjW&h9lWW47@ zhtcXX{}`Gq+1tG1hhUEUhGBOIYHp40Z12dCD1Lvxk^bAr%1|!7XjwwyBBDp9QBWDL z&_EDgwdC1QFUq+kw_oiuV>J=XH!{D~5-8q#ss*8F9*_l`{w!-=d)f^;4O;MEz;aU+ z+g(Gu6P5k6%)V=E!_A#^_GCN+W9ONua(&0_2&iGv7fwN74K>(m^i&q^?TW%(A^3<_ zc}KvrjDwJQSU0ROXXxkLIvN}n%dvP=7*!OJ1pitHMW z2aLg4)bTUmc6kq@=@F3mS$T|PohD3BN7|8J!LC`}IRaQw0j@B-1o?p~PrMvL^-Ekr z9mdzF;LC-pMBCxyPNnX1h72@7LT+;WQYNjOV*eRr6At z`*d$q4)}uJOQ}i8rCh$K>5FZ%5G3qL4-Rg^HrBxvu*vF0Ai-cmn0~uU z6%2rVhZ}Z6m@OcIE7@Tt>VDT$Wb6g*-Vt3bGkKBpA?OxcWlAQsRgIY))Sn2&4}uTB zOS}-)vYYIN@!dS2eo+E{)WdbBYLsKyGP)?`#e-dWX7BKi3nqLIJ^eB&r5b!3lz!3l zvi!2p1BWov$f!BWD|I&EyCTeFm=t_*F(#1QJbq)L8 zm*om&-{V*|!Tn3A9NZW=h2(q$lzJnrLBmfHXdEX=ayWI2>mCc&lWN!RaTkllD5{jd zk4DJ7b-|95uSpEAhPqGj?Vx7gA5Vu?vrV*yG82E__8ummhVFSD(wh-o0uc4fJsakh zl;=<4hVDodjT*O}u4mwpOHy528lrM6b7T{vWb4xJXMJJWzf5_^3q6 zA7diSW9&0GCokcst1ICzx@LW2qHn%q-E8B(vd}yo!?^v!>{n>!FqkQH?}-t(<5}zy zG}s#ib8Nh*XZw^l$aUvBmm@c{Yw}y*-q;FvH|q1js$`EF73uth9tWK3hhU6xV{3j| z+TPCFNHMwyj(aS>hb!z&)_+PpV;MyeBq|1joV(qcSTc?PZ?bZKF1!p)-YXjU#=+Y$ ziqJO8Uy>PhA9)@V{c`ii1IF{}*xgA)k4=KaLV#P7eeV%)6UE+?xy6s>Fev*`v}c#A z5*Ws`R;HpJqjZfDl}E{w#k!^>c;CAg;TO+K)FeR&QOE%d5)!Y4MWr=8U^m5}CRH9e zJ3pAPl}1Zst9mm&uk7Ka%)T`<>elGn^K-Tkzk);E23MAfws-n4>V3}U=BO?H)1<0L z8mfNNUbg<615r>YZ-J6hO6s(+hadE-@Z7;6T{ZR*hNWjwatKxzk|sOfZ&yFh@7H+8 z_R5gsq_;M2|BO8~RQ6mt*Cpu!D|BY_dsO3_6UqKgr;5wfowXfAbk-l-RdDM?6}sJC zW`YF@6C~WSRS@8!&_J0T)~7DsT1v#9+kEQz?chLH6U!J;siz|6>l*~y&W!LnhwKet zkweia&IDe#ImR}>-N^t`&0#t1*%HxK;!KPz9+iYjsBcNOaYB~HjW~fzMx2nNiT^_@ z%x4Ha*ckNBAp|5qnCVc`->k~7D0SLzl;>-z->+=Agf*Y0r{q^}T{>&}0-NtsRy3G2 z@I$Dp$a!wOInEYeMHGp@IKJ=3_(PeuCLV@q%;hW;5Xse8E#Q@)r+A17^rz+NzCwe9 zH=uu#I1g64QV;2HIq7;rB&m;4v~$Zg9oW{cwofl^4ho9xUa=}kBkexr8Z-!Jmf1JX z0DhU^ZOAzZoQH4qIE#(2vewHRPDO)JcjES>I=bMyDUbw*t7pwj3vWKIiyjpk!j}VI>{v?r;BtfpyqmI|jL;DWiu<4oWN_*!2Jp}FF=&vB&Zn7-K^Ha!A?o)~W2lgAB;n0=?Tmca^E>-vbo1p8dBwcR)0v*d#8LPw4VJ4 z(5{ocXc=&r`0^P`UUdQ^NuPiwY6qZbQP(`51!y(bdnK){_g;w%6~Clp78u6iYE18H zBKlL@SZo;TZ*4$o!Ln?!5N{;BeX8|Im1Q<<$ceh_SwJwCDe|l0i>cF=1tGE(sit;* zJs4AOS%`=tqa7;2?-|Sclp70LQJhWvK6N?3xzMT#dBa2hHey22-R5&lZM^IOV ziL0T_IwKp~&a(U7FCDZsopa!V3cm*TVZI)0nh{|M^QeaNer^mM5l8}!1Hs;1j#S_0 zEk~LgK+f6u=gSG1oBK7ZuBL^_Z_4N`h05$xgCA5PpCJ50W)bL+Et{1~E5+${^;L7j zzLToXqW6CAMXp1Ncm-XfhXF@1Z?8~lA%+|q>WW=NbY`15U@<0xnbduq&(mKz6)mU;d)*6*fyCh7g!>RWcGmsR* z+ZD>Nx*d}Ng=~4d#(5W@eQofiLPMlzSxa9=pYaM1e!ARbN#Y1`!{rPNqx=uT3D;LF z83LpW<_@b%Zq(c`w`!yeG0JL8Pd`&=6c|dXyqaZ2eu6E=N7a)b=mBmdMeiY%PY!p$7ILpG(Ci#YpLFP!VNc-|ll-A|8v2WXD<*Lu@rSteAvn1ICl>;K zB(;ir&6l5A86?Z?l;J!-T7G+=UR=`LduQCk$~H5Ycfp}fa-Rjml}Wr7xZJRP?-?9s zjpcfTv)ljLjN>Xf*~NEGEnKcQUFGMZ`RP{P#5IQz@ZaHW&wRdbMYA%t*+K*o7QtQ_ zlf8~hOVqi9U4lnBc+`D*{Zl?l`*uZ>qutrFcl5Z}%b5q+6|9*JRHqL=qw0jW#)+=* z4>CEd@I3TlVG2n*_GJ#!=YgzcRnb)KPDzLsQ&5N>1`d%Xz9N(CbM{+$nPTQ?-_gwP($K zrSF8yBfw>n^23SW2G8NxCBP>Y!wQ&v)SP>ja`BVY z?-(N>!F+*Wa!5afS%$Rqx(3=2B1(rMg2=_Ot32Lsf}G5KTywQ<*`@nL4lgy2H=9ad z>I&PB*+dY9;845)R&^#VUO7y@u4Y4l^Y>C)7~g>0!re-?A%OtXj3v1bo;eY$u9?al zLo=7HXx7w8oXU9GPe(u?j9@|v!EQLzk9P#3TZQ`N!LPX{e!qRbdx_7_+CJ4(Tz>o> zKlLtcP%EQ`CG+8X1oje3W|vR{A%tie?=l&$y8Fm3~SFFhv6uXKEZduip!BoYoG&Xf~G)(x8L#9+^$g-mk4ks84 zg40E1uU#T7KMD7=eiisFXC8sKB{zZN@}mMQbvnip@|mPM8B!gxH;k0%YW@^lbFyit z$b2p114ijw?uAPp(ht1F!NH`u(TZlha8nKo!`G3Y=$uaAAh}HI@V$vgI)@+RE?4|Z~enlf2hht35>Yb@nj`(2Q$!=;e z%;Se-$()dF?|O&@bG6`MILTYpO-QbdxU^eM1$PoA%Ut~nTXMxe^Y`1#_Q)!kw4Zb= zQWK-PXVwi6s$9v9;N~%^ULjp&bZidUTLA9Krzq4Ibri}3`tqGe#TLvFFNo4^t5Hh{ zd@~0UF475icCq1%@2%$0ed|)el2Tn2+$O74&82v*b~bIAt6b2Rq%2FGn)%{8#KKOp z1E*Pczx{UWSx#P}{6vz()~{b`o{B;W!JEf6uf(gpK40@N;N6w05&5CvTem`_V$P7m zv1uuL#ZZFfIjmjc9j-z&S!>d~-dOu#(CbnM4@z7v8*nO7OZqfO=T5<{p$L-T?0id@ zc!2~XRF$bd781VeJWKd0aeqLkpeVld9Q?ABlDQ7!^snLBdgV#+gt|UPJM4?jH?o(` zm9Vi=+1m0P|MwW7K;`LYEZ_n_2y9kp72iQH_eEPU*tU#!m^=%B+Drs1oOu5{%hT^g zQUs6hxqA+RKVIA3OR%Ox-s@r|QGiZ9BY2|Ur*-poM1U^=dbs(;EQdafhR!HCdh+6z zShaJ;8PQVnymw7R8L9H4YhXSggIeDOr|wt_aPd`rg=P!#iZG9l#?>SX-r2N|E;u|n zQ?9(E`eb`LZ_93Q`sxAwiiS61)(@V0xlMaCTfRi8gk@M`VcN9>&IQ6Gnmm-O0)Dc6 zquIN2GD_F!aQeO2)3qIqpy1Lh-BAlsBL-=}UL>D~oD4=y0p(#iOT;gdi)_vkTvwkU zy0wOS+v!5S7Vt}jtWC=$Of+1m^xJWAHmXAkXs5@xJK}e5x)~o%KV>Js(uqaW#`6*vmrTFjml0-NB4jPfO@%IqhxOs9TT&zKq$gwVh)e2>a!+VReE;2wW$!+zReU+f zajp^Ci2?}=!cA$3P3DL>8jM0(quFM4fg20lvd>HHU*o%^lKiv3dU%s7 zt$5v#WN@P66EYvTBKEC|gK%e6n8kmYNkyuOKMy_6Q)tzRD%f|XpE*VQj!pdidZnXd zrnjS7%joa7po0Q%&~tKi8M%@aLbN4G;1tqSn8IP~ZxQHG?e7x#stmlVXWq&4ihAGH zx!`%GK6f_cp<>O_!_~3bV*Dl(8C$f@2lU%@MY((G=#}5m5;Ly^KVgK z7E#)^-nnmll2qvJlF&J>FRU2|W6}x2=B>yT_9@H?P4>O9t1K}L@_#;r@eG}d>lLS7 zGb{ScdnFbUBWtiI!xdbP5+R)c-&-4GR0k9XawLB_?!jW_uejI0^YN7tasx}H_yhJI zXAQ!0&#Lsh$s6Slev)aq^TL!8VW46~11>*+a}u8@@+*{Yr#iY~=$Gv44!gVaZswJQ zDA(%Os;y;S{y|({VsnCH>Kntac3d;ItsVXK2z<+dn;uCI5=IK+(o*$k;bXl;X(>O~ z{W~%@QgcURc{s>LYxD0~?!Sce%Nkw{R%kLM{{UZ<^yH`Dazsc*oMe9q@od7>6+MI? z$tdH4Gn4e>#l@f!zk(TtX7A=(;-V%|BL!zfdL;6fnRS#`EjO!(SH8P7pQh zyLI;m7nf7~8{?LU$UE05)oOcG?9Z44i=9#K&2A4qantsy0S|II<1`e9!c=S)68#e9 ziq_Lq#^$cuVbgAQ(4jYwO z;Ax7DNfY6Eb1}s|sN6xHPE^KOqkzXoGOvl1vH1K_Y2yFG-kZlm`S<(dBc$wP$vT7- zS+XSSC?TmxA;c(ql4P&RMA^3xN=2a*G4>_PWZ%g;A;y?W_8BpTS$yB!=l-1YIiEV` zasSSJfA8P9zmLx!(=}aY=DM!e^}1f~*YbQlU#!4f{dmE)hyFQ{{{PJmvW3Q?Tm4OX zd_yQZi=j1pt?{J<+ZmCIIS;b}EjhL@28Iy6bN}-fa(nn+OGKy$;17toT^x0$Jx5lo z6nQr^6T904tk52hzpg4=u~SmqyOB~V-8mi`E_9TCLk6*j!m{SoamQ4 z=c<359RoS=4}5Kp{EJC34C%NHS=4<8MCH&$FeFWoqA$bsQ+IjO_ubX2`SXK+isxTbDe^yJZLbC#*-E`%pj-IKL`Nl0 zUi#(%m&=P6!sJH&n^h7ncQva@a4{u&KJeC&0-|X%sj6~Mx2G>tN^#Ju@E9%SiII!k zcUmELtPK$${NY%jh;Bm$YzVV-CXH#$l9Fs{OGu~3hNZ(`WQR;$20CNnO@JUPquzKX4XUcYG|=b`bxtIT72C( z9GdwXx*Q#&1JAd0f2a41h;1vg$HIHkNCYE+mg~XNk{8@UQ#z-8E%lCse8?)+Xos;W zhJ$R(7IWIm%LnQ{tS^Y92)xOjKg>N~KX!JH-%Y9fWQOr+FUMo3K9DrgZb37Pw1%i) zV4jJwQvxI91#t2CDcONFnd<1ZUN^gji^=vOUw7S;H?okSh}FH{Ym@65fA-VekS)Yf zate+sR&P%wu{~7rRl|Ui!-a}>8G|!DXC-e&oE-dm^x4c6m1O1v_*7|YfB3${i6PQa zbN-M+K{puQ@4ywa(1@l#tCc?V(g0iv*?f!M4qLNA04p8Ju?=ZC_va^n|M2}6QnIc_ zD1+i#JKiYPcSb2ZM5=^%?#ZcwZASJSP2!Qj*4Bp$2wr?Og6bQ;T^zeA-HUy#o_D=?=Suce2=_at1CH&&A zP(l$MeDp&}4yb7{qqiX^3AG<76n_ryI72N8{|Nux-c-C*y6%dp0Cp!}zC@TPVp+bQ5n3@WK9c%p5vD zws{+}C7_0+LaMeQc>j+B#$}0)wMSQeIwUwY>k2@P(Ab1wbO=BT5Z^7j=Zv)%+dmEb z>6pJ3$hk%QN5wwvKOWTY=PRxiK0xwW6m%2&`Jk{rUa@^V-|eHMKgIm-diT#4Gh%|` zNS126g?%BCX{Z$>$gc?baczM`Jvax)dM zWi<8zgzNzFZ+b}JcYrE?`SUUP(;EH1ZH+MCSz%hOs3?I#uPwd+92?NscjLx}Lr9!SJM&f4QXE9DvjmSCqVo?3k^zpEl5s6QuM?&;on;<{KC2e% z5XHP2m5r`=rrMk-I)X4PF89iL6y6`71nADBzwMd`piFG2nGNZm*0k@cqaZJ z-(dqV!ZtYyy#WH!&8gdv&WiSJh!Zj#crSQR5kxMcZm5{cqGS|n@@EGbb@&x|2&+91Jeo1m)lzEPacDDdP&fWgF3?tX9Zi2EXXx{? zPUeP}Z?V!oQ`nPE&Nn4vj$UEUz3yke3-Cg*EMXS06l;VyE#J?}=_2{eoj?r|M~l@~ zcm=J@_S11U_761{Hm6SUkRG(BO58QPBLi2b_hywQHo?ZsPK=hr!jEj^XV7M*;Fqpy zKs~CF2TwI~J4P{`+v#=hddL-4}ahwfA#t$HWC#LAngE@ZQPvoabm{krD%iw zC@OD%==GXr&5>1?Zx?+U+4*MUxVavpr<1Z+mvnYh4%i0u*;#@kWv zXXcQ?`##utF(qcn{olF$P<>0cdgeXU<^FrZa5h>~Sn z;brK=^+L}R#Wf>py#luv&Kz69Yuq^Fs?*WqyYyXfc}e03YjU1M(xs0z!{M@8R_`zt z*(m{+gDJuE&LgsWs0QWUTX+{Fj2=^YbvZEyLZ@jYJ6&{ipKUiQR5&*)etd25vQ7t(iKDc95{&R=vNxT zb)=zjhrVVGwNXLdiQvMI<+0oWQt>^v{7^3!KIK?m$P-Q)E1-S#viS(si4VtqkUwb7 zpaseB_wV@oqw8Gq$KBsqW-MnclUyQX)+U zE5E@Cf&0*nhTde|_fk_0vb4gAZ+eO$lMXw{T*MvgOLc57FU^(IzxvK+EhtEg9(!D^ zoi`~~eC^B5fCo0wG_Mi8J&41jb&}hVU7UG1ul-2QD7~HxkMd|Qd?JKFI6v`F9LL*7 zE)v|wAedB9%UkQR!UCrh$6qbe&)D4g`t*&=O(clj_cVx8Ee9sxWd9j9F7AtMGny`? zk4|2GaJNy$3@x%Gv2-I(te_SZ3Uh*vEwmxaQ1KxOWLNsA8QB1}uAql+792F=i}&48 z*!Vz2>M=jwWS_Cy3HGj2pRKX}WA_UzUdqUolvli43)p8O`}~|C|DeRL%!Xg@EUwp%_+h`M_V*715dalktQx_SlyR#!(|jctfYPSMxupLd~UwZs1E zWpT}J(AknRO46qnUz!Z@BTl@ydsI+*?sJ5$iaAHk+O15w-4lvhdLHj?3rO0#y%KdkpG#eEZy4$1fGg4;nkmH5L~L{iLVl+e*z@ zF#;*3F0+M`FDIXdiA;!;TUlmK8Cjd->>oxN8!<(kZ_|VPNM|s4(lV$AXHHi1^Ly&; zBl;KCH8|VaFMgP?J;~5v(_3e9=x!jJwByF!nY%E3z)P1n#Shr=gaU~y@3&k_T%Nb* z+k1=(-l}+AVQ-;r@qEfwk>pw#AY=JdD#FOh`~v2kRX|eCoE~(rP)(P$4|{g#t)!_Z zpy5l~D-j{VFO{IiaE@P}K_9|8@E>02FBZ%#96aeqmnJnTL=7M&qi6bQidVP9fg6kn zNnBGYBxE^&+qp3-@v~VDR%-K(3Xvnb;0{nta9m9DA-c619!&3CAI_)qydUUT(qpah zw=z7F|1#QV<-F!=!!A1s-K#OwtPkkvH%A-}U<#@7xc>X&q`bn(F2{Rj6?G=prIU^w z%2vNyd0~fL;=>(hA?jO(z*uL3hKjI7oij68><&TN7PYFo{cdma+Qs^a7Dd@SV4$>` zPdjpph|i{TB)*99I+sc}qsBZG{WrotKfp6;QI(R)2V%*0u4k{dVY7hf$JYs?+KV z+PL6+`>kd8gSf9z_;06BC1z7gZuDavOTG(Q2eNB-{}iYHwch{l`&X67va?%&9E645 zy|ZQ;k~`|ZQ1KZ~Fza6jdd5Xd;{f@3leY_B$a!S$bsERF>-_`52eKPP=YIPM7#n)J zQtwFhlV&7a1LvGeUd^b#UGG&p_syH5Nrf*_V$`6p1(!ZyaXaCUm=WJ~%A$|X$RKEyuxc&p(u$fM30WnrwDH;zOX4}f`IsB+ov0uqG@H$?oH^gc(D3J7U2<@ipWP`b0zJLtVN{mJ5 zYJ{)hy!_p5XU^D8rY)STtPQ^IFrY=)1QelnYcLS{7ZL zoM%K%&%ZPVziL|-7*IShW_Hs#+VO;}pZmG9(?&v}h1i5F{yGpOFz+E+B@u2<`~kYD|o9fC2#F zhR54AdAMxf4gT2-V-L5-cc+D10}-5$+pki`M(Jvx)4Q4W!XFj}+;*yy&onZ^f!}8T zNJXKXcj>&=gV0ey9Xo&J6qk|5J9h_Mf_Qz2 z4VK?0dL+ogNZE6=C;yacPIwNXyANaP;(v#Ix++C4l9o^9LWQ&!lRamu-UA9D$)IKR z!+G<7VTIN?Mw3VI+ScDP)6 zL`gF7tNaa?G!oI7+FGQ_7p#+9)8^UvguZ)#vQIpd{c%V}+e_N@f`0UMcw~lbv)RYM za|Dg%38Nt@?8W3yXA>H{@`#?`x$%bwuPEmi-j)%!%YAYSb)n(1 zzr0i%oe7+)+tPYqA=Qf9J@d@%7(pR^VQyR_f~xB`dSvxfsQtK6b8xKiyEjRpfdLXK za@TZnLqusWK-dXTn{4_7mS*`C0njl=uMN(Fja4w}y>Rzi$m6k(hKirIuzj{D!@&Cb z_@+hR9bMoUG-K3yO-WAoOv{4kBe|Zp;sm4eX1U}Kee2PRYnOnmY2S$ti5duhZ`2_E z?7Cau;-`08x0qhK4s3=}feN9;`AiB;m~bpM(PD3OB_`R=W!iYj@Vt$nRAu!sb!zMmg@5$Lhn0S_oword&-Hn zFg}2J)gkxdViR{GP7D;Tn`$PF9u$LAR|M?ko#>4xo~(atrpsw`?A@5}(Dt;6mO<~QOv13rW1 z@&yx2gBX2W{2ydRTuW0bb@*CiGCxaIhQ@I-mj@W_*&bz13#&IGOOwb)YxnD-(Jt|_ z`o&~H+>?GldTB$nPbjV}#IqxS_Ztb?k%|tq@ z6tT8FixI&y4G+h1R>V#oSomuG{wH?jWCO)bXLIP6y;44Jk+Ikk z_4ddR_HLrcv~eQUt$^Howlwy82Ue3+J*l{oOK5E>YU{@72>Ti=P^L7F2WILwbm; z=KxUbR-*o=CbB9)GHo@e2o_3b7T3JJsdIds`QE%&z7WU8cqYFGbNe$&zuIBR4eI-Z zV3Z^309lL(KSjn73YxKeezyb5?p>-1yzL=!`10lJ~F zUq~!Mc^i`KiBqC5%%Qi=hy|)|_D-*GAR7&22oaG=kvblP1`e6E38Uhem6OAnIEB-4 zmu+lZSO=NjU)j04Pd@h-jt~fs66@VjVQ8EnFz?`qABY67m@`X6%Xqm@& zd~GpvATDF-tMdM;lvKa#dsD9=elDd665B#Sj~cq-NQxH$+mgv#PtMC8Z)%VksZ-5^ z+sut>oj|J{Y1o$*EFI=^VDotucemej@5FBNy%gx;MOi2p!cp;5dGib)MQOTwUx_r@ zX>KCn_FNkCfzVIEuOVNgdf%sj#zhobbH}e7?e{BV$NR=%a7k5RO#N(nNoXhOs{h0ys??h=@V$^JS|IxBA@(rFt4 z{fw8-z5U7|eOZ~Ca*PtIwgKO@)WaNo3iw#ApW zKKtkBBW7cp&o<&f&lYTK0k>Qs8>s`w_lmcl52w}?o@ficj9l(2ew%5>i6bT1_up+$ zpemDJ@(m%hiDy%0l**Vrqt+{`a@RI?go#`7fa9TxOGC)J=DFAMB9y&*7E<&;la4-fW zspov_&HH9C_ayuKIoSvkMAXCf&#Ki{va^=$y1@%>rr1;s5#yX&wcg%Cw;#Uo^4)MV zA1+_o*JW>N^)8~TZ!)mk$7f$FpXbn_<>Rk6_ScmG;g+*B!UcjpAan_+xV2}V@FS~} zchb*R3truKPyDi~;XCf`%nJ>R&iH$9g^=cVh3NK`G1j3bG@k;-p|rHRhCu7GPxCGx zA6L8iUaWNY>c9X4OE%{ws_P|LH7Kh}po$=fJMzK@R|q>d;=PurCFquhG0K>HRTppH zt$k(qWNc08r=8bLv}SxyJqVX8&k5?As=CAUyy7&CNWcI>D?q8D>K)l2Kx|0N54tPF zf32{HvAmF5W>l83JN4l~X{p1IJ#mZV~Vt=Z^CM8f)5mrzoTKqK68q(Ql=nZ3XPJ`UU{*{=9!gx%2-bp2v9N zpkwLAv;u!9FV#1Fnkxz+m@~&lJb%gVW1Z>cz`z?DA!pu9irBq-o!C9D9Q=vv>SoA6 zI+HHa;Fd@kDdulC%BV;!v;RNLGHUM!JqZx3AEzE8wu4A*iF%&UrC`={%jn75OAZZ* zukU2ISE%54m1VItN5sY;x6E1Qzwi^R|3+&Ky)^)5{)HJg{>AqhHVXgt9~PgR0RW?9 zG7G*cyg@kVU7HA$JqUX`O@Rv@=!lvz^)R`kaw$Y6V~D+UYK#B9CH%!faFubou;IP=U^qgSG5|9-3EY-yHO zmq~()=U&iH(}*#s z8nV#@e-iB)bS3)uzM$HsuKXMb!|Ntc7TcAB0n%TtbpG$nb5**~6AYL#<+OSDQMT zcj_+oD4Q@Im0HP{XFtQ`b!O=6Ra)WJE49Vt5G8uY6ogg@a~T~ew}>XRsYT8C9+^JZ;Bvr3PT5$KwwIQNbgO3}C6O-&PTx6oqICGVPjJq>jE0Em zFw;Rt&BG^ib7QpLbZn;jtQqq6Ms~j$%qA?=7_J4-cn|hg`L1F~+U2xDy&dxc0H&@d zO=i90)U2Q|9O=Ig~G zm&BGsoI(DA8~LN!#jjRDxUaWkKu^rnFN_V#XK(u5?jY`SWu?bgA4#E!)srk&^4RqT zW2Z--y{7Vbhh=JyszU|rCg)T<-X#>gG5w0Q*Qrs^+n6%Qjr6lQ=8?m3Qtt{s`Tv(f3NYua9Dt9`hIHF&}w~M(fQX{gH^6p^bX`Tu6xMx{<0 zEel<&UX$KsZ*7TD_)=2oitk;`ILiJcSn@>VgQM3Y*A!!_T|Ol^bmb;rCk>%I^Dg^h z{Sap;5+R*#Bt*ps)d_#*lr9o@XC_`&be(c@qg5e?z4lGDo5`2IB!#Uwq7Sg8QCwrl zd87ndO@k1y+kT_dv>4`38q9JxwOl0yj12E!E^S%~ ztKsC{*gdOSog;_1Tu7Zv?1hgP;4n*2FNz7E<63!-WKZNqMfjN&V^4T1INA&pDY94g ziVhh|MaDRV#Z1^SA-XTQwR@t*7f2-pXZS9^m3!^)wOrOm>b5H16wVXQ##Am#8^-N3 zq8Nl<=@gB7?|48+eXG{c@^rWhUkPQ@<0S{;u0Qj`%w-?j z*X`F?-0$Nf->(yTGoZj!%Ingzg3y#WOwoX0;L!cc{JR#*C5y9T4thMFu-PEa%zEfa4CNfQ@eGhWJ;x(c_xN1I*o#lCVNX?(Pn9lT zqhfJfV{D{qvhS-_oHrx=L!$TdiM&)MYc1<>;a!cmE2h!1tHS}i_l172j*NH-XVGj@ z{(3La>gqgAY+T*z)c6xXcapU?k$geetTBFAf(>M5}H6)2+h9dusJZ`b=-@7D&HW6F7A~n04o+!=04pn1=`< zT6{TIWW!OI0N}yTI!{i;(`qG0)ib5a_e#-(cdR~i+-<2K#dQAu?!G(ycvMq0{Rl#q z$kjCG4_yq$Jdc4^M(GWGe)KBAamxC!0NtrgBcX<*tNaqa@hGD3$lEkh#ZpK{rF@~F z0d6HWi(rNfgNv6RZIGctLGs|f!FkM*)QCcS^>?*cbcIB!&{Dhd3cJb&Wy?-IJMka< z-%KCKFNpG|!LTswSXG;b1>9pwytM9JRe}v@d0nIw>Me_yyoflR^+_{b(zjgYi~iKu z2Al;DBve#2!&p(xi4baVWfh_26wxkti`_fWQ}lisw{~6M)0Fbv#r4)6sN1y1Naud$ zkluh>sY~JO{ZWW>MNSm6OZE22U7=N*K5atO>Tc>%T42DJN4Vnzi^elE9+*6{Bvh=W z+vjBD=kv*5JZiHZU>OmXZWI`@QD1$5OB`Bq(P1HDG}_lUaP^SS{2sNmbWXcOMI-J? z{9T@n*&XN?DS2}%-FC;v*5l3AP=FQeD*U4-4z$?I1Z2uZY?-kKH5?rj7?F%z;<0bu z+a%M&jQ?bQqf}m=q4Cp>P`sWX4b;Hhqx{_K*k(}UY&LKfW9V3x*$}UPhUsZ9A5R6Za$MS%TSn9!#E_&q{LO!Y?d_+SZ9@^*AS6nTVc5gnUHtWPt&B?dbez2mv#zpq#FSzTD`30}S*oA3lpgRB) zF$)<)|EwvbkWu62PTXPj%A!N~;jh-;&n?TO?w8x2=4TP!$$az3v|K2`1yq)#+B3Wp zSX#Q360FV1sHM`Yui4pZz_Y2w%lqI~M^iG}9nP{=u(;c6tw|lb6 z=X1t#!Fa6K{*kz_m;?F$4hs3r4*mV$BM2(2L3kU|We}o@{FsO{qjBD%j5e^6JpmCO z2e}s%S770=HO+6JIQF|*rH-*3S;=L_J7Juw4aq&fbBk;Dw?N^5@a)cA${**ghxYit zIrO9^!_+@v(x^Yl;6Y1O_tG#xqx>gF^{<_$f48H$4%1(Isj^-pMC-+j2t^b zbfc%Mhg4gCMlQN-L-s-AVT;*8poC_c(GVGd#6olag7?NeJZE@kv*TBi`2Vu+*CB2P zVEfHl0FOPCa!7cVo??UPCYli>7q&z`OlD=Poijve4*qr&5TG@~H&1%Ec~f|(2SH-4 zAASqd2f*~@xIr^)C2@crVcD=}p$otdsGom>AN-HEe?Rhp zl`M}#QFAr$8nh@nA4p}-qUt@TQ4#3X2iQ0)Ns0`-XoJijn1KI!bS~p>y~}^U&Y<;> z)34HxkdS3Ob+t*7VO#=1t|rb8!cUwS-gl_S3<6Ox`6j<6T&!b?Gh-tUow~!)rllKE z`|4E3ip!hv*5GUXLR}>>E`Bm*5=o77XBB&uVY+A>fI7HwZOMuM7}qfQZ%KM#BOAlPR&ji$<$gGQ{w##Gh7MS!t8QPqefn}J}FdZi(ccD0Zi zOH?MY)qvt36##q|!3u6@&;%uG1Iqx?;>~7&(s%8GjwE0suLVhQxFfPzBLp-Zu9+f0 z!Qeg#oIx9aU_@M}Jpv>yxU?j_hZB<>u&!io;&J`~&cCxRvRA9&h{>-klyN-er! z%LpWBDTGmuA_vR~k#x>Q&}Oy-B@BU5P9%+=YZr2QxqlmyfcXd|^kU1OM))2CaSy1UP-qI5@|>IsYbTF2~v|D5$djL8!(* zOTL1{^7H*lB)fv~r1WU-Fi^k91eZx}g)Ufne}+fhX8uG5H~A+;fc-3YRPnDa{g*e@ zwg9=0YVc8UwjCVrk5V1@S@d^8e|71 zCByhG;uF+aqoS1a^OYeGM~3(O>WI$b|I)RXVOgvvmQAmO>Zs%G8QU5(2=sezP%T`1 zUDv>f=&4a3(e&Pmx*BJz@sHE^ooMfcb^thX%et$v;S0xt!M%5SdwUYqhR!_yXd<0_ z(p0rpjj8Ogbn;n~&zJysLK-isBBmj2zt2}IJpcy&#~GCk*k7#59;H8R_g~zIKW+Da zI8c9jB7b@!zz6%&6ZzliS6hG(NmD&uw&e(1tfH57mBZjHnq2YrlkMbqM`@#uitSix z(4>i#6^lgJG5&S1#{4lrss+IeS)@^KM{&9-0RS>koHk~MtJ5Y-N+a5Y_RHlvJBRYv zbmhmlxIs9Z>OOTJ%~WCKjj>YX+dTUUU~)u1{FeFwu<}1dp$+Rl=iUD6QtS>;8|C_q zz~k?aOn+Z#{a1Ydb9!=N#rCJ*{|EZ)zah%|t*!b`asFe(`Pb1De~R-TBhJ6_fBq@X ze~dW)$`$&jIR7!?{D+lp{uP&>-=Fh8Mx6h!I>ql#?O*ZzKSrE?r62t#J?!$GbTz6M z7-azq;`~0N(JE%6R=96C>Vs2H^}A}vQ+5+AdLFKK@>q{T1g_-2ea|?00M&tHqJP9z z_Ai}LXse(cLdbsb<6<&C^q_G@N$k>DtIsc_4$G7me+gw$j=p|wFTY!_6XGN-|3}Ls ztjysl!k(CeQ@XEXi4(C18GB8(wR@@jeEz%1>?bvzBrq;Rx?&d>(=6m}NXG!2F9eiu zE6!vs6%ydB85>Q*wFJSa#gKuVc6I6v0IeR68-1~~JfOn+^}u8!+rC%dMGl&4!jj6U zCNvaP1fY(%39$F@ShS$JH?Q4TP{9cgbeS`~d#HlVv^8e>P-Q>?yNeCfscpXdfxfik z1FpIKKdIe8&XW?NP+N$t{AN0>vd))soytKTz-2{*Xpq0;FO2F*#`K@7sQr*_#>ps} zeumNT(2;#QDr&)$<+pFOKXc9fV8lkX{g^(Tl6SL7D7}Lvd0Mk0cX{ z3OE+g z3}iZvU$NQioUb~4STqzK2?s;or*pF^VT+wJv1Ypwm&s;0ZMJ$N%u?EU+o?~N>@L0E zo1=EqSUavDPUQ$jlvlayB+GIu|EB75`XG{U-g{5bc{5^WEJd4q^NzMhob}Y5x)hG~=O{v}G6Ph*h}t#8B^{Gxbj=7T+aoUSCSx zuYY3sLJJ`{Vb}R;FBh+ReVE;U@`#IWx4sKO7X(?`3{nr~$Fteh4J|((@;9n7EpZB# zK3#liuk_}A?hZ?8nJUzyMy^DxhpiZCO>YXaqAcr+uqGYNA=# z2<_tUq5vvGu1{*&IusrcjvUcf9+PWQ_R)ua_)m-*zr#5E9`Ep<-Xr)47`lZB@d>>J z$%_2ui}?Y~{X@N5A3&Oa&2{pF|AgV68xEqMz{ZX3hLM1K@t{B?McX9!0Kxcvg@>PY zn4k+>>9kSx=&NOcVBy^g0~Dsj24C8K>+g&dHaH+>08#k4^(Bp#K3F+QJxloF^$^ut zh5l0C=bL23KjRA_$TPy6_p-@$CrSv|$U}}peuXMi+IHI77hBAaF`eRwaPfiQg);|ArEBBefX{rOv&&$8#LY}-owIpV|PR4z?k`lQv$&fW5I z%0w*6@`SnRVF`$e=VltMu5RCt;r0_|O;{oN?v@O(v@%f#iNv#Iu3?`p861u^FCVXC z^UQ7ZgC5{5&Cd_#r!at(gte5Y(PH+|t!)F!Ie7cAclna)1$VEAxmQC@?<#9*7)5bj zBhTJbI^Jsb?PAd=1Zr~u=3FngMN)v7W7a2OXCE2_7R=eIw~Sv?JiF5sdFVlVxMQvc zAz9gM1i1t0-hUrno)vMOYD2hZbLUF4S&cwogsEw)rbNww(%@@VJnhwsjyIpAXI5

o#^%+Y#7Vi$>E~XX ztks_xr|6pWQNV(Vb~9O0TZz$=P3X5C7WB_dbYrU8dN}>`paALHV$AIQ;3HDp`qE+Y z{UJYU`})AraK%k2#o|bzGurV|EgLGkyvhqhy1b4Af6v z-4D_1e1$kk+*pz)ir~x=8x(^kE;aCsFE|JdaK7<;S@$Bh#(biwS+(FqRKBVxD}O)% zw+pQSVMt^oWfA-Dh3uosli7&cO_>rOBr-XS&!GvE!y1D7G$oVY=h)>&`q%C*?l~qh z%*g_Q4z(goiKcYcJrpIxL89(Tm{Ogg{|uC8ZlVG6D(HIC9gCB!ch-ZioVzZjY>@Y) zW>+)Bvjy!9A3xuYWdVpXT{eeGo3%ty#PNzDHP47-Rd4@1(>qr%!uRfPo<6^?jmQ}< zQhhnB0t4g#CTRto$3-p{;=BbeUOcX2h>vw0_4hJ%RP9Lt*e;jI;LZEz^!M!IocuB& zz20@8Llf&8+2DZ`+V@4%L7Qe6XU}}b$HDV z)O7|XO8`NEKf#)oPz=C04O@5h zMronv&V%NMv)mA_$Qhkme`V`vgT{XDpk%B=d4+$%vbuHH{Gc&Ye!sU$Zpm`v zxJ^LAZVP>sbK1q}_^*#gkK!%*7WFO7RZL{>Lgqa;!y78INV|ZEd}{M8WJl^ERjlah zxZjz)w?&d(TrJEz(+>`7w>~ZE<(xTt`S@91cpwmgGyNz)EOOB~$euV@q@wUZgXARW z_4DWzNqsQ)&O9`eQj`Ixh^vq-mUbxN0(4Bi@mAZ593TZ$4Ch0-{W>Hn<_>aQ$pxHQ`}@@95T+~0FnJ&$TPgWRqa4|Tb0WE#;$~cPI~O_6vz01)Mrmm z>;(=W<7z*ZKe1_wfoQ!LgXVuRa1ClJoDl8fR6xr-j8Ir!ZLoQ&^vKEm^A|S$4i+JE zLvJuJda;wxFf`2-8KSo^e(e?*l1)8bMKth#ll`eE-No&;&($=NiG)iO?BSsHYj)Gr zzNuX-qc8~IvqO(PA;n=ZP2!A1zRFC=RGaBR|5(?Oni@+XZ;29j7e@ve-s5EE%4uG` zKGat%opp%7*oEgZ6!vJJNFhj>=9w2+=R%9(16OZ@IYw5lP_}?5Kg}5u8{`-_<6tnls^i9@Ab{%$z z1?yNJ*&h$Pi!Mu#MI0uYo~j#AEJq40q?9?>b+e3Lo35&WS2W33dPqEI`KrOl&cesA zzS6cbu036~xRBjggm5Gp7Tx2=^OaZ0%d~zOQiAUvXHIsroL0U6u}0*PvvuDOB9*W- zF!fcHXjyenmdL*m;jfK3Q6Vlh2D20j zVBPm-vYKIg5$B(nMF!l<4YE6>RcFiNal}T*=@XB@E}oVXvwp2F+aGCgIfH}TqnZ$v zmI9}-92dz3V8C}y4mqoNE9ca^SB1F8Lt589bd=dKvUbh-O_|;eIaj=)a~Ek;v1$!Q z$|6ZVWS*JB(^*16$)^2$sa&-1Y}+$$uti4tNsd+tg7V*)QU_wF@0~I>8y0}FJRv~o zyAyK}7DTW3uUUASc5~%Bg*xHZvgSas?|>82&CX0mCBMY^zCwE+EV%0F^6YmCefI>h zz?3XA9q&COw$Z2*jposDnOQpf)h+DQ}yP)F+;9O@`=?)eu@L znTO&<2d=L}nNm2Q*LwuuW`?YF@j2iNAK@y-$zRnTAMY*Raq&c72g4xZ30` zZqC?zG7xFAu=AFa{e)Gvnrzgh^wZfyDEdK>I80L(7ea(kzT_#}qEV++PE3{-cF6j? zT*c(H!rVK` zJBeo@3qDjt}4i~4jGisrv% zt(H0}vCfp>xsgROARoj5$Z8n?6;h$Nf?ZxkZiClWDqTkYD+K<*8$Eib0{G#WqL^kd$PEes1&g-SW$6|kIgrb&rCI2ksr~ivIt@H z-7|F;W(cBG8u-okHH6^_?!ela^*!mct#tQ^mq{JgD_|4 zuAl}?bB3>HrhM$Vbemz9DwFHpOo1Eb(%w>GQx+#;3idYnAN|yGl4THHwy_iorV^Dz1gzw?*VtKMx`g+jg5fg?eY1WMuesWc_I5S~sBNpNI zV{U}vW1fsv`j%zb6kmss+Z9Ns1jBGL_}BWo3?+!HsPr+BG)BO*dWV%LMti(&^_q#b zV>{jK&C%2^EjSwUeA@1l94|5P;h~AIo#29@_wA9(D9P%HYa!9)uf%Lv*uA`=E^op9LNcP{b|n=-y+*@ z*6^_xjZ!q{Fb&PdwvU~9sT&&2$mkFza{V-g3yL^0T3E5fVqr15k(Rq8b5=;~B1^&v z9i8j>^Fb2AWa}Ah8|vdMPHAygc5#eo*1pA=eXULk8gX5Xrl~h^EX6 zRl3z@qooF=A2kY;Z}E2weequ?DftSdE$*-w(9cg0T#BAqG9ZB%B`h{biDW(EmZZah z^K$L#x_vVGod2N*GRrxtAn<*=daTNb4teT=^x*CZ+g*7P4@`J z)KuX>^R)=4pmGiBiEN*{~Su zb>d2^Lcm1xmP|!^ypx88NOZF6sFO>~fe&iQU5yv>_my#Tvg`!IjrG)jj8iUiTinE_ z10EqWc-`Q7c}3V-tN7zUe`t!&v->9OXY;hr-U!;lSQ36v-3n38U_EV-jZ0S+EYtO4 zrabYw;k_BzpVBW9hT1}-4_7f}P`?lao`EZ2zCZ?@8+-^)z9TTE1_5lRQ#RQr1GZ%E zu;rX!okndt{h)E*u5JmhTn1s@CL4K@?{Na@9?naVgR5D~>s3mg?dy{Yha62kgHnwk zC-Ih+5Ozma2_RlRiqaR_(2WA_gQP47$r86Fs3y33?X&hYk zBTE$vUdJUIM3lI5g42(DpZ?62ZlKH&p5&agGe8+SlD-~ z!Q_^k%|jIxUYQ#}h#jm?mDxE}_U=&DS7X3umANe)Bw$IIWM3D2KLmT?I{Uezg50_o zZ?)P|y&$*56^3f4;%Tp>5Eos2k`h0;wrPs_gOZLVUe~tYU|@~hQ)P|${<6OAly+GS z#O=A5$+RyRy9?mxyx!B&AqSqc!FSi0;b9TB4%DQa$AD_dK(J6rb}P zVl^5HY(X7a)2GOmd=C$cLwFOfdhW+EqhkCcGdk@wJ5M%sO_sj85(gOBX71|Fl5WuQ zmXRKbJV9kv1h;TSuMi>k!NRoIneMUEN<~v`Etl?O_U5ox&b3Lq^##f8Eo~>SP>#*N zLm&w@u-&zNpi~Q#`fBwTD~zA$&12=HOl6;34}z?Ok~^lzrc~ChLe)*2z{1k?drM zEXk6xM9iISvL%W}CWEqX;YNtDMz*nxr7UBt$yQ-73NtE^WrpICF}+vsdEbAY`+1&o zp7*@xIp=+kKjxq7T-Q0*{I2i%`TahhuLfh(v8GZfRt+wJGHIJw%v}|j8?9K{xY4#V z`t*HMjyqyhCjoK(E9(Gkz6s6UxaZ#qb~zo2UU$s&^2iCTdrG!)rEO6YdLzX5O`4|k z!QjNY1ypKHpgyb+UyJWshoFM?vgt>$OZEX4FTVctsEMm_lJ+$BGqH-3e4?Fy?NY@F z%amg_a_ldf?{<^Q%f}TX|GUOXRw|-HWGV7~l4{Pvx(3BamnHX%VjpL<`KJSSa&4Oj#DalqU(M*B_ww+rkVs5JGB#vE zk?Xu+&#cV_Ql;=OCSvTItIUN9mRq))Vq!YA*b9509+D@>(+?ZR^QDus zdUV=t2F+8hJi2=QUR+`HC#&1_Qb}WkJGrJA_3DT+=@t%9w;oOxOxf}Ursszd6#t0h zII2|~U!FSFzsifJZs};N>$%F0PD*RKFH96wC81wtducd2c;KInnodFVD@`qKB>*h4 zEfWd~stz$x1sNk#jl37Qr$vY--M~hrn(P<8Y8>pUcNf!20d3=^V+s!>dJ-lPCTBj+ zUnEGzuWAynSq#cJ))$rYOQH~e99B){T4fQ5AYutx~ zkA1j2r&_PfHyM`cO7VrU?>& zo;=L_8BVo_^J(;~PYZU#1>3~1;jX8I#q^mAN1fUOx7p# zgD?z1F-!&Uryy2}$#|~mS>vE}mXaLx7w+x{Zl!wM$K@hsH%cS^ly_r)H!g5(@~b+Y zya(xZ6yJ_Uncwt(v80{Xmf4(Xf2FF{imlkE?`vnJJZmIxn4LG2YJv$rMHvWM=Ro57 z@`c9MhKd#~#;^tb>8A!Fg+E1{}sTba;5m`)aYVfN} zIp9UydNnw$nWN)V(vgM;fA!hXVN7(wjAS_$%?4T0=?XhXO-WI`g}OMO=iTk{eoe?x zzFqr^&Qk}KXsE7lx90Kr&EYU4u)yFn+f_og>q4G;HD%?Cy`xKInTPs6ujg*R_u<%6 zwbf7Y8<}>A8(O5#R|lB`wo(Y_aM(dS0t>kTA{D=wVv49+d7+vwUmE3v#^ihGM|?`7 zn7O;*n74no#JzB^*`zv|e{*24AO$an9KP`Z*=8l?o-{0W!O14_viw#g zUP(_1O*qv7L0zZu_Q!$1q8X4^<7}?F(bb$Yk@v>2WbH;T2T64&M>9f`i;syV{QNqD z^9LGH4l4L_dDdm0H2r?|_R3@B>wn}pd@@Xqa=w&06~Ac($RSqqZtg0h&g5RA8rOO1 zo%9JbkVthmvbVWDv(Bsbxd+ms;9`RdScmE+Dj$88iXHJ8kwHxKDxiI-h=T zrzG2++3w*l*uWz&AdypHCMIyv)c|x+@4Xx;qTi2~ZT1U;ONA}TpNU(bLP<8_Y^`7Y zq34w0_ITDmuAedoLNhXsmw>wNNwzM zUT<)*Jmz5u3_d&+(vx>&#^r60Ecf}+H_vXn*uEQ&0So7n*f+}|Z1#(mAk@i*qb^2x zj4N~;qI&^Z^A9;}1GjVRvMQ>s45TI=iId`u*oY1ao*|rk7LHQfwOnd~<0k1sh%UVY zR9&}g&D5cvveE6TP}P9m;*o7q9O5g_*_V$${bH(i#)Pu8cQd#V)K)l=4go7sVisHa z((JKT{QDm^@lLOuudb~6$-6yH8EU>YD4{VRvM9s)(`{Fi@fx&pm`D2w$|yNAL)!cP zx8^I8L8sV>l@By-&&ZIk*y)C&TGBKJnBz0QwD0!hi-7)+c#j)Y7DHk!n)Vn%Ba!aM-0%Hi02a1bVz{33UKB$E8M*-WaP#fxORe#KRwiSL_bdXZMOB=gr`IQrF63!s07Gnh3kq_!0dg9eh}pBvy#2NpWT?grTu)Wtu0N; zzU%UpyBl%tf4*4M!WQBY{!@?U%T^XDXOV*!QmknjNkK1ou^d!AI<_?Z2uj5#b@nq5;exJh|_a%HkJ zCB`3|DBc{w5Wf6Pfg$`>bx)mQv|E3}sWagmc2`l(^?VXiqZ2X(n@6ndQ9?1Xjt`de zm1*rCQsjZ$UVHiS<+E)Sc81&s7fh!Kk|iXM;~6SKGWH}@Ci_hg5@7G{XiV?JaH;mB zS`3M}H0Y*AYAD-vENPR6l ze;d?TH`Nn>J1%3m+Js*?fIyQofHuz;=8v{dHg}kII7DC*qdD8h-b11c3%nn&6XcJS zAp$%<1##>83B%Iu2cZpCD*qMXU(H_St`v=YiDhm02?Kt(f zGJ*8g`^?+dqqVTuudF*a!7uJdcW&BChk1`4qALM!Xh86s^Yu2h^3mWf&8nbK(ZtU8 zuPrB{o|;K)ZhW~Ic!9;|=L4EO$U~n;^SS_oy+0eDrxY&Lisn{d*szVCm5ZSrbxRIp zi5L|44Gr|eaNBgjv3ZVFlL~A+-P}KfJbE~a^Gsu*s`YSi1(U&ju~aj$-NQF;L_Og- zvsWNTgE6BCG7JtNY}+cG!jYhRec|Idv!GB5$kld-d!=$=)^EECU$Z7vN}UpB;}6gm zN3RldOjcpLEXedeIL9KIe`%E$8~c6QHTsqaW#)40fu+Y+O9JUnWQK&E9GU6V65dH* z9Gu%V25iKOw8ouxrasO7Q6+F5wnKYp-4_CMdptqQ=c6AsYP!&=Fmc6nkYFyvdqC7bC_=C zRdBK5V`)a@C7DdhGUdWy8RZLU7ALrQd#x*2xcRlPeJOzAc6g9ao;*3GXGZ6Kx@B$d zwRirDVpTRD51zzvj3J{J^0TdV;&%r@6-B|ac{IWQm=nr$!e0#^tEW_Vnh0(+)MRe1 zoN23^=_)U;Da*LODGZUjHaP`9#1KVFQx+eoDS;aShwkM*^f#C&+Nkk6!%QH&d8I5_ zoOt&79<&EqcX6b;V9v?gPD4WSJ&(P5x(xL->uZ&Iu3@+yWqSygtblR2g1nLucIax7)F9Sl@#)aFiXbYM;KO|s81)oOq1W)SZ5 ztsB|aem3j+*<^YlRZ?2_gHqOZtPn_LqdMS8f)r?Xo*)5ymd=3=C|t@vg==dgo~ioK ztv1}=T5^S?5O1hplMmdDDB`pB&4WH&9E3NvGt91;4Cs|EduN2Y-OP5LRaYCP8Jd?}d zc$#KRGpo}mmi%mBQ9Zp~VDp^RHwP1f$g(_7)=_^|X8PB) z(t%{H>oMtvjZVF(h;~cg0slOkc573N3qbH!h)+9@iTPyf%6v*Ji=6WIsC8IT#}p6( z-hu^SeEYNXFS-qdBPse;rbu@(Ish4CK`1P zb?H|4<|p0d4zKm?ZkTM^4f3M8s|%=?vHT%|>^Sr~D{(C=;l zi*&1#={#pqccL~m?6DyI2jew^8GzG3;QtiuQ5XEs_lK~a9KHsPB$wIQA4hCe)XS^V z_2(XFWlDXPO^mTNv(cVhMPnY5xVH-2O{X)!lgqHpB z#F>VMBTrjAQ{y~k(yV&*z!QIwFcd5GcB>a9g%gynr`V-dIShOuJVaDFIE{*mMbF4M zdYiUJ94AK`NQJ}B`HxZ1AX2l{vRM^2QAO9KL55=cIjg>UT3YAxdmoXl6-`vf zH+HmpE>@2pcro8DO@V~}kZbhgc4{)O)G4<#ln$~Jw=w#|&d66q^K>Ww#So2Eja?rX}6y1*B72%wvJSOP?5R-Z9vln*U8N* ztqXE=C+fF8hW5+wi`o`N%MVH$mNa;^eK%#Bnj4z=mI%ftBbitAp&|d|Y}H@pOWFUe z@W{h&aCVg1FQz}E>0sSd2kpZA6#EX!O8%3xU(jFw7Z?(deheVTNZEBpq2B+1gu}QP ze8Zt+9p_8k)_SKIll3FdrJ4mo0!VMDI2Nr>azi7Z6GNB{s@H*&m0^?k1G;Kp5j02{ zC1l?Nxl>aXFWSNsMTS$=58V(=Ad3-GRfpAg%g$EH#NkX;N=}V}>Meg+>wiQb{-0j| z-}!d`l#T(TAKoxZ*n$EH3Nc)1KuFwoR_C#S0!W7(HVagoX#B%Ma$cMm#9c zXhK)VucUxV8=;`mhI%vTW#z2PT=HO15BQ;{p^W%#P<6r&+~MfG0#@eOU1?BE!Ua^g z-~~4vfC{Fck?}idhA4c?1XTEF2UR~roj^&8Yv^vD-_6%|``^Q$9{Hddr^EFMiaOvR z8acTK#@Dad-^vvSz@tcXTPT$T<`meT&^oA+L7D@*(gH~N{@d=a?Y<@Kd)axLRg6o8h!VaOhX(n6i;5a5s%!AdGSX?mGKWde@3nKQ-Uald{ z6|eE9ja28f|uA0GWDFaY^+rSJamGZ`@y=zcG`2 zquBSE{Ydej5p&-a_FZA$75=@hpf-j?khjCzj;d};@KK2=h30`3=^fcO*>f*d1hPK8 i!p&rHh#)5?`DE diff --git a/tutorials/propensity_stratified_evaluation.md b/tutorials/propensity_stratified_evaluation.md deleted file mode 100644 index 9ab7113ec..000000000 --- a/tutorials/propensity_stratified_evaluation.md +++ /dev/null @@ -1,81 +0,0 @@ -# Propensity-based Stratified Evaluation Method - -![results](closed_loop_diagram.jpg) -Recommendation systems are often evaluated based on user’s interactions that were collected from an existing deployed system (see the above figure/[source](https://doi.org/10.1145/3397271.3401230)). Users only provide feedback (r) on those items they were exposed to by the deployed system (e). Hence, the collected feedback dataset used to evaluate a new model is influenced by the deployed system (RecSys), as a form of closed loop feedback. In this situation, [Jadidinejad et al.](https://arxiv.org/abs/2104.08912) revealed that the typical offline evaluation of recommenders suffers from the so-called Simpson’s paradox, which is a phenomenon observed when a significant trend appears in several different sub-populations of observational data but that disappears or reverses when these sub-populations are combined together. In addition, they proposed a novel evaluation methodology (Propensity-based Stratified Evaluation) that takes into account the confounder, i.e. the deployed system’s characteristics. - -See [the original paper](https://arxiv.org/abs/2104.08912) for more details. - -Using the proposed propensity-based stratified evaluation method is as simple as using the classic evaluation in Cornac: - -```python -import cornac -from cornac.models import WMF, BPR -from cornac.metrics import MAE, RMSE, Precision, Recall, NDCG, AUC, MAP - -from cornac.eval_methods import PropensityStratifiedEvaluation -from cornac.experiment import Experiment - - -ml_dataset = cornac.datasets.movielens.load_feedback(variant="1M") - -stra_eval_method = PropensityStratifiedEvaluation(data=ml_dataset, - n_strata=2, - rating_threshold=4.0, - verbose=True) - - -models = [ - WMF(k=10, seed=123), - BPR(k=10, seed=123), -] - -metrics = [MAE(), RMSE(), Precision(k=10), - Recall(k=10), NDCG(), AUC(), MAP()] - -exp_stra = Experiment(eval_method=stra_eval_method, - models=models, metrics=metrics) - -exp_stra.run() -``` - -Compared to [the classic evaluation](https://github.com/PreferredAI/cornac#getting-started-your-first-cornac-experiment), you can simply use `PropensityStratifiedEvaluation` instead of `RatioSplit`. The output is based on the defined strata (Q1, Q2,...) and the "Unbiased" row represents the performance prediction based on Propensity Stratified Evaluation method: - -``` -[WMF] - | MAE | RMSE | AUC | MAP | NDCG@-1 | Precision@10 | Recall@10 | SIZE --------- + ------ + ------ + ------ + ------ + ------- + ------------ + --------- + ----------- -Closed | 1.0864 | 1.2743 | 0.9009 | 0.0658 | 0.3791 | 0.0799 | 0.0625 | 200008.0000 --------- + ------ + ------ + ------ + ------ + ------- + ------------ + --------- + ----------- -IPS | 1.0864 | 1.2743 | 0.9009 | 0.0658 | 0.2003 | 0.0049 | 0.0459 | 200008.0000 --------- + ------ + ------ + ------ + ------ + ------- + ------------ + --------- + ----------- -Q1 | 1.0911 | 1.2785 | 0.8987 | 0.0572 | 0.3637 | 0.0612 | 0.0479 | 197556.0000 -Q2 | 0.7544 | 0.7658 | 0.9935 | 0.1985 | 0.3680 | 0.0653 | 0.5584 | 2452.0000 --------- + ------ + ------ + ------ + ------ + ------- + ------------ + --------- + ----------- -Unbiased | 1.0869 | 1.2722 | 0.8999 | 0.0589 | 0.3638 | 0.0612 | 0.0542 | 200008.0000 - -[BPR] - | MAE | RMSE | AUC | MAP | NDCG@-1 | Precision@10 | Recall@10 | SIZE --------- + ------ + ------ + ------ + ------ + ------- + ------------ + --------- + ----------- -Closed | 2.0692 | 2.2798 | 0.8758 | 0.0623 | 0.3720 | 0.0723 | 0.0547 | 200008.0000 --------- + ------ + ------ + ------ + ------ + ------- + ------------ + --------- + ----------- -IPS | 2.0692 | 2.2798 | 0.8758 | 0.0623 | 0.2026 | 0.0077 | 0.0547 | 200008.0000 --------- + ------ + ------ + ------ + ------ + ------- + ------------ + --------- + ----------- -Q1 | 2.0840 | 2.2929 | 0.8730 | 0.0428 | 0.3390 | 0.0387 | 0.0292 | 197556.0000 -Q2 | 1.3782 | 1.3908 | 0.9998 | 0.5807 | 0.6892 | 0.1170 | 1.0000 | 2452.0000 --------- + ------ + ------ + ------ + ------ + ------- + ------------ + --------- + ----------- -Unbiased | 2.0754 | 2.2818 | 0.8746 | 0.0494 | 0.3432 | 0.0396 | 0.0411 | 200008.0000 -``` - -`SIZE` column represents the number of feedback in each stratum (`Q1` or `Q2`). `Unbiased` row represents the estimated propensity-based evaluation per each metric while `Closed` row represents the classical evaluation. It reproduces Table 1-b (or Table 2) in [the original paper](https://arxiv.org/abs/2104.08912). Due to the random splitting, the above numbers are slightly different with the paper but the insight is the same! - -## How to cite? -Use the corresponding bibtex entry to cite the paper: - -``` -@InProceedings{simpson_recsys20, - author = {Amir H. Jadidinejad and Craig Macdonald and Iadh Ounis}, - title = {The Simpson's Paradox in the Offline Evaluation of Recommendation Systems}, - journal = {ACM Transactions on Information Systems (to appear)}, - year = {2021}, -} -``` \ No newline at end of file From 51b72aa18f594ced2642c3da750cebb7f40ec0a8 Mon Sep 17 00:00:00 2001 From: tqtg Date: Sun, 6 Jun 2021 18:24:13 +0800 Subject: [PATCH 7/8] Update eval_methods.rst --- docs/source/eval_methods.rst | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/docs/source/eval_methods.rst b/docs/source/eval_methods.rst index 1d0c2378c..d2da18167 100644 --- a/docs/source/eval_methods.rst +++ b/docs/source/eval_methods.rst @@ -7,7 +7,17 @@ Base Method ----------------------------------------------- .. automodule:: cornac.eval_methods.base_method :members: - + +Cross Validation +---------------------------------------------------- +.. automodule:: cornac.eval_methods.cross_validation + :members: + +Propensity Stratified Evaluation +-------------------------------------------------------------------- +.. automodule:: cornac.eval_methods.propensity_stratified_evaluation + :members: + Ratio Split ----------------------------------------------- .. automodule:: cornac.eval_methods.ratio_split @@ -16,9 +26,4 @@ Ratio Split Stratified Split ---------------------------------------------------- .. automodule:: cornac.eval_methods.stratified_split - :members: - -Cross Validation ----------------------------------------------------- -.. automodule:: cornac.eval_methods.cross_validation :members: \ No newline at end of file From dd5a127d00903a03cd6a61cbb6a31dce63918ff7 Mon Sep 17 00:00:00 2001 From: tqtg Date: Sun, 6 Jun 2021 18:29:13 +0800 Subject: [PATCH 8/8] Update README.md --- examples/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/README.md b/examples/README.md index 6ed0cfb84..063ca0c74 100644 --- a/examples/README.md +++ b/examples/README.md @@ -8,7 +8,7 @@ [given_data.py](given_data.py) - Evaluate the models with your own data splits. -[propensity_stratified_evaluation_example.py](propensity_stratified_evaluation_example.py) - Evaluate the models with [Propensity Stratified Evaluation](https://arxiv.org/abs/2104.08912) +[propensity_stratified_evaluation_example.py](propensity_stratified_evaluation_example.py) - Evaluate the models with [Propensity Stratified Evaluation](https://arxiv.org/abs/2104.08912) method. [vbpr_tradesy.py](vbpr_tradesy.py) - Image features associate with items/users.