From 02ce922583e95e1309417da093c6d75745c8d4e2 Mon Sep 17 00:00:00 2001 From: Oleg Gelbukh Date: Mon, 12 Dec 2016 22:51:45 +0000 Subject: [PATCH] Architecture document on OpenStack CCP on 100 nodes scale This document summarizes requirements and recommendations on placing of services of OpenStack Containerized Control Plane in case of cluster of 100 nodes scale. Change-Id: I8d883ad0656cfe4e7a402a5fdef34be802f08cd5 --- doc/source/design/media/image18.png | Bin 0 -> 32318 bytes doc/source/design/ref_arch_100_nodes.rst | 1455 ++++++++++++++++++++++ doc/source/index.rst | 1 + 3 files changed, 1456 insertions(+) create mode 100644 doc/source/design/media/image18.png create mode 100644 doc/source/design/ref_arch_100_nodes.rst diff --git a/doc/source/design/media/image18.png b/doc/source/design/media/image18.png new file mode 100644 index 0000000000000000000000000000000000000000..3c1119999f9451cdf85cdc1839e55fd23a5f62f2 GIT binary patch literal 32318 zcmd42ga0m{;rFeki4#7$b6zfgD zzxR3ng?rBvdCu;f-PzgMot>G_iPBK}h=WOqiG+lNqo^ROg@l9(M?ylbc!h!}x&E4X zi}*veky4dHLINjXJzAh4?q9fTeUwJ3o2EWS+`P6_&{9Q0@?k_m3JOI+x zKRi5?mX>Z#rwp|Omu5OSTPcu};>k*I{DoJZz;as3J%@&dI5|0kJ#~L=7s<-X8fv{K zCnHNqNqIObn;VEpiL}_<+$1N(|Mlw^Co640K*03$w6r)oJ2MR-Az?&BL}Fs1uCA_x zFpHFw6d^vgj*iag=qM>EX?S?Jhld9d5fK9egQlh?BO@an9UU1NnJ^y%A0J;^TU&&` zAu1{=Jw3gIgoL)XHZd`=h=>Rc4GkVHCM7vRZGl@~W1zOG5CH)}T3Q+z2_6m(&Z}3i zBqb#;*Yi0L7YmJ|qN2Hh54@XrI0S0@Z zd|s06;QQZ4_n-CUP5#lHgTVzx@TD=M#VExp-J=di*QNQ$=VFeog@%$qGkBsMh zBbIA|zKM41@5|4dCOH54P4o@w1+$&TVVd9mFva>;l6*+>VgDh&S2uP^yc?A96&i0r zy8k)3m)s16fpvu)N+Epok{Nx`3PW9s!b)KeQk`o#iE$U?*$WAhgV_Te*^e*gAwspOpM z!))kz*pq?&&`3X#F$w^-uo$Gz$XG$s*)W%-oT7Z^z^DI=AfX7N|Fn+ZKkcauAa=8$ z$yeWfZEDMsf@KU5oii9X)Vn{cGlU16AYI-Pypgzs#~4^7(L}_1z_d z+ksJsJ=iz&`|Z{GD)ipPXw7+bv5p4WXTaSg39kz0 zjRRCEabKSXqh|Ii!vATQ~S6XrrTuYM*TT8^vCesf&w^(p@gLm zO+E9GmmXdM>qfzb7);6vvVfF38}9wmqkY=xe2)gEh1fm4Qhz6O(m4+%(6E3Q)OHs$ zChsap@gZjHpm^9f7{|$g^?jF=#=uAey||mjArWM-m{{^$=ml(GXomop5i@R5aN|US zsabhr62(gJ`ez}jTb#jV)$DzAD&@?6^MEHjHyt*GMUnJ+yLZw6 zFUUH?wW;wt_3Lx)uks}B?jLwBo-!Axai8%aQf05_=_IulfB4UQdyG$$oyig#WBNTQN-&! z)}W5xlrh-d&i!^$|xNuV5~@t1aF35?6& z*F~Zu$ZTY?P9Yy0GKq0TpTmbQn_#tHoubxH=S9f$gm5~gBN#vE%Vm`nSQ4cVHw;?m zf}FTWVdr5h((^EYvmn)5kj(UnvO|s47iv3t&<(zHu}$`Q@Q)up0@?Ineiw#ShE8_~ z{~xWJ1Leu+g~R-`I2g-&ty1{Qn}~=z?bCeYYC!@zQ5|?mtx2Xd$p}gQ?T`stlFd4% zltX86)glY!jaMTv<(+qvupN=xiTM$>$iJCq0Y6xpz%G9=*DykGTmq}nvb^QGg}irH zq3^{K`5rc$(_3(N_L}aWZO_HlsR;sH7mPsN^+RGd%qofxY4VfYx7IT!eZ{;V#INmW-1k1L{mph_;P@b8UNHD{(<*VXZ;zwY5*dtgO*H6;E<9~lg0 z(mS7sHJe}nGGjtWLO|dWo0>>3r;IJOaPoc^pruGe0-^-tXc4sAP~r{HkT+xlEdU~D9aq?FO9gpu zKUKgMt2|i;%0z=453IuqnLrqYGsZ1iwt6v&aJZ#qZ-1eSg#Ry}Y2)laQ8-a~t`_?% zoT=R~T|McBZc1$?humx>8(Y2ofgltdB^x_+?>%lXo4V)2Umn&4oE^F5K|Oe^24}1GLUQUGK}XY@*=Dto_Q+>A2Foz2VhUyw{8&!&*zzm$L@J<^7;J zC5m_a!6o)|j1j<(C~Y1ukBQnAt)4(D0!;7tW_qQ+Uk-d2y40 zOm?xqZqk$97dKKt|HWD=V^3bUm)?whJuV?eqA~(No9;5g-V!3wYBi(dBiqZFXY$3k z+Ji5SOX$w=yl?zBVgQU2Rh!kcannW(^5HRn@Ig_`=&FHAPzsAVCYg({GD)S*EGt!%$bI1l|)uN+3~hlA7A4I1z?}|`NhtWX6uL952@um@Ud%@prv~I{dG2f|9sz91_|LK$R5 zk41c69YU&;6Ip;b#JvRTf}83bk7Q&9z?IXpY2svmz^?=xfjd!5;E8ZE-hX=;06|gn zaL!%)x;W*pL~h)pLFi`{ie`abIrzdQmleWrP?16`L<+1wntXd5l1x4s*YhELc?=&s z-zm}?b+-Q9RpaoEJ=Wt|aCFUd!}>dA2E*AdFSx$b$_S}v`d84zFt80%kOL&s=sg`r zxnjA@{VuA1@{=3AI7ns)x7ur~!m|wX8u>IzD)`UyoXQH`hod=u(#b=^ZC2qBbW~D? z{$S44SRXxzm0%Q4DINQ?2kw71A_gG2v4JCEVzjXBM;PMWk1=}4W@UnI1FidtTuo@l zewcKKros81B1>PcN;|d?1%s8vQEbFh>qR|^?J1u)n^)qv3tN_#B;q9zl~@6q3C23s`@fXl_$5e5td z8JbYsYdcN;V+r30ZY&+_J1M{Ew0~mDArsB0bF`)|Uf*9zR@T>}x^*DBj*4;zlC^a# z=9Cy@ekSLBh^ZCRLfUt1qx(|*2}Gh|q))}Ptf3W`XKkphM=<{Q=%*$GM6VuEU6XMA z+R6z+QHr0EF&V=E(afe%8>eGW%s7wKiE@ED#jB@c#xq?%X1zb-l>Xj|PqcX5FfqAV z(QWKIQ3HdL!yK-|8wc?L}~O-$ZOo3~r8vf-{0{Kx5tF3qNDBn=u1`^>qe&r5>X$IpFh z;gg+aDEDyYkaV?_!b04R9Z~T7i($+)Tyd2Hjf=3U^cL)s^Z0dR&qII|(komjXDuJAC-K*D$0=xc14%pk znV`dvnq_xe-%`1ldq$pwkNA8yTwhr)&`;eQOQcLwmxQ=2{~orqQ?^q$wEDn*kk zVqqN|CWaAlLX@}qcF#+?3Rk4vR!M(y7}6@H8FS6S#?UJ;jAwZUyNw`n3?l6DKVI6{ z?}%DI`>}&gcb&ak3I1;O18sV(`U@Jub}hliRGF8@@8#o{QDX9*($NA3NaGWr&Q|SN z3?Yvj@!mp%PO@km8ITDzx+k&CGd&f1aX}*C7#jDWoOfqgWwr@ZlWaEuRC(P9g6{ld zW0Pk_f%i^fx79S!<*?heBRBwbfAm{rbLXcJf=lhY-DXNT&bH)= zOlaL7(v03HXnYoub+MbF4SRleo4J5jZ`|~jBHWE&xl$){UGAuXe>1rRf0avHx@1o# zQ;xw3ZG4I_?U$LsBxWUm+NKD5%~DsR6zxnbvYK%G=+a@lJ}bP-jDZx=vL!adY_+-j zRh0Ac9SBp21rz(R?+Bm5hVT7CBcMqPZKPfi)h;rX=8k#@TJs?bxceH-Vv1NW3VwgB}}}IiSNIyV=1>yk%3WubX>W`O=tO+%AZj#x2_#Fmx+kBy#~dfyzkpZ zD|d^;f>gj*yng!oJLfSxz?%!bd!5>4Ouvq<=`&23&%{yA{n}~Z=L|pTZOA0C&EW!O zQ2ArI$`&VUFiNowoI9-gtJ!BaxuXDYANLA@8HH~uMv`7&_>(Q^8O66eEi-OcT&uzP zt;V71rw4=93=ajeo*YW^(Ra&_F)6}F48KQH20xLE*;R)a=3AjN$HS}Pr`1m}_jk}M z!I4jrwO8wm?4TI;zZoe;HS+S&;3!!iPDf6sU`MGm6Z`8)c0!dm&d=4ZhNRm^0a@GS z7HjtJ;wmkcm7TcEx)?8jV*lf?j77^ObeKauM zB!FxU{PZ>{AFH}>G%lPW4o0FB0h`F7y#1i)_>Mr2Qx$AXAs$+WqtCBMQ|_Iu`ZM<} zD1Hbx!!j{(MbARgy&~o3B`Iv^6;yW+cnC*ASn#8P^xwbc`SFX{LS#9~h zL9i@j-qCDKgZ9`?U$AunY^}02zsZn%EmAIb%bOk4nI-?BNzzGifwLr5{n;08sU2162ew@JmH7duQ?7U<=F37g)E7`CIT4 z9!8$De)qqTgCqO(eQd4F(skv6N)=mzucti9ygCPV*BW6W9u`!wOumwP-*G=$c#s*< zuWDW2>91ww=_L2OX)BoXbrOT+`mHdUgY7ZbZy(@Cna%4l_P5`BKTZMIZF z?(hM@S9jE(9eWkq-Zyg6O$WoEarQ;9G%1(3LV2uO!5%wGVqS+w!)WOF5T_`qXmAH- zkLcEBPyIsNf1+w-1x*7CV!$0Q4U|5n1B%VxN)n4I<;eJhhSN%em$z(}@{)uIxfPuo z_-Q$#Xq?jnNg(pPcyvgPD}TbF(6^s1p2=QO0En>D)d&Z{_+Nw*+&(8T5A@n9H3 zxO@(6>@e`apieVe20Sm!QO7)0EsSPDFVYC-ZIGdoCzvXK+seVpdrCSc1hgT>KfI=y zbh1In)@O)QqTOmi!eD3N1ukdhD6s7n*A_IaB~?bWAJ=516O(1OxAAXhZ)Me4wNa_& z4-S{HHQ);mTkS*^InnAyByG4xHcebg+}Xry+`-2N*Pmht20y}a5BG71A^jXc>~ zSdOGzB2%M$`uB*qBE-1mI;-JAmM)scnx}syTO^zg7zU;^`v7z1KfK}?z8S&=z-Bdh zcZbfgp^>KC1yQ0H(>^;iw?K612n~hUcQxQmp}8b)tGH$(C4j#0@FtNc1*3=>&i9Hh z#5n*d-H|u4YmM-YzhYhD9ZC|9*zb|0L ziZJ>B^_0q6H8 zN$ch&Av>+ET%cAhS@Sd{K|y(X5YI;NK~a6eN6V|)Crrwn!2n4^Ct3K0BnYNyi1TIC zhJK!8nsd!RHXSlw+<+lI^H$dw+$8nV!d;GMq^H|nu#WZy1E&SY*fs_T{F9sojjxVb z7tDv&g+{W+4b$V;(H`Id^#!5ib`l|iKAtyYf&GZtPXOLFID`ZIg8T~`s6RpTvv$^ZvJjeFOg+&h(3*b|$E>6Bo$w+R z&Cz6^prprP(>A{tZUeRj>K32u51(}efMvFW#?@aTYw90O#&;0R#0``=rOf4-Y_l0N zh!@3}u413Y!DeATg$%^sHp8TVUibH=+(Tz0X;XWdU}9?uu^0d@gU8-p>w39{RwOXX zA0ek&hx2fw@v?~vh+Xx3ybDiHFBjdUQ@&5?dsOX|gv`Oxx#0)+V|(MGEmdk3-3*Y7`C1I!{-^w|&R5r{(m&Nb z1fuXtzQ5Z1l1j3=V24aH&>-N<=!FYRWcCKA|@L=d0B>tIry$=udT-E}(IOX!M!gzLPx74TOTg_NO zB}?bDd>z!(Llf*6F?mfW!i139@(}9Ip0N=_x^lou?Bu!1$S*G2=a=utvh}6vLY{tx z!zfRAZVW$lJUxC2v<`4>KR#W{iXZtiDxWTGNgDER+oz6RA8&SUUgdzzNVanW&|Jub z{HQ67Hqxd(M=y-NZJX3;fQg6($$))LvXPD|NieuY&mwF=eJN*g?~{ zg%tfY)f&j=A1DW54CJ>rwuh((go&heZ#V;ZKXSa^Lpj!@uc{zxqM# zdHN<(HK1iO)o#Lne11@eODrD^y&Ys#zwNj_7-tQ_Vi@um%JdGO)TevEEUsKe{wHBr z%=(jwjX&s@m%~^Fn}gpu7Riu5XAoFMlBs>b_5^Y2L7Y);fVa(hgSe`Yt0(gSlNDvl z(*plb-h_Nm4HFyw-$v$OcMwt@Ok7*<-XqzB&^V#4<}-Z!1j{Mrp_d!4@pRs3CIw^5 z9?|8Ce_b98owspt_NqlI=gQo^{K&C=jXbgQB(&J*KEp1OUniXHL-Ll*TY8zc&v=<@iDkF1pgbV60Q*4u6CaBwC-d+NsSq(znKV; zdZO9k*|x$wwiLFYi#_3ckv*Y^kT6R;j+^sq#-R}8}8!83?eDH z33qDbA3^)2nszLHL1@YeA)^CyH1gkCbS)KOJ$U}js)tN`G$>KRdDuJk@o^)FzMfqe ziGswutqQ~!UH?JCuTz{YWa+CfFR$%u(~|PKb=VXxF&qb_jmk^C`)j`0$X4u+`E$-M z)_W(KnQy6}q>2{0*;HJzO@8sw}-kjNKh0_(-weA?pCcZD%+)(Ew9 z!Gdg0=)n`0X9>Uvx7?Cne)d?szjGAXUy2FY&~qn~!=)nFei>AMf%z5~ z(GrQtV!ykMStGfwb13jre=VYg+ES@%;+j!Bu#;!blPuK6h)i(FEUF}odJ@-tn^d3Si7#V+Z5(}#3V5M?6MMu0`ug1$a&y))uI?es~Tg(jL)L=5m zdb_FhNj9HCrZ{&$Gm`!4%TzM{TtAXQVKDhaae3 z+H%5t#f1X1_{~3sGGpBH>N5`mmvVaoHHs#j|C+PQkAPl z0S^xNy>rTgmFjFOFi(bV(wS2-g2-gGeM6zs1Q{`a%{ZJ&L(Hx*N3tdwnWhA52Q^sZ zLdR1&wzz>|{GT42==CQ_^vjtH+f_p_o2)FS@hh#Yz@TzvRvn2)9n!Nq(2F$2Gj2Ld zJIHG8&f7R8zg$@mmYlKd@cHvw5J@U)M*GW5riAQte-{mY4|-5VpPfTDO>`dkCz?3g z?;f85dvxYc8`0~Ln%!5k3(ts!F3SYEAR6LKQKe6f=$_SsE4_{><`7d|UwQNHk8qoz z8MRrj*7vZmz!!&)<#8}wm^as$T5%@9!pr8(@6;zACWd_PH9dmDqMHac8ooOY@0Z4& zHI+*<8KD{ej<2@+U;EBjmJCpCUu=rQnlXr3I@|G`OZL>pNc;Zdz8^u{Y5|;*`zQ9!1fNy=HlV~`1nCARuDuhxzJ74Fj9yBSYV+GwP2`axa_Z?5cO?3{QvZN(3h5L2;ccN-Fpe zi`fO4AvM^+auh&OcTD%%_jIG@W$T>sw@UGT;9;cU4_M<0zg<;hYelQM1?d8FC>SgV zY!q&Yz02H=n(6=xV?Jm}XyJHN<$U5Rbh_H>y_}T2 zhUW`J|LDMPtcni_o?R%1#>=-8m&{EoH{Pe;%3DHS(?IHjYkz&SwU_)iW?b<6db9d9 z_A@I8=9@Z)9oA^KEVb+{lZ|yLN<$sCZLzW&2{`tWk%Q6{>hA~^ytFlou>{>rA5$xtPEoEyiCv@YQz{=d z;K=UDpcD?Lo0GrK7tXZByFXUmz+D>$)b&x&whqJ}pDu zCm;f4(d(nOCl?01a)YI_3)2?=q#1?87*I|9(B6|>CQ3fpdRzZI$Xf;G>U(y0UMf;) z@h>Qm%WRvV0$P=*&AzfSEotvAR6pA^Ar{nBpdqzTS=+axKre#TYtB*L` zqM1zXCL#6>vU0C%S>F4cx)BzN;1Nn&nib8gUj*%EcjQxxU;Gg2v$M087vGV8e!z2A z9lXS2@w_wohGW@eDv=an-URc~{h#hQ4&y2&m^SMk8N0K`l-MK_)+{E_JnO!qUmEun zbv?2){>({*#*2I!EG4c-r4#smvyd^87>~pKWoi+XoYSASNp|*!98;<^Lp3x1Elv(c zvcGt{GWB1a+5I$2L%20?_L^LO`b%B8SK4x2R)H{i3{`5w2>Nnr!yz*D@`qEtQm$W( zIn7Iwi<;F$0*8_45OC^z;t-S0*jyL}dlY7EJE^_@&3Z5lM+tfqT&)E4!ccQ4q{&-R zX397Dl4AX4!=Lx*rr#`171uR8hYV6Pu;3uqVz4+~(4(*aC2gI|UvkjGk}mnvn)rN* zqb2jnj+nS<9aWFX3UEA&M)LPNKJ_Opijk8wk^TLy(i><0lPKv*EV7l~-pm%QCM#l- zGGHn#38;9=MET>2$Vs&qMC`bpOkYox2hN$PWAe4-SX!kn!lQBQ=sP(oa6`Is_ngB& zw?#2J$BI7i2ZuUq{A_zwmyT(hhLvXTd{w-N{TT}8u857BMHk+6Y!vk`J@f0%M+Ncr z#^KoWqh9pxc%MB`N+2=*U??So4!{UP3@14dg=sQ`O z?KVzLEMCqq7P~zxrPDj`KD1TGZuxwxJS2q})s6b{-#|=aR{hV40@iC!bRO zJd!gf0}p%o7@#fG`7ekI)U6xtE;qYy9S!K@tt}RPo7a;;akp{oV=&S8J5+=T5elDJ zfVJLcM3nyFOPS>p%wFYxcstIk$^fp-EkB zjll54RfLZx{-zGiVBD7cgD){vDs`60uj)Sh z5f&$}EovnXW?S+SfRh+XO-1lE4dhGyh{d73{KZrI_c0brGzMmMFUuq=@`m+#upi^u zFK1?C?Gt~C@c|LUujP-tB8()W-BsQeAxQ-2D*(XQ9i|~9qTPS@PEW3q zEWF(IEvsVkn3*~%6fUe&AZK#g5NnAr)@6HGk)8g%MCc3P^3Q-a?(1LfPe%jRO5Ac= zd(UT|&?0whB8b;ILsM6|Z#MCdX6XQs1w*Q)W;2kfP>H!ZJ6x-(id+A!a#2MER#hm! zR(Fi{+&Fm|E?OvyU}XxB%uLvKa~+>|`b&;70b*I{mwQ}C0oudk!3E6LsxtlrYlIM@ z;s&1So#}CTI5618lw5qCA5DQG27w(@csC@VdZ*SIGhb&S#C9luq!ztyWwUt3eq=A9 z3e)0m!9(x{bi1`%5b&AQKygaXwhfo5d1SjP8#-TLc3ha$BmoovYGkxH;L3egRo%@Z zI>Nku-^cJBEb7|U`OESqIY4zbQit=ro;NFM{4Hb$gb<(m_(l+;CgYSPjgddAjj-%l zAn+=E=)8oCd~RRK2&~;&KntzItbvuDp{cH!`sEIoKew_uI6P*hFwR7;A9XOD+epm_ zTX&=(4n>m;pjp_bs!e=BaY7Lvq1R6OYS@CgMG^yR_Dk=ECJq8y-o0}AgOf>Ac>+{dmkTYED^ z2aDL2;La->rSVC8qu%zfpTlK{Hx8Md6yR!PuA@1j(t2=d-r2oi7=xgm_TtG=Rms$N zryVBPJJ1NeH)_r>dAiqgnd|C%P(AngX?9^FQyD%)Enbc$k|Q9}P^i^NVpVA^EW7V- zIvGB6{zZI%Tq({u-oB=tcjB!|37AaOUDYE|(T1L&4 z9QG&7e(6I9TSD2*hzXGCkeqntSkWwL0)zFV3sbNyE_lM`vauY@02T!#eE)%CRYVDn zd4WmqC+zFY1v;qxARtS?yf2Z{T$bi;WT>E7NIK07;&cboC65_)_c+9t-He$4g(d=c zqXYp`Xp$eScx7i8b7bKBlg4Cs$wbYFc8|xGiad-gE%E`1Mxly{<+|*b6!H^ajkcw` zb~N8C^k8uau{oHmIObO-C%Z=ysn0r28g_dcSsf;RK9E(=r&2vj(~ZirBT^JSi*A&) zp(uF~vrhSEHf^c`sVQ49&KU2MgeSw}C{JiMY${NUwlpAMS@L^_Xh$7y_9_=&6hLq$ z;Ip*7y$g|B$j={MBYaVLhsTNtY#tq8r6#mudcRp>E*Mz;vK+PGWFVT3E**4DF;yhAx?=3M=#j+q6d0o>*TS+8vIoLQF%3)UfmFL|4M{SV0-( z?BsoE2p`P`;Ov+vW6X+b3{kzFc7_|h(Z8v7$UGLyzO${2V3?pWi0SdLXu7FrtS1B% z2tH`QPn@r(t^v+ z0#!*mk~b|B$!>1?3$J||n#BuMY0@eP$gbbReoTC(C#&quEx%^f%-IN~`a! zoVI2bza+EDUhC}6m}NHxWhLpViu1^R%laZ0^7&1??jGgkws5pzT?cJC&Q7qNSx;e~7$g5{tTan+ zWL~2Kb1YJ@R?G5V>txpYbX`?eQ`~%=F5(Tw6RnR|f9|4g?i4C`bL-=imZkKFg`Y$k z_>frr@^2dJzoSQ0jonQs&9|aqT82+(W2X(W%nFFS!SEx1G#a98dRSi^^$A*omWVl9 zUsH3oQg0m8hoLa$e@QPd3^)C%**MS|Nf|Mi-I2F2+wL+@`r9+4#`xVRoK0g|DfU1B z;=tELjR;#V_Z5zqK4T#Kv^)U@WCMIP)P~pjCM!N4xbICP&pucES=%GmNX(I^TB*o` zO=mzqo1Qq$hw-2fcB=NcwR?v+|JTe$?JQl675I84T1$-MRk@cDv1A=Q$mFeo4wK8W z^YhY@QS~Ic@KjWphmvJ1RB*ZCMVmquGNw|6XLCm@C}$~4cM2W@V|bk{fD3gFjsi_G z>!o(NO9^`)4tRaGS`4BURRI~oEF#ouze=y_Q$~>68ri69cYJ*6G_>1m=mUlPc7aUo zc4Nt$o~pr(BJ?h(sEk3?U*{V|fos6SjWimA6-+15lYUqXO{wYve`g{Jj##2G&}3Oq z*-Sde+@5DLOO@(ZGa;E=GOO*o)KwfgzM>!9dKcajGP)hD8cfL2g+^i8-xs)wbDrc2kV!LiuMrk2{G1b@M1soU$%I zFhk+Tt7QaqgTzE>mcW=yw7kxd6dC?fZ4Rn(Fq~HFs7~`VC6~2Uc^aMEfa;+e$sAJ+ zlDDxgsnhvXabQKU*x!E;#Vmb-Q8V~9&dxA7JzIigiVRl62@F(@RplF>#3LI64c$Pe zv)HKg-E^RJ=tvhKSH;PiJQLi-(j*Q7XG4^oL@q_(6gD=}vO+tW;&4G@{Uv&O1cfpW z;#lFpU8v`4f=FX8r2+ESM@HWr#<86yZI2;f8=3RB#7_8st48-&*{9bK=;?aA7bi<+e_)|dHIqq%M5sBg=Kl@Nd1|-y+%pf zWb#3e$+J;Qmo?6|9UccymQIR>X4if5t-(vL`st628k6T~`qKD;=+Au&XNHlP z-{2rHRhK*CtUY(t<~I!yoxdc4n_kd2bz5QoMQ*$G*y>MP>=Or7n?IA+t9ViWJ`v(M zcCYIvLtym=@EB&(>C-RAWWWyQQ9*i^Eixx7B~{upO$rGJFf)#`@;8dwhZwAw+OKlo zZY^WYYQ)9?fZ&U}LQ4bqQpm)fFTthXPD_JE6`_@%!mEECQAI8W4GeNb^Sl3Bl5)r2uQ{41QTly zyS6KyC#Rz_n#o7kf1&yv(ji=Wu*s!w^pAguajl>z%&AEvgzN150nR<#W7654S<5L{ z)ed_R4=_+>^i}|&{fpT=01sJxwaSLm;?8zA!dgBjmkS`G>yfo>pmJ{>E}E@h+XF_z zTo0K4f-S?)Y1i!bSDa%{r_iu7+|2TVyMG+5k%}H-1QlNeV_bJOWYM-%$GR8XMIb7S zNk!>OX$HVdU#!XW5^9oi-6$|G{&k8-oR>a-5v+@xdicQ!R|l2{C}3ry-1Vpl;UO4q zWdq?2hsFIU`0f{Lc_;S3jG)keHosg%%3Yr3`}ZP26|B4W&o~ChmfevQ@TN165AWyP zK}xM|5~CD8R^z~tq^411MMjq}BI}pqVOMOBc@~qHZkM#g3~Fb${X6#(j1=YKYb;kO zz|7Ef*Q3{l`!B-4ihw-$urg*~x1=$1UG<`cF)IaSGIpmCk{BV=dRxJ@(5#F;tejC{ z;m#*2+b`F~F=Wi$2vo6`ypDz0QKlC4PQ8qS{ca^Z8;#o*lK;l5H5z0d?X@1%S|TeNLX$CvE1^WB@X=# z46w_y{t!GA?Wl=iyI>{H3r^r5`C5i~Kpx%3Z|kAXh+oCigJVjo7vMBP&6zly zW2UMn#K=dC3D`j!uJoWN-&&cHww~*HM-_N@{6I9yZoMEfOXW3IseHx($0#|;z|HM$ zq&3fDuHK>epvwP1N}NLZMbc%{Ih z)6`DE(0a5J6H1cOyl-=k8q#TWE^-Yp?~yQA(b6CMJR`6BoeNkp45vVW+E|8TxIAIy z!|Ce=%R**Q@`{|WLn$a~FcC;xGRxll;(9f+QhaCxt3hu@@u zw^&-!==4<>euMOmrd! zu&@1jsTj0I0MR6h3VzAxUOEa8_{J35YYXLU1V(7;Azl~|csKAR0DNj)Ocg_zA%{?scVCCQ*wD{?Lao|9 zM3;jvCyD;Dv!lIjvJh#d2lWPY$6jKpVkM^Ov2^xi+2KSb&NX9*QW_t0 zIKssm{Rg^I>5V2|fv*h%zi#d2Jsax48|NTPvozrFV~{Z+WW!|rS{4+~z@Rd4hf0yt zOI(?~jf%_r=u`kuC&UWCqG+fxwiuwqmll|y$A|)1G=oG-lwH1w-;9Dbvcwm1nyA(bz z>`2{FZOVl$?z74=_7U{EwrG-CH~@vl(*TjcUyit($^H}Uj3Dy+UfkNjVD7)J#A^;` z#U?8Z64XpxI~oMoo2HH<8iDK=$Nel%Hg#~hJzj=X-@`@PlnSG$ zy}ti0SwhL<>A=#;49Y(|Nw#X8x4=WTsj-9_^Qv`u==3g&QSy^TesB7^XBtd@r?p0xVE&qB-8g_o6!tQ_9d`^J>TTA5yI|t0waZ|w< zn0NdM;^hSL`Wj{Um!N3}e1)n+?XT0Lae4jh#vw3Q50>z&(I=Dhb#`aROf;i(+xdO+ zM}xvG2B|#f{y37{-XsgZpWJJo)R7G2A+KBVVG@OQiCO3&evob8kcLA~s|6 z>9c)1?5oe7qzQW>(RDo!HeYatH{YF_HpB`#?fEw7bzW{zoJD4gcQ!|bJagPmBu2A)_Z(?j(wh{Dh5<2&_@r~q~{`gW=OKLl-<0bCjp)_L}ak{qI{#oM5Z zya>kRecX+nr^jxxqo0ir67c=nwUJwQCNoIwMqnVaU%=h-EH)(w3npsv`4F8~@Jz2y z1ne7hv-m*YX9c`{@u2U*MOQ%0a#XOH`oRv=p1Om_9G~!e1`K<$;~aaSVTkYKlU`CG z;MKa}R)L=v;N4y=obQrITbhe8T-HY?{bIRG1nDrPMkJYp9!@kWAM=|Rtedj`E?)6>g=@fakS#S2|{fJY5 zpu>01RAN__8%+K}H~qNWV%c&t!oTi8w@tA1{s70;i($bBFFwWx8oezqKM7)pS-XRp z*KP{pRqM6JfP?4-&H@2sub<#9@TzwS7idN=tOnG*(Yp;hR!0~qb6fW29@x{1Y`I&- zC!%^T^`k}F==J(r8uf->uk?bAjG`FJ{jzbkCAmTBXqQ8(n{?Oew-wU`JjIrUT3st@7cq$@BeA;t%KrfqHoahd^+55(w_@ zu0wFQA-Dy13l6~v?lwShcO5)P(7{3i1ifeS{p!AY@7DYG)vKbQm^0m{&*|RXyZ72_ zbwj)$B>u|g>h{<9&Cs<|m!9+R@H$y5Bk+iCn6)jTNLO#32~h@_ecAoP-x@o2o%^D= z9-l-<+PHR&b38a<9^cXD_(O$JSDaW<4a3e@?;P6o9>+LL=(|wHs=|8}&>`KN)Op2c~c@w@W&%Y{RJJI6#Yup zyHUc&No-$!vamKfNg}U~;}2y}i*>K7WqPJjAKnmgKtlAfs=()^%{>BU*GpQ|4zfK5T#bB4>hZAhnyYM;mL(39e*Wk;p7qaT#u6-x`LJyUi^U8Q(1h{rkuRImzm4YsrgG z!ma4=QUX)YiqCwW7Kg9hlu6*?3tp#ftCPG)9ppwYfcCj`lj3!qJ+6u)tdSc5scgUp zG53)k@G~+ZRM5;<(9`7yd%-<|{4$ z9#V9^3;D)25n^MXXq+i&wl^2fh{!{n`PZyliP}@QwBDLU&}m_%Rz5* zib<(8R(C#{;ECwch*tDE&F8N8GimJAqi@3f-_TYQ{4lx%<24Auqe0av6$B7uZ20Ay zBt%n*RsdRj*A4X|uQq~4W4TuuHbK{G9uD~X!hADmgj#C}5b_E!=R?qZmV>I({!bl) zAw5N5LufRn`v7_}T9se6#b}AAh0-Qqh(cS)C}VGocjF;Qg$7F8U9DS~_cz`U@I6-v$#JvSRsHf=R&^8xxnXsh=Kp{1Z3^0pe4 zVMxKPq~SGkyQ_4RQRLjGKeXJ!1XuN?W`l+uic!Dl%HONlcUF zg)DD|H<39ni8CJ-Q-#t>3rKiEFpQu9(-dXGE1|;J5c?;e`rLtV5FrRZ147t{|(mGU3KOMRCcIjGtZ_-uqcpik6|xQmmplh)MLP zC1rg%gWjtY*91R+C4kCD4=$B9GpMUzyjp1uDB(peBW`yU^<8pcJCR8o;*ExQZ zn|j3Bz0Qwsw&?9QYv;=eFw{NKr%q4Ai{@Erj*+##pVNN zBv@s)Ri^!Q*~GVrcE{)T1Q}Waup3g6zA1PMX~74?xkRB3{cP+re=4TMXQ9p`AIIc} zF^v-sP#U+?hZUo1u5sw6nkE~pQn_%Rd=NIcn%X+4Z@+wG2Vbe;XC$~c27O?dZ!8du zQ-;ky24?2ruNaw&(2y{@Txtc+lVTV^PNdnMRO^zPUCmIn$aHR|WydYBy%wvm8_7{V zJshXNOw44OXFK$=8(+Qi;BXHkgY#u&?OL+4@+wv=S1MC%VkfkwSyeOWmI9=3l1Pzy z>Dz2BZb_q$W*efAZ_kqZLw_5kYcz22FB8^jG&aN#6cFfZ&C3iWj)lY(DjF!@rPdEp zGN9cDr*xrK&EYs|!)lS)8rHFqgwbuy_pkmbAjmP$R2|H{XX;s5oIfdz090Apfi}=j zPEPS+EH%#TT~MTH-_#-v1!;YGWX8!s{~%0*eJ}>2x6Ec4uFhMiY+ z^3D_zAZImJkh(}$&4KkU8(#c1LHKL#a);!_y~g4VYKd&4ml&wl1{X!sh)3s)7#7v#SCD=3cZf_kHA*mz^|W4D(8bJI z-N8s`l_r}netl#k_W25AfBl`^*}LnLs7p>6;d?(Vo-LNVdaY+W1Z;^b191YrSbHyA z`15#{J_O%JLsM5}y}5mPw10Rtkt6JN(D4XsHW6N_!mNKZ^G&>smh{nc9;Z&enV%Ct z4jnixhbBNj1gyN7$g{bHmGvwY#msEPVvL+Gf3Gi33KVA*+d^5*Z#lA8rtsoQi}^6! z3B@-uZ=om`!h@u+j7qRC1)nwG+%DR$-Lsha>`iY-Wbqs{TR%nZRq!SLwrqu32sp`# zscD72%g;#Bs&4Q3Q{!llQtahD|DWE0xS5k|y80_g0!Rf&0#5FoNmsn>k9X*=mJWX_ zA%Wp145Wo2yNEqp-|57bZz%g=DbT*Nc_{!MjN{}{F9RN8&sgwgCJurX}c*V z^hL(eBdSlGf;>Yn%j;s!9p|R>q-Vi7Oq%6^1E-_FaB=z+l*}8x`i=heReSoCvAE6S zU~dPu>5#>BO}keMR6?Se%r=5$3gxJ_;T9xw3G#3H|C>pS2dqVP(_}}UWp{H=-dTiqr z!7uE97^VpWIa#S-P7!en4_eUW)3kKQmC4o-5ywnF(w5~QHHWg2Kc6kHV1V|8=#$x^ zuZ(F|DmsR5;;Ka=cHP4?iiT}p8k6@XThm#n03M{BNm5dTd011Ql#)1&&gS*r#FXu$ zwmAB`=?c^TQ9qE@_d)iGK+`5)Q#+vJ zmL5#@Y5vfw1~nxr=fA|K5ZQ54p$q4#9vG37?!$C{0DK-qXfE`@dyvd!4yQARiu-i4 ztw)!Wg>la&&*%NS?6GOCb5R{ExLg4cTduj@9BZ$)N<>tsXFG8lihTd!!ucSm8sMr~ zJ~P0B3>YnQzkODNi1quYKCmqRtGoot3LDLRLq3BG=0$F|1qj**_L?x?_WI>A{SF=M^?-atuM>liABsMF| zhI?0TP=fU1SP|=-0Nq?tQYNqH8##-2(6-e2?Z>R?shVZI$I~=tf<*O~M*9{OM%e9G1NMUAr)^(B?t2`mBt58m_+ z9t(*v-5s)VA=TB6YncwWK6SX* zLwt8#2WSSf+84us&H+r2a6hU9u(*&c050_ubnZSicP%=62!L&iZi)O=Y5BN>anQi5 z8rLWr9Q=Ma>a1TGxp}y#Xxj|usI%5)HC_j~hj5GyWo5LB{;YmZrp9lq+rjyNYZgR& z3kcw*xSr)Jduz5a-iZ6L+uAO06|5>dWY7p{Rw#0rIAZhXpfRn`aU~HadCrZZqK;95>Zz;e-NBi%RuWJFFYSqiPKD}9FA#cWw zrN@jan>A>&{;Nkux%hZId=24Kaya&2q(GYI4m18V=HrdwKg7dS9U`m`K?d2Vu_ zw&*`r5M`)F>XlJJMTnjGok;=D&;CRReOps)5#r4!Ay9&RMzAPXOk$@d(<;oW$&}AN z91_zdk8f%m?eMnQ98g&S;L+&)C)6Kj&r^AZ#T6lqQ{UcjWQ9$9d!6|`1%q%ly&lyp zY5HTppG?9`TSjbNi^{%F&m6$&s$X0n;Ym^EKa;vft2G_Wf4u@G%x46GDqqp=O!mb- z>*M8Po+MYQjlLQG5*AvW!WR%)Eur`|I<$t%LgR3y!s@AF%K%y{uRDC_ftY*shf zPCn;BnE*k`N=JF-TJj8^Gy{Cex)Yd^&K_7;TY%5E zzVZ5~0*b^SyW7lUjzZyBaK)GF^7(|z-CGO=PtbslxljkAUGB7ww+`Y6PlrbX?Xz9s zAwSQZTr~GvpD`6HHSElvFA2L0f?8BrlwN0jWT@7BU9!ldY#^#QR-YawgtzRL&T5z( z+j92vjvBl=qT^fjKR7twl_rLqr?Q8~wsKqM^9i|_Vgr!+*Dk2q2y;?HC)7iSeYnb3 zE~5!KVV0?2$!ZC&P30(%6@uRjpr8}cImghswNLf?@u*`;_bq^LLxsa~n6%X`o?Xhw#Rg_b$5$OtG;!+mf*+^Gso0r!!`g5H4;;8R4?gwg0G30f$~* z+Fq7Y-&I#TQ=26I8HkHAr)2>@!lCHqfzxG~+af$L;%(1H*6sVcl2{yzCj~LAMwZ3) zhiPau-~49R*_Ra?x(sE^ZEN$?AL93A)v;)H4qZY~5LTGU{c|LhuA0|y$X6+vd?5F| zO5R7Z5iw5PXdtO^W-VHIL+%heFH4UA;BVRM!WjH!my>eQzP{YU(QBszV;0M+5GFD5 zB~71}!2KM*}Y?a21L9z0<`%|e}A#@`gX#YXsMpK~^`j@PN%{lX8h+|jUrVk+l zX%8R$7B+Pg{+6}11IhRLy-u5i(Lk#KmUy*?8PbcyqWdm{LEMCI*R-#lKPT!%{uNMy zA7A5i?ie&ipcq1dnqPhhUdF}cHjT#;;w7hQ+-(#?#}0M#(1XHLfgNrw;0M{vGR^uFYgn2gf6YUP!{!U0z#75igu;ktKB$#i^`cQt1AA$YeZ&=l9=xHw(nz?66gotmbxMk{vc2=U9%0# zkdl?dMeXTDr=|4^G;xJ0=*BI)R@`UIZm%DVG%6T6UC|%^ml?uX=M!*KRH7Niq$m7` zHmNM3cV+8xtKhCy1eJOCzW29meYaofChc?O>25|?ZzfJ*;fJN%>;e}XZUz@D+7pB{ z*+^siNWByovP*KwH>nuD%niy|MLSPIVSrA#jK6)WKTZPAK$!xV=RuM0qHJg0k!&U$ zqPhhFyhQ$u5H+Z=<9OJ;JN1uy1?nF~wae1&n!Qzq+S0T%5zbWG=OGVi@z>ZU-TBPX zhgy5NVAoJqOE{l%Tc*yZJB-v;L`G@{0aE9EAr`zit5H3xj{u0p0ht)~g))6z(1G4S zi2S&NChR(a*QZ;Pn)c&h9)36_?E8JNcjc+!GE35>-8z1Ax(7UShO!^?Ot7p&@{9fv zs!_6wKf&JHjj5^iQjX2zgnm5NKWfxJ#8>eDq`A99xLmtBgqxiFMTPP4l6<}u$iICY zV{bcKYps7j1Or1{Xc(;8MhO})0eiUUgl_#XJ*wNH=2bo-VkEX+ZMt-mnzGO$!gGqD zBa@yG&no42(hPyxQuYzHz_S>1wkRg?fag@Jv4AOG@dIY?=|02%D(12@V2j zVYOxo=7&VplwBJBOv_3MgGI6J=1wdUHhhQ=zxs9Fljr3|wayyopHTwz?cXmS21>3+ zYITPJ3ozGW3Zno#H!U$scZo(La2!lku%e(Dt5&)~s_=FAqAut&~M8Aff} z^5MRln*I1Sw?6&Atf|%Gex))@tA*$Z-fb07vwV@IOM?JO1ekQQ(Ue|t1lnc)`l>iy zG$r|!%CSx%weTvwulqRe z_f#*nkB@fV&;Jgz_KkQj)VJEl$JFD_q3xMdi~dUu9uy13WBsm3ETd&J&abg8AV8;z zQgv2t-))RdyQlOejmnx{PG5FqzA~FYV3z3x!C?Q_^!f;ItK;ZO<5|!>oMi~yiS)+V zLxFDS?W2PWiLJLR=SuX4LJ!+>_w_B1&X|Ju3cw)Z90H}u5r^Se8SAK)tBxw7Wf>|U zd%P;{pVV@Dd#5b2yti8JqEQ>f_8gT=^L1nUau9lS$(84|5_cn!3B0RCVW&DvW>zEY zB!cdFyfy)?H1xUm!hS_GuESV_f;1v?8bahPL>T!P1aVa_q!|1;^&OF-K-4%DiVR>z zNdYIp;zMfjWlX{Gp}-)@B^zi<3Hu0~X ziInR9vGUo^x2=9ax1ra1{kxOn^ze$4UiE>NvUJkltYprV3Pt8Smv!H12hdEqSIuik zhs))*4lhRU9(b?KnH4J|Qz+P&JER8QenV@7A-WR}ruQrVcLIP7l&o9K_bK&Ka@}A_5kC5V_7B*it{cFU!`+K9cJ3 zw;=}U@?8lt`u=X15~kNUSzgk5@S$E;1UYy_!hG=~Uz&nyWZ7`6B-px9j^x2 z?k&;qY=OI7yXbcWf;NI1nacf@Jd~b}_BL}I{dz^P9R{Afl})Pv73HPQg(mQyAdsg^ zvBo5+=5FahyRh3T$*JIzFbL2z;= z%Nqw)N@mQa^;##4L(2x?t$xn28rD6)A&Tz2DBj5|NjAN}v`DtIk^f__I9HqK0+Xn8 z2sg|Lv)&oNTcY$tu325U{%-xG#NAA^1dRUzdT?GLPkv|oiZp#D(!7*Wgyxh__QKBx zg@;}|1>9KlI;Tc}EaAOr^G6EClWX*Tto*ueU6MCX_TcaL+u=VNGUVxhrQUS-bTtz> zuyqy}6k>{L$)1>Z)zNKle+{tq)_p8`q^<04aL*Jl*gbL>$7qhvKWnR7jEQuJ`j{mF z@O^DOODRbOLaQj;slbe^xaG$Ug7tyYI=(`ts?mpj<~cdHzmI!^D%fX1hX38jFNQSw z1CAi$w=Pq9u=K-p;z-@SsS&XV?Pw;E?9Sl*2fNA7<|P~^n!P1E%SxPcLuziy-haif zK&3IE8ikjZ*^00Cc{)hM%DEx(m{RKDP#r+ZGhdJV_~7@zWM$P(Qi>WAWEtVG$iFI) zb&z*XKStueci!T{Jqu_f^3Q6|6{XgEftN3|MrDck(1>atJjpfl(l^+MQ!`Vc@V?@MWf z)rW611~GPY%|xESfCJPAATIeJhezk-akPl6%+ImV!{;1@#J5Us)^XoVW0cFCTfgNz z4Cl|K4bM4~e#brq@uv_t0y)j9+ZJaP50aHwG%f0-HE@;U{R+>*Zt87v&Uol(;mh#4 z9g|afMt2_!e>$Bx(0>n?Sn3NX*ko^ne%Jz>XwyLPKhh*_?nZWc)0x&N`hQ7da@|d; z;}NS2vMGBlcI0wyn7_-_lnhkf3ps|54OE{EhEZ9&>Mq0$yi(> z^V2Mx!$$kUg5L&rm;?kSgiRu~n$k*CYw0aE;P z?hM-IW>KaiwgNu^9qz;*LXO7~IbBg2{ipTc=(S@Z<+Z^~b1# zXF!H#a#%$>REE_F?@CN$z>RKG_qtqyxFo^4n+Nf;JRA&5_t*c80;H&($=n5w#-gIfN+G~wWJ;|i zj_VIxORR(Bz!K9c;8ki~QVjA92w|((1x=41txwPAm@lwZyhuq^B4&^zM}u9)Naz(= z>aqM%G^=mX4aDmU$Jn-M8)LWS95sw;hF0PWX6dw*V4LC>IN2vR%uP$Rj5|cT!2P=B zyf5p{Er{j3?L4pOD2TnYuXpHH%_NP*8o<_cDp4VbZ}6h(#cZ&p$LdUDxuSocBj)l6 z?i3J&Ak1xv<>fqhL6eagJ{K5SYW!^)8|<}N&Y)>Fc19K>1n)x6isxy&=R|eD^Yh!8 z>Ue1Kc8=~4wpu8a+{@cfnQY{-&hfKIyj3rjq*zNvB;_M3;aVy@)>PDh1ouXLqH_=V zUV3brI!{Ld+OZ?1lfYUtDWz=CKw%-B`4_Hh?)R0%p;ce;)c^FWEpdDvl507nWew|R zW6sjIxtZ!1A~SpU5b;|S<~=6e3*?lTHygcOJs(O$TQ!>p^z5lh5C!Z=enKfD$URc7 zjS~9^6Lv#SS|i?xXie|x6BVN+@8Ow1L7E1_^k*fx^BEUib%>lp?{W7+lz|!tdw|Ro zb~5xKzv_iXTtZL2fM<(IVBgx1aD?!;-t{0W#|@q(T{BLHSwxUoS} zsM-FWw2eO%m`1%od+x4@ZxPf1-!}H${lZ^77>+wf!#e)~9M|x=W#xE|s)<`#m&y6P zV=6wCgq((od_wBQ<#reJRoTm@J&?EPLbHAhB`}Y=eEclbeQQae2h&{T8fBgvid`TbamT=uiFX77@?8^>#jao4$DoRzOI}wOPc5D%uP@#%F$y_CL!8zvkn5s5X1=(K=M6vxqodnxZjEF1f2iu#n~#W z6BXL;t18qV6^Ix+oHXZ?Pg0?Hc&Q&F1GMa7AX!guwX^c4FvP`dCvOl%2Gec(9cAtS zLkZQ|;SxjUH*-5&)AX#d&u#K#7EvUX?C(JOErM|LU&XLxFD zu8k$h^PYL6WGvj(jEB8iVQx~FTdc%|0x5x-2&2_FGbAT;JvaS{JM)eN|)^dGZOUY%Usyv=9$z~}WRF4^yVQ$lcRDPa%r?nMj5aS8&i4gY7 zdz%%dpLG|D_;iotIYr@yx+&bu;FRk!bo+E#ZO@*YkSpc`zhG2}3Q0cI1|O-M)1>X! zZ&e)=_i$I(hEC2E8Zq>0zZfs61jrG8-^AP>C929IA%G+jG(3mjzO1RWWUV0 zo&`22Qgjj5kZNBEwK(xLsPCyY|ISmOWqvN?T{7Xr9LQ;bWn;4Zv5gbOuq9A8d0EQQ zzJ>gRx9%u%bAc4Et<3V!$x3QS_1H3=yc39lyoBGtjs4FZGT%nQY@AOnw~?9DxAmb< z?8iv$uk5s8!FKmEH=rZn4R1{+fMyscSB!43zHGxn8|KB6Iw&tt zN5C`5X$13M|IRwO)7Lxdti@{ij}~?_K6?!b%_tse4(;zcYv6ug++HGeR%b3)CP9WG zW#I5M%A{kowc9~`91x64_K|F*XsfkOPfB|^2&%-g+JsSEIPi}GcBxco@+y{@^f>Sl zDPwR(&s$GME{yzxgN}L?%*U6QbS(b;=88@u(a36>n}K;C^vPKyNXo88-dF2lJbVO?0a_y5#Q}tfLYyDkKb~ZXG@4w6=xJL)~ zx}*mjJD3HTftYf758M^50P%06<|N)`mVe>;N9eK~m@H+kFdZ^x;l_j40Dx9)^%;iL zcdKd4IeF-aw3q90{E-NAWJ< zJls{!dP4?;7@9+ujp;SPn zt*`4EER@prPOc@hBI#a6h4!qC)%slcNu|RB_W*Rnnn&aNa(`ND7#UyhCslkC{2gY! zhA?-VaX1-07ph~1z~hks6`jC%y~O6#aIr7H~uZ_z0sz7$K#aJJu3s!}rN~4@jEh1_1Vd>tw&rJCl0=MchYmJbSzV zmOxa#lSO!BACRct0VHU5*X&y0vjDJY5v#COO1UluJyN~RFM?+}bKaQj{CPV@wiNB>E{%hJK z>25EbhzZdpy|#6`Mfqq0mFi+l^mxFLtX;!fcCmacLuRcVnHy=TX2ja%1N7&@zIiX< z1ooyTs^6J zv5O0jVFnLO;+7if`k2*%RpPwyNVq2=^p%fg60E4W#|_j;o6x4;qaDBJvCD86U8^Ly z;(IUAPlE@Casb9Ip)#cQs#muPEedCo#Ad9hza?RTDt1KM3L(-6y;Xn#b#2f-Ve8JZ zkUPx;@kSUQ{lC{3=#@NnJ02>>*k*(g%9XMgh7ub>Qn*eKjQ&jmb3obPPKn5j`#JH~ z{y+Th0Vo(A*#~ZhTuUwOan(!Ih9(=wvVvk{d#@22f)&b%uEh*)tQcj;C-31!^C8U% z8*qeA9l&#as1Y<@${}S$ije0yaj)5_`i$PG^;X+@Fdw=AfO0^NKy%9^s6Xx!|L(t6 zNb!j>Aq9$}!#a{pgQPHh82eTHp<*Od=?#CorE%)K75y9FD3*Z2lzOOrWa>pTnz(+|0>1zlU;ehVu zX*lKdvN-6iq4ZENx`I>4TN>6PN1^dD2aw0H<;0O)>8*sYR z2mpL0M`XKMG&KjPVPBT%y&f`8e6RNIm7cxM*yRQv=fk>_J1T&Wv17t_U*WS}w?f@S|v$)0uZJ2R@X^*$wV z(cT!gM)zF1&{hSDIQ8TNg6(DpZvHzK@E;~vweCUvgG~Be*pAV1Tl|;OWiNOmfeGch z$)1VmCvlv+g_X2(+P}W*2l7^yWmvQDz~+_3F4eG(`DZkiz^N5*w~sZN+(OOOk(U^W zI1PI)kGW#-kgMU}OjHfcnJQ?+wzW4yHmSC!+iE$h=m)yhyaD7%_xe2eV<8$|5&&aS zTB8KGwB`JJxm8i?ZIayzu3|yG$fS=yQ$DS!qBa1Opg3p=bmB+vUYcF}OCsThR)je( zRDgxg+BW1!4pa$B26?W8#v;5t_{w`qgsju5iZcV)eEIk*B)@qe9#3u!2jG(!_+x ztT^k55H>31BK+(v&kCZn-MVo;BOR(%v)Nj^hhhOsgO z-65&e6R__7EnerAjz`5FWImt`f5iabQ&)2zM%G%E$`<;a{KH%VYW0_PV_|1V=g&jc(;#T;jxwigkJT&BFRI-Ei`nqEUy{T5-qPlosk00%tY)O!?~X#Kk9NxhI4F9^!s@kfl-narkepP=^h3x@TRt zOr@iS?i=bpn8)FNkog74e9(n2F`z8azR=kj0?l}F0~>4#ly>-T^i?Y1fH#L5JfsN_ zFLye=hohgOkWNEnK}%JxuIn4&eHQ2SMWf@{@bV3y{rt~L62iZ;F^NNYK{}xzDZ$a5qulw!sYEFkH@ed9@$1`h1>EPqVz6 z%v*Zm7q$~R22#X}0k$D@HY82mispzHd#;?6@>+64)@w$Yni&}YV+8J13ANSRLK(?Z zngcC+7ocd?pJRKyaWCaHyc)rQy)0AXsOeR-!DJ%i=Tu?Jmdoylt#F^D?xgt?(3t4s zqJLOcBMC-;Fh5^)vuX&lOmUxLgLBL)IXw`f4A9MzkEn=Qeyut_(}7Ped>e}+sX5xL zK4eXwrw}CS218Rr>ZO)vbnmlPY+G>JE#CSk)%T5f5T4jZs%Dr7zm9OB$hGF)yr^C7T+q<#8*?VFi>YRAIfT3aF!t;L@`WN)=I+o_wdLktT}f>P0{XP(?Vyfi$#d?iS%iq8T{}^ttf81^&IY3F7W( zdrguNq7}mV)Cb_PUhrRQaR}_sbr9yEG2ZTP4Q0xYcYHoSxC3zyPuu5m7eFgb5aX5h zNGxrrp?|xe=od#TsS`nz?slg90-?g(7;LTe6l?AE(#N^b$Ujrufp;4#q@ktLHOS$B zdJN`d=Mc*%n(QWtGxvGpEIw!LF7)HJe7#W%PHq_o+0JuJB7NY%CGeq8ArkH8-T8wC z@h*P1XJ!qlxa&qVuS<(tqNV%wMZh_YBmTW5`{J7@B?$icv}FBS&(mKA+v%rc%InRC z&1ajHmrsF4hm?I!cPGM+-@%lgHJ(n==QnTm^K`H*)|}E~fO55jkQPeevraF*=4Jll z4v)JV|AHHtmYF-uT#CT(l6!M>!O7=8uz`b%9^Vl__(tyT$T4V$6kv`b;M+!@V5chp ztG=EJzx#A_brEE@*6DH{<8%tGbyK_u;q7S!;$y}l!U)19F_N98SR4C!CYo*b-%lS% zeH-H8{85f9VSoT=O*!@8=Y0Pp|0$&RdOg5=myB;0PA?6ay1P@k;W*tpojT52Kkc+i zC8obRS@Np8``jYf5T9mJbMAHhh%CGri$4SFW4ZOX9$s1#ynZ%`Sm2bC)W_g^`Ao%W zI9aXUinJG5WpAOBvnMx}bVZ}3bp2Qbv~;@DgF>O4=Nmp`62QO!n}ih}A3#E;TGlba zYd%|PR)MqEodIqialk^<$EjUV-UqN#Aj!4+lkH}so&Q(jjZVMs5D3K1cNP#ghpLsX zV>)!mK;mGA9JS%C0{@Z*n)v~d59z6Z}9tm}2*1E!6K<};JC z3E@+i@Ji|O-NV_Y?>E073F)LEx{jpOJ(s5}%4_rns+#W;4%)x6aW{97#>o3vP~?ny z*kB~8gK43qgcRkb>C0aJJ(KDqbx+K1Xxc*7clsXu9fkWi(n4I=fRme3(zKYaOm!ra z^wga>3fT{{M4a_PE9UQ{JE5)b8uk zY2cV}T6Za3_m5`o7VphnEr37IxHvfZSUI_XA5AXq_gwGZ^9V3=aJ=W>(59_q{l67( daQbLt>Gl6!fc_I+B2eI&oRqR;wYX{U{{m#vY{LKm literal 0 HcmV?d00001 diff --git a/doc/source/design/ref_arch_100_nodes.rst b/doc/source/design/ref_arch_100_nodes.rst new file mode 100644 index 00000000..e2a0ba79 --- /dev/null +++ b/doc/source/design/ref_arch_100_nodes.rst @@ -0,0 +1,1455 @@ +.. _ref_arch_for_100_300_500_nodes: + +=========================================================== +OpenStack Reference Architecture For 100, 300 and 500 Nodes +=========================================================== + +This document proposes a new Reference Architecture (RA) of OpenStack +installation on top of Kubernetes that supports a number of 100, 300 and 500 +compute nodes, using container technologies to improve scalability and +high availability of OpenStack Control Plane services. Containerization +of OpenStack components will also enable provisioning, patching and +upgrading large numbers of nodes in parallel, with high reliability and +minimal downtime. + +Introduction/Executive Summary +------------------------------ + +This document contains recommendations for building specific clouds +depending for different use cases. All recommendations are validated and +tested on the described scale in both synthetic and real-world +configurations. + +The proposed Reference Architecture applies the following open source +tools (among others): + +- OpenStack Control Plane is a scalable, modular cloud controller with + support for all aspects of virtualized infrastructure. + +- Ceph is a distributed storage system that provides all the most + popular types of storage to a virtualized infrastructure: object + storage, virtual block storage and distributed file system. + +- InfluxDB is a time-series database optimized for collecting metrics + from multiple sources in nearly-real time and providing access to + recorded metrics. + +- Docker containers are used to isolate OpenStack services from the + underlying operating system and control the state of every + service more precisely. + +Highlights +~~~~~~~~~~ + +Highlights of this document include: + +- Hardware and network configuration of the lab used to develop the + Reference Architecture. + +- OpenStack Control Plane overview - Details how the OpenStack Control + Plane is organized, including placement of the services for + scaling and high availability of the control plane. + +- Data plane overview - Describes the approach to the data plane and + technology stack used in the Reference Architecture. + +- Granular update and upgrade overview - Describes how the proposed + Reference Architecture supports updating and upgrading on all + levels from individual services to the whole OpenStack + application. + +Overview +-------- + +Hardware and network considerations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This section summarizes hardware considerations and network layouts for +the proposed solution. It defines the basic requirements to server +equipment hosting the cloud based on the CCP RA. Requirements to network +infrastructure in terms of L2 and L3 topologies, services like DNS and +NTP and external access provided in the network. + +OpenStack Control Plane +~~~~~~~~~~~~~~~~~~~~~~~ + +The Control Plane consists of OpenStack component services, like Nova, +Glance and Keystone, and supplementary services like MySQL database +server and RabbitMQ server, all enveloped in Docker containers and +managed by an orchestrator (e.g. Kubernetes). + +OpenStack Data Plane +~~~~~~~~~~~~~~~~~~~~ + +OpenStack data plane is constituted by backends to various drivers of +different components of OpenStack. They all fall into 3 main categories: + +- Hypervisor is data plane component backing OpenStack Compute (Nova), + for example, libvirt or VMWare vSphere. + +- Networking is multiple data plane components under management of + OpenStack Networking, for example, OpenVSwitch. + +- Storage has multiple components managed by OpenStack Storage and + Images services. This category includes such systems as LVM, + iSCSI, Ceph and others. + +Granular Life Cycle Management, Updates and Upgrades +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This document describes the strategy of updating the OpenStack cloud and +its components to new version. The strategy of upgrade is based on +containerization of all those components. Containers effectively split +the state of the system into set of states of individual container. +Every container's state is managed mostly independently. + +Hardware Overview +----------------- + +Server Hardware Specifications +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The following server hardware was used in a lab to install and test the +proposed architecture solution. For Compute nodes, two configurations +are used. + +Configuration One + +- Server model is Dell R630 + +- 2x12 Core CPUs E5-2680v3 + +- 256GB of RAM + +- 2x800GB SSD Intel S3610 + +- 2x10GB Intel X710 dual-port NICs + +Configuration Two + +- Server model is Lenovo RD550-1U + +- 2x12 Core CPUs E5-2680v3 + +- 256GB of RAM + +- 2x800GB SSD Intel S3610 + +- 2x10GB Intel X710 dual-port NICs + +For Storage nodes, the following configuration is used. + +- Server model is Lenovo RD650 + +- 2x12 Core CPUs E5-2670v3 + +- 128GB RAM + +- 2x480GB SSD Intel S3610 + +- 10x2TB HDD + +- 2x10GB Intel X710 dual-port NICs + +Resource Quantities +~~~~~~~~~~~~~~~~~~~ + +Compute/Controller Resources +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The number of Compute/Controller nodes in the environment: 100 nodes + +The number of CPU Cores available to hypervisors and control plane +services: 2400 cores + +The amount of RAM available to hypervisors and control plane services: +25,600 GB + +Storage Resources +^^^^^^^^^^^^^^^^^ + +* The number of Storage nodes in the environment: 45 nodes + +* The number of CPU Cores available to storage services: 1,080 cores + +* The amount of RAM available to storage cache: 5,760 GB + +* The total size of raw disk space available on storage nodes: 900 TB + +Servers are installed in 9 racks connected by ToR switches to spine +switches. + +Network Schema +-------------- + +Underlay Network Topology +~~~~~~~~~~~~~~~~~~~~~~~~~ + +The environment employs leaf switches topology in the underlay network. +BGP protocol used in the underlay network to ensure multipath links +aggregation (IP ECMP) to leaf switches. ToR leaf switches are connected +to spines with 40GbE uplinks. + +The leaf switches use VXLANs to provide overlay network to servers, and +MLAG aggregation to ensure availability and performance on the +downstream links. Servers are connected to ToR switches with 40GbE +port-channel links (4x10GbE with MLAG aggregation). + +The following diagram depicts the network schema of the environment: + +Figure 1. Network schema of the lab environment +''''''''''''''''''''''''''''''''''''''''''''''' + +|image0| + +No specific QoS configuration was made in the underlay network. Assume +that all services share the total bandwidth of network link without +guarantees for individual processes or sockets. + +The following models of switching hardware were used throughout testing +effort in the schema described above: + +- Spine switches: Arista 7508E (4x2900PS, 6xFabric-E modules, + 1xSupervisorE module) + +- ToR switches: Arista 7050X + +Network for OpenStack Platform +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +OpenStack platform uses underlay network to exchange data between its +components, expose public API endpoints and transport the data of +overlay or tenant networks. The following classes of networks are +defined for OpenStack platform in proposed architecture. + +- Management network. This is the network where sensitive data exchange + happens. Sensitive data includes authentication and authorization + credentials and tokens, database traffic, RPC messages and + notifications. This network also provides access to + administrative API endpoints exposed by components of the + platform. + +- Public network. Network of this class provides access to API + endpoints exposed by various components of the platform. It also + connects Floating IPs of virtual server instances to external + network segments and Internet. + +- Storage network. Specialized network to transport data of storage + subsystem, i.e. Ceph cluster. It also connects to internal Ceph + API endpoints. + +- Overlay network. Networks of this class transport the payload of + tenant networks, i.e. the private networks connecting virtual + server instances. The current architecture employs VXLANs for L2 + encapsulation. + +There can be one or more networks of each class in a particular cloud, +for example, Management network class can include separate segments for +messaging, database traffic and Admin API endpoints, and so on. + +In addition to role segmentation, networks in one class can be separated +by scale. For example, Public network class might include multiple +per-rack segments, all connected by an L3 router. + +Figure 2. Logical networks topology in OpenStack environment +'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +|image1| + +Control Plane +------------- + +OpenStack Overview +~~~~~~~~~~~~~~~~~~ + +OpenStack is a system that provides Infrastructure as a Service (IaaS). +IaaS is essentially a set of APIs that allow for creation of elements of +typical data center infrastructure, such as virtual servers, networks, +virtual disks and so on. Every aspect of an infrastructure is controlled +by a particular component of OpenStack: + +- OpenStack Compute (Nova) controls virtual servers lifecycle, from + creation through management to termination. + +- OpenStack Networking (Neutron) provides connectivity between virtual + servers and to the world outside. + +- OpenStack Image (Glance) holds base disk images used to boot the + instances of virtual servers with an operating system contained + in the image. + +- OpenStack Block Storage (Cinder) manages the virtual block storage + and supplies it to virtual servers as block devices. + +- OpenStack Identity (Keystone) provides authentication/authorization + to other components and clients of all APIs. + +Guidelines for OpenStack at Scale +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Currently, OpenStack scalability is limited by several factors: + +- Placement responsiveness desired + +- MySQL scalability + +- Messaging system scalability + +- Scalability of the SDN + +- Scalability of the SDS + +Scheduler Scalability +^^^^^^^^^^^^^^^^^^^^^ + +In general, the scheduling strategy for OpenStack is a completely +optimistic determination which means the state of all things is +considered whenever a placement decision is made. As more factors are +added to the consideration set (special hardware, more compute nodes, +affinity, etc.) the time to place resources increases at a quadratic +rate. There is work being done on splitting out the scheduling and +placement parts of Nova and isolating them as a separate service, like +Cinder or Neutron. For now, however, this is a work in progress and we +should not rely on increased scheduler performance for another few +releases. + +OpenStack can seamlessly accommodate different server types based on +CPU, memory and local disk without partitioning of the server pool. +Several partitioning schemes exist that provide ability to specify pools +of servers appropriate for a particular workload type. A commonly used +scheme is server aggregates, which allows specific image sets to be +scheduled to a set of servers based on server and image tags that the +administrator has defined. + +MySQL Scalability +^^^^^^^^^^^^^^^^^ + +As almost all OpenStack services use MySQL to track state, scalability +of the database can become a problem. Mirantis OpenStack deploys with a +MySQL + Galera cluster which allows for reads and writes to be scaled in +a limited fashion. + +However, careful tuning and care of the +databases is a very important consideration. MySQL is recommended to run +on dedicated nodes. The Control Plane DB should not be used by Tenant +Applications or other solutions like Zabbix; those should provision +another MySQL cluster on other physical nodes. + +Messaging System Scalability +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A majority of OpenStack services use AMQP implementations (RabbitMQ and +Qpid) for message transport and RPC. As with MySQL, there are examples +of installations in the wild which have successfully scaled this +messaging infrastructure to tens of thousands of nodes. + +Services Overview +~~~~~~~~~~~~~~~~~ + +The following paragraphs describe how the individual services and +components should be placed and configured in the proposed Reference +Architecture. All services are divided into two categories: stateless +and stateful. Each category requires specific approach which is outlined +below. + +Stateless Services +^^^^^^^^^^^^^^^^^^ + +Services of this type do not record their state in their environment. +Those services can be killed and restarted without risk of losing data. +Most of OpenStack component services are inherently stateless being +either HTTP-based API servers or message queue processors. + +General approach to stateless services is envelop the service into +Docker container with minimal to no configuration embedded. + +Stateful Services +^^^^^^^^^^^^^^^^^ + +Services that keep track of their state in some persistent storage and +cannot be restarted in clean environment without losing that state are +considered stateful. The main stateful component of the OpenStack +platform is MySQL state database. Most of other services rely on the +database to keep track of their own state. + +Containers that run stateful services must have persistent storage +attached to them. This storage must be made available in case if the +container with stateful service should be move to another location. Such +availability can be ensured on application level via replication of some +sort, or on network level by redirecting connections from moving +container to the same storage location. + +OpenStack Identity (Keystone) +----------------------------- + +Keystone service provides identity to all other OpenStack services and +external clients of OpenStack APIs. The main component of Identity +service is an HTTP server that exposes an API of the service. + +Keystone service is used by all other components of OpenStack for +authorization of requests, including system requests between services +and thus gets called a lot even if the platform is idle. This generates +load on CPU resources of server where it runs. Thus, it is recommended +to run Keystone service on dedicated node, althhough they could be +co-located with other resource intensive components. + +For availability and load distribution it is recommended to run at least +3 instances of Keystone at the scale of 100 and 300 nodes. For scale +of 500 nodes, 5 instances of Keystone server are recommended. + +Load balancer with a Virtual IP address should be placed in front +of the Keystone services to ensure handling of failures and even +distribution of requests. + +Per these recommendations, the recommended number of dedicated physical +nodes to run Keystone service is 3 or 5. However, instances of Keystone +can share these nodes with any other services. + +Apache2 Web Server +~~~~~~~~~~~~~~~~~~ + +Apache web server wraps Keystone and Horizon services and exposes them +as web services. However, since Horizon can generate significant load +on resources, it is not recommended to co-locate it with Keystone, but +it is possible at scale of 100 to 500 nodes. See below for recommendations +on scaling Horizon. + +Figure 3. Keystone API servers in redundant configuration +''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +|image2| + +OpenStack Compute (Nova) +------------------------ + +Nova has multiple services, not all of which are stateless. API server, +scheduler and conductor in particular are stateless, while nova-compute +service is not as it manages the state of data-plane components on a +compute node. + +Nova API +~~~~~~~~ + +Web server that exposes Compute API of the OpenStack platform. This is a +stateless service. Nova API server consumes significant resources at +scale of hundreds of nodes. + +Nova API server should run on a dedicated physical servers to ensure it +does not share resources with other services with comparably big +footprint. It can co-locate with more lightweight services. + +For availability reasons, it is recommended to run at least 3 instances +of API server at any time. Load balancer with Virtual IP address shall +be placed in front of all API servers. + +Figure 4. Nova API servers in redundant configuration +''''''''''''''''''''''''''''''''''''''''''''''''''''' + +|image3| + +Nova Scheduler +~~~~~~~~~~~~~~ + +Scheduling service of Nova is stateless and can be co-located with +services that have larger footprint. Scheduler should be scaled by +adding more service instances. Every scheduler instance must run +with at least 3 worker threads. + +Using multiple scheduler processes might lead to unexpected +delays in provisioning due to scheduler retries if many simultaneous +placement requests are being served. + +At the scale of 100 and 300 of nodes, 3 instances of Scheduler will +suffice. For 500 nodes, 5 instances recommended. + +Since they can be co-located with other services, it won't take any +additional physical servers from the pool. Schedulers can be placed to +nodes running other Nova control plane services (i.e. API and +Conductor). + +Nova Conductor +~~~~~~~~~~~~~~ + +Centralized conductor service in Nova adds database interface layer +for improved security and complex operations. It is a stateless service +that is scaled horizontally by adding instances of it running on different +physical servers. + +To ensure that the Conductor service is highly available, run 3 +instances of the service at any time. This will not require dedicated +physical servers from the pool. + +Figure 5. Scaling of Nova Scheduler and Conductor services +'''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +|image4| + +Nova Compute +~~~~~~~~~~~~ + +Compute service is essentially an agent service of Nova. An instance of +it runs on every hypervisor node in an OpenStack environment and +controls the virtual machines and other connected resources on a node +level. + +Compute service in the proposed architecture is not containerized. It +runs on the same node as a hypervisor and communicates to different local +and remote APIs to get the virtual machines up and running. One instance +of Compute service runs per hypervisor host. + +Services Placement +~~~~~~~~~~~~~~~~~~ + +Nova control plane services is spread across 3 nodes with the following +services distributed evenly between those nodes: + +- 3 instances of Conductor service + +- 3 instances of API service + +- 3 to 5 instances of Scheduler service + +Load balancer is placed in front of the API services to ensure +availability and load distribution for the Compute API of OpenStack +platform. + +Compute services run per-node on hypervisor hosts (compute nodes) in +non-redundant fashion. + +OpenStack Networking (Neutron) +------------------------------ + +This component includes API server that exposes HTTP-based API and a set +of agents which actually manage data plane resources, such as IP +addresses, firewall and virtual switches. + +Neutron Server +~~~~~~~~~~~~~~ + +An API server exposes Neutron API and passes all web service calls to +the Neutron plugin for processing. This service generates moderate load +on processors and memory of physical servers. In proposed architecture, +it runs on the same nodes as other lightweight API servers. A minimal of +3 instances of the server is recommended for redundancy and load +distribution. + +Load balancer working in front of the API servers helps to ensure their +availability and distribute the flow of requests. + +Figure 6. Neutron Server services in redundant configuration +'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +|image5| + +Neutron DHCP agent +~~~~~~~~~~~~~~~~~~ + +DHCP agent of Neutron is responsible for assigning IP addresses to VMs. +The agent is horizontally scalable by adding new instances of it +distributed between different nodes. DHCP agent can be co-located to any +other services and can run on hypervisor hosts as well. + +Neutron L3 agent +~~~~~~~~~~~~~~~~ + +L3 agent controls routing in Public and Private networks by creating and +managing namespaces, routers, floating IP addresses and network +translations. The agent can be scaled by adding instances. High +availability of the agent is ensured by running its instances in +Corosync/Pacemaker cluster or using orchestrator-driven clustering and +state tracking (e.g. Kubernetes events system with etcd). + +The agent has low resources footprint and can be co-located with more +resource-intensive services, for example, Neutron Server. + +Neutron L2 agent +~~~~~~~~~~~~~~~~ + +L2 agent manages data link layer configuration of the overlay network +for Compute nodes. It also provides L2 functions to L3 agent. The agent +is specific to Neutron plugin. + +The L2 agent used with OpenVSwitch plugin generates high CPU load when +creates and monitors the OVS configurations. It has to run on any host +that runs nova-compute, neutron-l3-agent or neutron-dhcp-agent. + +Neutron metadata agent +~~~~~~~~~~~~~~~~~~~~~~~ + +The metadata agent provides network metadata to virtual servers. +For availability and redundancy, it is recommended to run at least 3 +instances of the agent. They can share a node with other Neutron or +control plane services. + +Services Placement +~~~~~~~~~~~~~~~~~~ + +Neutron control plane services run in batches 3 for redundancy +of Neutron Server. All micro services of Neutron could be co-located +with the Neutron Servers or with other components' services. + +Load balancer should be placed in front of the Neutron Server to ensure +availability and load distribution for the Network API of the OpenStack +platform. + +L3 and DHCP agents are co-located with instances of Neutron Server on +the same physical nodes. L2 agent works on every Compute node and on +every node that runs L3 and/or DHCP agent. + +OpenStack Images (Glance) +------------------------- + +Images service consists of API and indexing service. Both of them are +stateless and can be containerized. + +Glance API +~~~~~~~~~~ + +This service exposes Images API of OpenStack platform. It is used to +upload and download images and snapshot. + +Glance API server has low resource consumption and can be co-located +with other services. It does not require dedicated physical servers. + +Glance API server scales by adding new instances. Load balancer is +required to provide a single virtual address to access the API. The load +can be evenly distributed between instances of Glance API. + +Important parameter of the Image Service architecture is the storage +backend. The following types of storage are proposed for Glance in this +Reference Architecture: + +- Local filesystem + +- Ceph cluster + +With local storage, the consistency of the store for all instances of +Glance API must be ensured by an external component (e.g. replication +daemon). + +Ceph backend natively provides availability and replication of stored +data. Multiple instances of Glance API service work with the same shared +store in Ceph cluster. + +Figure 7. Cinder API service in redundant configuration with Ceph back-end +'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +|image6| + +Glance Registry +~~~~~~~~~~~~~~~ + +Registry serves images metadata part of the Images API. It stores and +manages a catalog of images. It has low footprint and can share a +physical node with other resource intensive services. + +Services Placement +~~~~~~~~~~~~~~~~~~ + +Micro services of Glance do not require dedicated physical servers. They +can be co-located with other services. + +For the purposes of high availability and redundancy, at least 3 +instances of Glance API service should run at any time. Load balancer +must be placed in front of those instances to provide single API +endpoint and distribute the load. + +OpenStack Block Storage (Cinder) +-------------------------------- + +Storage service manages and exposes a virtual block storage. The server +that handles the management of low-level storage requires access to the +disk subsystem. + +Cinder API +~~~~~~~~~~ + +This service exposes Volume API of the OpenStack platform. Volume API is +not very often used by other components of OpenStack and doesn't consume +too much resources. It could run on the same physical node with other +resource non intensive services. + +For availability and redundancy purposes, it is proposed to run at least +3 instances of Cinder API. Load balancer should be placed in front of +these instances to provide distribution of load and failover +capabilities. + +Figure 8. Cinder API service in redundant configuration +''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +|image7| + +Cinder Scheduler +~~~~~~~~~~~~~~~~ + +Scheduler selects an instance of the Volume micro service to call when a +client requests creation of a virtual block volume. Scheduler is not +resource intensive and can be co-located with other services. It scales +horizontally by adding new instances of the service. For redundancy +purposes, the instances of scheduler service should be distributed +between different physical servers. + +Cinder Volume +~~~~~~~~~~~~~ + +Volume service manages block storage via appropriate API which depends +on the technology in use. With Ceph as a provider of virtual block +storage, single instance of Cinder Volume service is required to manage +the Ceph cluster via its API. If virtual block storage is provided by +using LVM with local disks of Compute nodes, the Volume service must be +running on every Compute node in the OpenStack environment. + +In Ceph case, it is recommended to run at least 3 instances of the +Volume service for the availability and redundancy of the service. Each +virtual volume is managed by one instance Volume service at a time. +However, if that instance is lost, another one takes over on its +volumes. + +Figure 9. Cinder services in redundant configuration +'''''''''''''''''''''''''''''''''''''''''''''''''''' + +|image8| + +Services Placement +~~~~~~~~~~~~~~~~~~ + +Cinder services do not require dedicated physical nodes. They run on the +same physical servers with other components of control plane of the +OpenStack platform. + +The instances of Cinder API service are placed behind a load balancer to +ensure the distribution of load and availability of the service. + +OpenStack Dashboard (Horizon) +----------------------------- + +Horizon dashboard provides user interface to the cloud's provisioning +APIs. This is a web application running on top of Apache2 web server. +For availability purposes, multiple instances of the server are started. +Load balancer is placed in front of the instances to provide load +distribution and failover. Horizon does not require dedicated physical +node. It can be co-located with other services of other components of +the OpenStack platform. + +For security reasons, it is not recommended to use the same instances +Apache2 server to wrap both Horizon and Keystone API services. + +RabbitMQ +-------- + +The messaging server allows all distributed components of an OpenStack +service to communicate to each other. Services use internal RPC for +communication. All OpenStack services also send broadcast notifications +via messaging queue bus. + +Clustering +~~~~~~~~~~ + +The prerequisite for High Availability of queue server is the configured +and working RabbitMQ cluster. All data/state required for the operation +of a RabbitMQ cluster is replicated across all nodes. An exception to +this are message queues, which by default reside on one node, though +they are visible and reachable from all nodes. + +Cluster assembly requires installing and using a clustering plugin on +all servers. Proposed solution for RabbitMQ clustering is the +`rabbitmq-autocluster `__\ `*plugin* `__. + +The RabbitMQ cluster also needs proper fencing mechanism to exclude +split brain conditions and preserve a quorum. Proposed solution for this +problem is using 'pause\_minority' `partition +mode `__ with the +rabbit-autocluster plugin. + +Replication +~~~~~~~~~~~ + +Replication mechanism for RabbitMQ queues is known as 'mirroring'. By +default, queues within a RabbitMQ cluster are located on a single node +(the node on which they were first declared). This is unlike exchanges +and bindings, which can always be considered to be on all nodes. Queues +can optionally be made mirrored across multiple nodes. Each mirrored +queue consists of one master and one or more slaves, with the oldest +slave being promoted to the new master if the old master disappears for +any reason. + +Messages published to the queue are replicated to all members of the +cluster. Consumers are connected to the master regardless of which node +they connect to, with slave nodes dropping messages that have been +acknowledged at the master. + +Queue mirroring therefore aims to enhance availability, but not +distribution of load across nodes (all participating nodes each do all +the work). It is important to note that using mirroring in RabbitMQ +actually reduces the availability of queues by dropping performance by +about 2 times in +`tests `__, +and eventually leads to `failures of +RabbitMQ `__ +because of extremely high rate of context switches at node's CPUs. + +There are two main types of messages in OpenStack: + +- Remote Procedure Call (RPC) messages carry commands and/or requests + between microservices within a single component of OpenStack + platform (e.g. nova-conductor to nova-compute). + +- Notification messages are issued by a microservice upon specific + events and are consumed by other components (e.g. Nova + notifications about creating VMs are consumed by Ceilometer). + +In proposed OpenStack architecture, only notification queues are +mirrored. All other queues are not, and if the instance of RabbitMQ +server dies after a message sent, but before it is read, that message is +gone forever. This is a trade-off for significant (at least 2 times) +performance and stability boost in potential bottleneck service. +Drawbacks of this mode of operation are: + +- Long-running tasks might stuck in transition states due to loss of + messages. For example, Heat stacks might never leave spawning + state. Most of the time, such conditions could be fixed by the + user via API. + +Data Persistence +~~~~~~~~~~~~~~~~ + +OpenStack's RPC mechanism does not impose requirements for durable +queues or messages. Thus, no durability required for RabbitMQ queues, +and there are no 'disk' nodes in cluster. Restarting a RabbitMQ node +then will cause all data of that node to be lost. Since OpenStack does +not rely on RPC as a guaranteed transport, it doesn't break the control +plane. Clients shall detect failure of a server they are talking to and +connect to another server automatically. + +RabbitMQ service considered stateless in terms defined in this document +due to the reasons mentioned above. + +Networking Considerations +~~~~~~~~~~~~~~~~~~~~~~~~~ + +RabbitMQ nodes address each other using domain names, either short or +fully-qualified (FQDNs). Therefore hostnames of all cluster members must +be resolvable from all cluster nodes, as well as machines on which +command line tools such as rabbitmqctl might be used. + +RabbitMQ clustering has several modes of dealing with `network +partitions `__, primarily +consistency oriented. Clustering is meant to be used across LAN. It is +not recommended to run clusters that span WAN. The +`Shovel `__ or +`Federation `__ plugins are +better solutions for connecting brokers across a WAN. Note that `Shovel +and Federation are not equivalent to +clustering `__. + +Services Placement +~~~~~~~~~~~~~~~~~~ + +RabbitMQ servers are to be installed on the dedicated nodes. Co-locating +RabbitMQ with other Control Plane services has negative impact on its +performance and stability due to high resource consumption under the +load. Other services that have different resource usage patterns can +prevent RabbitMQ from allocating sufficient resources and thus make +messaging unstable. + +Based on that, RabbitMQ will require 3 dedicated nodes out of the pool +of Compute/Controller servers. These nodes can also host lightweight +services like Cinder, Keystone or Glance. + +Alternatives +~~~~~~~~~~~~ + +RabbitMQ is a server of choice for OpenStack messaging. Other +alternatives include: + +- 0MQ (ZeroMQ), a lightweight messaging library that integrates into + all components and provides server-less distributed message + exchange. + +- Kafka, a distributed commit log type messaging system, supported by + oslo.messaging library in experimental mode. + +ZeroMQ +^^^^^^ + +This library provides direct exchange of messages between microservices. +Its architecture may include simple brokers or proxies that just relay +messages to endpoints, thus reducing the number of network connections. + +ZeroMQ library support was present in OpenStack since early releases. +However, the implementation assumed direct connections between services +and thus a full mesh network between all nodes. This architecture +doesn't scale well. More recent implementations introduce simple proxy +services on every host that aggregate messages and relay them to a +central proxy, which does host-based routing. + +`Benchmarks `__ +show that both direct and proxy-based ZeroMQ implementations are more +efficient than RabbitMQ in terms of throughput and latency. However, in +the direct implementation, quick exhaustion of network connections limit +occurs at scale. + +The major down side of the ZeroMQ-based solution is that the queues +don't have any persistence. This is acceptable for RPC messaging, but +Notifications may require durable queues. Thus, if RPC is using ZeroMQ, +the Telemetry will require a separate messaging transport (RabbitMQ or +Kafka). + +Kafka +^^^^^ + +Distributed commit log based service Kafka is supported in OpenStack's +oslo.messaging library as an experimental. This makes it unfeasible to +include in the Reference Architecture.. + +MySQL/Galera +------------ + +State database of OpenStack contains all data that describe the state of +the cloud. All components of OpenStack use the database to read and +store changes in their state and state of the data plane components. + +Clustering +~~~~~~~~~~ + +The proposed clustering solution is based on the native +orchestrator-specific state management with etcd providing distributed +monitoring and data exchange for the cluster. Cluster operations will be +triggered by orchestrator events and handled by custom scripts. + +Failover and fencing of failed instances of MySQL is provided by scripts +triggered by the orchestrator upon changes in state and availability of +the members of Galera cluster. State and configuration information is +provided by etcd cluster. + +Data Persistence +~~~~~~~~~~~~~~~~ + +Galera implements replication mechanism to ensure that any data written +to one of the MySQL servers is synchronously duplicated to other members +of the cluster. When a new instance joins the cluster, one of the two +replication methods is used to synchronize it: IST or SST. If the +initial data set exists on the new node, incremental method (IST) is +used. Otherwise, full replication will be performed (SST). + +Since all nodes in the cluster have synchronous copies of the data set +at any time, there is no need to use shared storage. All DB servers work +with the local disk storage. + +Replication +~~~~~~~~~~~ + +Incremental synchronous replication is used to keep MySQL databases of +members in Galera cluster in sync. If a new member is added to the +cluster, full replication (SST) will be performed. + +Full SST replication can take indefinite time if the data set is big +enough. To mitigate this risk, the proposed architecture includes a +number of hot stand-by MySQL servers in addition to one Active server. +The access to said servers is provided by an instance of a load balancer +(see details in Networking Considerations section). + +Proposed architecture allows to quickly replace failing instances of +MySQL server without need to run full replication. It is still necessary +to restore the pool of hot standby instances whenever the failover event +occurs. + +Networking Considerations +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Load balancer is a key element of networking configuration of the Galera +cluster. Load balancer must be coordinated with the cluster, in terms +that it redirect write requests to appropriate instance of MySQL server. +It also ensures failover to hot standby instances and fencing of failed +active nodes. + +Services Placement +~~~~~~~~~~~~~~~~~~ + +MySQL servers forming the Galera cluster could run on the same physical +servers as instances of RabbitMQ broker. + +Ceph Distributed Storage +------------------------ + +Summary +~~~~~~~ + +Ceph is a distributed storage system with built-in replication +capability. Architecture of Ceph is designed for resiliency and +durability. + +Ceph provides few different APIs for different types of supported +storage: + +- Distributed file system + +- Object storage + +- Virtual block device + +OpenStack platform generally uses Object API and Block Device API of +Ceph for Image and Virtual Block storage correspondingly. + +Main components of Ceph are as follows. + +- A Ceph Monitor maintains a master copy of the cluster map. A cluster + of Ceph monitors ensures high availability should a monitor + daemon fail. Storage cluster clients retrieve a copy of the + cluster map from the Ceph Monitor. + +- A Ceph OSD Daemon checks its own state and the state of other OSDs + and reports back to monitors. + +- RADOS gateway exposes the Object API of the platform. This API is + generally compatible with OpenStack Object Storage API, which + allows to use it as a Glance back-end without additional + modifications. + +Ceph Monitor +~~~~~~~~~~~~ + +For reliability purposes, Ceph Monitors should be placed to different +physical nodes. Those nodes might be the Storage nodes themselves, +albeit that is not generally recommended. Proper resilience of the +cluster can be ensured by using 3 instances of Ceph Monitor, each +running on different hosts. + +Ceph OSD +~~~~~~~~ + +Every storage device requires an instance of Ceph OSD daemon to run on +the node. These daemons might be resource intensive under certain +conditions. Since one node usually have multiple devices attached to it, +there are usually more than one OSD process running on the node. Thus, +it is recommended that no other services are placed to the OSD nodes. + +RADOS Gateway +~~~~~~~~~~~~~ + +This service exposes an Object API via HTTP. It have low resource +footprint and can be co-located with other services, for example, with a +Ceph Monitor. Multiple radosgw daemons should be used to provide +redundancy of the service. Load balancer should be placed in front of +instances of radosgw for load distribution and failover. + +Services Placement +~~~~~~~~~~~~~~~~~~ + +Ceph scales very well by adding new OSD nodes when capacity increase is +required. So the size of the Ceph cluster may vary for different clouds +of the same scale. In this document, a cluster with 45 OSD nodes is +described. + +Instances of Monitor service require 1 dedicated node per instance, to +the total of 3 nodes. RADOS gateway servers run on the same nodes as +Monitors. + +Control Plane Operations Monitoring +----------------------------------- + +Summary +~~~~~~~ + +Monitoring the OpenStack Control Plane infrastructure is essential for +operating the platform. The main goal of it is to ensure that an +operator is alerted about failures and degradations in service level of +the environment. Metering plays less important role in this type of +monitoring. + +Currently proposed solution for infrastructure monitoring is produced by +Stacklight project. + +Stacklight is a distributed framework for collecting, processing and +analyzing metrics and logs from OpenStack components. It includes the +following services: + +- Stacklight Collector + Aggregator + +- Elasticsearch + Kibana + +- InfluxDB + Grafana + +- Nagios + +Stacklight Collector +~~~~~~~~~~~~~~~~~~~~ + +Smart agent that runs on every node in the OpenStack environment. It +collects logs, processes metrics and notifications, generates and sends +alerts when needed. Specialized instance of Collector called Aggregator +aggregates metrics on the cluster level and performs special correlation +functions. + +Elasticsearch + Kibana +~~~~~~~~~~~~~~~~~~~~~~ + +These components are responsible for analyzing large amounts of textual +data, particularly logs records and files coming from OpenStack platform +nodes. Kibana provides graphical interface that allows to configure and +view correlations in messages from multiple sources. + +A typical setup at least requires a quad-core server with 8 GB of RAM +and fast disks (ideally, SSDs). The actual disk space you need to run +the subsystem on depends on several factors including the size of your +OpenStack environment, the retention period, the logging level, and +workload. The more of the above, the more disk space you need to run the +Elaticsearch-Kibana solution. It is highly recommended to use dedicated +disk(s) for your data storage. + +InfluxDB + Grafana +~~~~~~~~~~~~~~~~~~ + +InfluxDB is a time series database that provides high throughput and +real-time queries for reduced support of standard SQL capabilities like +efficiently updating records. Grafana exposes graphic interface to time +series data, displays graphs of certain metrics in time and so on. + +The hardware configuration (RAM, CPU, disk(s)) required by this +subsystem depends on the size of your cloud environment and other +factors like the retention policy. An average setup would require a +quad-core server with 8 GB of RAM and access to a 500-1000 IOPS disk. +For sizeable production deployments it is strongly recommended to use a +disk capable of 1000+ IOPS like an SSD. See the `*InfluxDB Hardware +Sizing +Guide* `__ +for additional sizing information. + +Nagios +~~~~~~ + +In Stacklight architecture, Nagios does not perform actual checks on the +hosts. Instead it provides transport and user interface for the alerting +subsystem. It receives alerts from Collectors and generates +notifications to the end users or cloud operators. + +A typical setup would at least require a quad-core server with 8 GB of +RAM and fast disks (ideally, SSDs). + +Services Placement +~~~~~~~~~~~~~~~~~~ + +InfluxDB and Elasticsearch subsystems of the Stacklight solution should +run on dedicated servers. Additionally, clustered InfluxDB needs at +least three nodes to form a quorum. Elasticsearch scales horizontally by +adding instances, each running on a separate node. At least three +instances of Elasticsearch are recommended. + +Stacklight Aggregator service could share a physical node with other +low-profile services. + +Nagios server can be co-located with Stacklight Aggregator due to its +low resource footprint. + +The total of 5 additional physical nodes are required to install Control +Plane Operations Monitoring framework based on Stacklight. + +Services Placement Summary +-------------------------- + +The following table summarizes the placement requirements of the +services described above. + +Table 1. Placement requirements summary per a service at scale of 100 nodes +''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| Service | Number Of Instances | Number Of Dedicated Nodes | Can Share A Node | Requires a Load Balancer | ++==========================+=======================+=============================+====================+============================+ +| keystone-all | 3 | - | yes | yes | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| nova-api | 3 | - | yes | yes | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| nova-scheduler | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| nova-conductor | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| nova-compute\* | - | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| neutron-server | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| neutron-dhcp-agent | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| neutron-l2-agent\* | - | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| neutron-l3-agent | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| neutron-metadata-agent | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| glance-api | 3 | - | yes | yes | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| glance-registry | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| cinder-api | 3 | - | yes | yes | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| cinder-scheduler | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| cinder-volume | - | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| horizon/apache2 | 3 | - | yes | yes | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| rabbitmq-server | 3 | 3 | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| mysqld-server | 3 | 3 | yes | yes\*\* | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| ceph-mon | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| ceph-osd\*\*\* | - | - | no | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| radosgw | 3 | - | yes | yes | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| lma-aggregator | 1 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| Influxdb + Grafana | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| Elasticsearch + Kibana | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| Nagios | 1 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| **TOTAL** | - | 6 | - | - | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ + +Table 2. Placement requirements summary per a service at scale of 300 nodes +''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| Service | Number Of Instances | Number Of Dedicated Nodes | Can Share A Node | Requires a Load Balancer | ++==========================+=======================+=============================+====================+============================+ +| keystone-all | 3 | 3 | yes | yes | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| nova-api | 3 | - | yes | yes | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| nova-scheduler | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| nova-conductor | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| nova-compute\* | - | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| neutron-server | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| neutron-dhcp-agent | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| neutron-l2-agent\* | - | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| neutron-l3-agent | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| neutron-metadata-agent | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| glance-api | 3 | - | yes | yes | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| glance-registry | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| cinder-api | 3 | - | yes | yes | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| cinder-scheduler | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| cinder-volume | - | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| horizon/apache2 | 3 | - | yes | yes | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| rabbitmq-server | 3 | 3 | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| mysqld-server | 3 | 3 | yes | yes\*\* | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| ceph-mon | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| ceph-osd\*\*\* | - | - | no | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| radosgw | 3 | - | yes | yes | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| lma-aggregator | 1 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| Influxdb + Grafana | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| Elasticsearch + Kibana | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| Nagios | 1 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| **TOTAL** | - | 9 | - | - | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ + +Table 3. Placement requirements summary per a service at scale of 500 nodes +''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| Service | Number Of Instances | Number Of Dedicated Nodes | Can Share A Node | Requires a Load Balancer | ++==========================+=======================+=============================+====================+============================+ +| keystone-all | 5 | 5 | yes | yes | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| nova-api | 3 | - | yes | yes | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| nova-scheduler | 5 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| nova-conductor | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| nova-compute\* | - | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| neutron-server | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| neutron-dhcp-agent | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| neutron-l2-agent\* | - | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| neutron-l3-agent | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| neutron-metadata-agent | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| glance-api | 3 | - | yes | yes | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| glance-registry | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| cinder-api | 3 | - | yes | yes | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| cinder-scheduler | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| cinder-volume | - | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| horizon/apache2 | 3 | - | yes | yes | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| rabbitmq-server | 3 | 3 | no | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| mysqld-server | 3 | 3 | yes | yes\*\* | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| ceph-mon | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| ceph-osd\*\*\* | - | - | no | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| radosgw | 3 | - | yes | yes | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| lma-aggregator | 1 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| Influxdb + Grafana | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| Elasticsearch + Kibana | 3 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| Nagios | 1 | - | yes | no | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ +| **TOTAL** | - | 11 | - | - | ++--------------------------+-----------------------+-----------------------------+--------------------+----------------------------+ + +\* this service runs on a Compute/hypervisor host + +\*\* this service might require specialized load balancer (proxysql) + +\*\*\* this service runs on a Storage/OSD node + +Total number of nodes for control plane is 6 which includes separate servers +for monitoring infrastructure and RabbitMQ/Galera services. The following +schema describes the layout of services in the test cluster. + +|image9| + +Data Plane +---------- + +Compute Virtualization +~~~~~~~~~~~~~~~~~~~~~~ + +QEMU implementation of Linux KVM is used as a virtualization hypervisor. +The KVM runs under control of libvirtd daemon, configured and managed by +Nova Compute microservice (nova-compute). + +Network +~~~~~~~ + +Depending on the picked Neutron plugin, data plane network might be +realized by different technology stacks. Reference architecture employs +OVS+VXLAN plugin, network data plane at compute hosts runs on +OpenVSwitch virtual switching software. + +Storage +~~~~~~~ + +Local disks and Ceph cluster RBD volumes are both used to provide block +storage capability to different services in the platform. + +Ephemeral +^^^^^^^^^ + +Disk files of the VM instances running in the cloud are stored in Ceph +distributed storage via RBD. This enables live migration of VMs and high +availability of data in certain cases. + +Virtual Block Devices +^^^^^^^^^^^^^^^^^^^^^ + +Virtual block storage devices are provided by Ceph via RBD. + +Images +^^^^^^ + +Images used by Nova to spin up new VM instances are stored in Ceph via +Glance service's back end driver. + +Snapshot +^^^^^^^^ + +Snapshots are stored in Ceph. + +Database +^^^^^^^^ + +Database files are kept on the local storage devices of nodes that run +database servers. The replication and availability of the data are +ensured by WSREP mechanism of Galera cluster. + +Monitoring +^^^^^^^^^^ + +The monitoring metrics and time-series data are written to InfluxDB +database. The database keeps its files on a local storage, similar to +the approach taken for state database (see above). + +Logs +^^^^ + +Logs are written to local disks of nodes, plus optionally to remote logs +collector service which is the part of Telemetry component of the +platform. + +References +---------- + +This section contains references to external documents used in +preparation of this document. + +1. `Testing on scale of 1000 compute nodes `__ + +2. `Performance testing of OpenStack `__ + +3. `RabbitMQ Clustering `__ + +4. `RabbitMQ High Availability `__ + +5. `ZeroMQ Status Update `__ + +.. |image0| image:: media/image03.png + :width: 6.50000in + :height: 3.68056in +.. |image1| image:: media/image02.png + :width: 6.50000in + :height: 3.33333in +.. |image2| image:: media/image05.png + :width: 6.50000in + :height: 3.68056in +.. |image3| image:: media/image07.png + :width: 6.50000in + :height: 3.68056in +.. |image4| image:: media/image09.png + :width: 6.50000in + :height: 3.68056in +.. |image5| image:: media/image11.png + :width: 6.50000in + :height: 3.68056in +.. |image6| image:: media/image13.png + :width: 6.50000in + :height: 3.68056in +.. |image7| image:: media/image15.png + :width: 6.50000in + :height: 3.68056in +.. |image8| image:: media/image17.png + :width: 6.50000in + :height: 3.68056in +.. |image9| image:: media/image18.png + :width: 6.50000in + :height: 3.68056in diff --git a/doc/source/index.rst b/doc/source/index.rst index f43dc73f..bfc94dcf 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -46,6 +46,7 @@ Design docs design/clusters_on_k8s design/ost_compute_on_k8s + design/ref_arch_100_nodes design/ref_arch_1000_nodes Indices and tables