Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1WUd8i-0005pm-QW for pgsql-performance@arkaria.postgresql.org; Mon, 31 Mar 2014 14:24:41 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.80) (envelope-from ) id 1WUd8h-0006qD-Jh for pgsql-performance@arkaria.postgresql.org; Mon, 31 Mar 2014 14:24:39 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1WUd8g-0006q3-Hm for pgsql-performance@postgresql.org; Mon, 31 Mar 2014 14:24:38 +0000 Received: from mail-lb0-f179.google.com ([209.85.217.179]) by magus.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1WUd8c-0001Fz-VN for pgsql-performance@postgresql.org; Mon, 31 Mar 2014 14:24:38 +0000 Received: by mail-lb0-f179.google.com with SMTP id p9so5775840lbv.38 for ; Mon, 31 Mar 2014 07:24:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-type:mime-version:subject:from :in-reply-to:date:cc:message-id:references:to; bh=VRAqn2OSNKS3dndISg9VuhcviBn5+vsawA7KLtEN3Yg=; b=jOz7LMIYq+ISpP/pmEoV7yaOmNHolhHPpgwSVCuGUhvZvfI8ocGBwI3qnzCiA7GWY5 XV48lQ4LGzLsp1GeQa32Xpda5NZEahmLXEu1ggqgg/avVBk4LeOwXeUR4AZe3wWvdnn2 fn15j550aF/kD7QD2TKbzw5aftXemda1tpt977rabPJgBR4rmYbiJ1hDc80B6js0FTEx VMuR2CLIUTPIbMhVwyq93ZmtAVdzHfiaW1KpSR416AWMbf3dSm/LL/2F+XZAtNdmLiI5 bgEri1PuHQ/6GGUtxHLrBsBMnFqvaslg4WI1RKqdAfVpGgHITsdBv3JoBiYQy1h+Jq9G W0uQ== X-Gm-Message-State: ALoCoQk/CdXg5eoXssFJabprPfvlYCoBxcL51xJ3Ya5OIDSDyibWopOgw2qc4y2eCLjekc6jtELX X-Received: by 10.152.23.227 with SMTP id p3mr2025123laf.31.1396275873814; Mon, 31 Mar 2014 07:24:33 -0700 (PDT) Received: from [192.168.0.16] (x1-6-84-1b-5e-20-07-92.cpe.webspeed.dk. [80.167.117.15]) by mx.google.com with ESMTPSA id d9sm14146609laa.17.2014.03.31.07.24.31 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 31 Mar 2014 07:24:32 -0700 (PDT) Content-Type: multipart/alternative; boundary="Apple-Mail=_BCCB4ADA-6425-4260-B6F8-AB4F871F1AA7" Mime-Version: 1.0 (Mac OS X Mail 7.2 \(1874\)) Subject: Re: Sudden crazy high CPU usage From: =?iso-8859-1?Q?Niels_Kristian_Schj=F8dt?= In-Reply-To: Date: Mon, 31 Mar 2014 16:24:30 +0200 Cc: "pgsql-performance@postgresql.org list" Message-Id: <2ABC90E9-8312-45A4-96E0-CFE202CCD9CE@autouncle.com> References: <24BBD031-7384-4FFF-848B-90F5664C0063@autouncle.com> To: Merlin Moncure X-Mailer: Apple Mail (2.1874) X-Pg-Spam-Score: -2.6 (--) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-performance Precedence: bulk Sender: pgsql-performance-owner@postgresql.org --Apple-Mail=_BCCB4ADA-6425-4260-B6F8-AB4F871F1AA7 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 Thanks, this seems to persist after a reboot of the server though, and I = have never in my server=92s 3 months life time experienced anything like = it. Niels Kristian Schj=F8dt Co-founder & Developer E-Mail: nielskristian@autouncle.com Mobile: 0045 28 73 04 93 www.autouncle.com Follow us: Facebook | Google+ | LinkedIn | Twitter =20 Get app for: iPhone & iPad | Android Den 31/03/2014 kl. 15.47 skrev Merlin Moncure : > On Mon, Mar 31, 2014 at 5:25 AM, Niels Kristian Schj=F8dt > wrote: >> I'm running postgresql 9.3 on a production server. An hour ago, out = of the "blue", I ran into an issue I have never encountered before: my = server started to use CPU as crazy. The server is a standard ubuntu = 12.04 LTE installation running only Postgres and Redis. >>=20 >> The incident can be seen on the in numbers below: >>=20 >> https://s3-eu-west-1.amazonaws.com/autouncle-public/other/cpu.png >>=20 >> I imidiatly took a look at pg_stat_activity but nothing in there = seemed suspicious. I also had a look at the postgres log, but nothing = was in there too. I have pg_stat_statements running, so I reseted that = one, and nothing really suspicious occurred in there, expect for the = fact, that all queries were taking 100x times longer than usual. >>=20 >> I have tried the following with no luck: >>=20 >> * Restart clients connecting to the db >> * Restart postgres >> * Restart the whole server >>=20 >> I have run memory tests on the server as well, and nothing seems to = be wrong. >>=20 >> No changes in any software running on the servers has been made = within the last 24 hours. >>=20 >> The question is: I have a streaming replication server running, which = I have now done a failover to, and it runs fine. However I still have no = clue why my master suddenly has become so CPU consuming, and how I can = debug / trace it further down? >=20 > Using linux 6? One possible culprit is "Transparent Huge Page > Compaction". It tends to hit severs with a lot of memory, especially > if they've configured a lot of shared buffers. Google it a for a lot > of info. >=20 > There may be other issues masquerading as this one but it's the first > thing to rule out. Symptoms are very high cpu utilization and poor > performance that strikes without warning and then resolves also > without warning (typically seconds or minutes after the event). >=20 > For starters, take a look at the value of: >=20 > /sys/kernel/mm/redhat_transparent_hugepage/enabled >=20 > And do some due diligence research. >=20 > merlin --Apple-Mail=_BCCB4ADA-6425-4260-B6F8-AB4F871F1AA7 Content-Type: multipart/related; type="text/html"; boundary="Apple-Mail=_BE59B16D-0314-4D22-9919-D24757751487" --Apple-Mail=_BE59B16D-0314-4D22-9919-D24757751487 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252 Thanks, this seems to persist after a reboot of the = server though, and I have never in my server=92s 3 months life time = experienced anything like it.

Niels = Kristian Schj=F8dt
Co-founder & = Developer

Mobile: 0045 28 73 04 = 93

Follow us: Facebook  | =  Google+  |  LinkedIn  | =  Twitter =  
Get app = for: iPhone & = iPad  |  Android



Den 31/03/2014 kl. 15.47 skrev Merlin Moncure = <mmoncure@gmail.com>:

On Mon, = Mar 31, 2014 at 5:25 AM, Niels Kristian Schj=F8dt
<nielskristian@autouncle.com> wrote:
I'm running postgresql 9.3 on = a production server. An hour ago, out of the "blue", I ran into an issue = I have never encountered before: my server started to use CPU as crazy. = The server is a standard ubuntu 12.04 LTE installation running only = Postgres and Redis.

The incident can be seen on the in numbers = below:

https://s3-eu-west-1.amazonaws.com/autouncle-public/other/cpu.png
=
I imidiatly took a look at pg_stat_activity but nothing in there = seemed suspicious. I also had a look at the postgres log, but nothing = was in there too. I have pg_stat_statements running, so I reseted that = one, and nothing really suspicious occurred in there, expect for the = fact, that all queries were taking 100x times longer than = usual.

I have tried the following with no luck:

=        * Restart clients connecting = to the db
       * Restart = postgres
       * Restart the = whole server

I have run memory tests on the server as well, and = nothing seems to be wrong.

No changes in any software running on = the servers has been made within the last 24 hours.

The question = is: I have a streaming replication server running, which I have now done = a failover to, and it runs fine. However I still have no clue why my = master suddenly has become so CPU consuming, and how I can debug / trace = it further down?

Using linux 6? One possible culprit = is "Transparent Huge Page
Compaction".  It tends to hit severs = with a lot of memory, especially
if they've configured a lot of = shared buffers.  Google it a for a lot
of info.

There may = be other issues masquerading as this one but it's the first
thing to = rule out.  Symptoms are very high cpu utilization and = poor
performance that strikes without warning and then resolves = also
without warning (typically seconds or minutes after the = event).

For starters, take a look at the value = of:

/sys/kernel/mm/redhat_transparent_hugepage/enabled

And = do some due diligence = research.

merlin

= --Apple-Mail=_BE59B16D-0314-4D22-9919-D24757751487 Content-Transfer-Encoding: base64 Content-Disposition: inline; filename=au_logo.png Content-Type: image/png; name="au_logo.png" Content-Id: iVBORw0KGgoAAAANSUhEUgAAASYAAAAeCAYAAACBmrOuAAAgAElEQVR4Ae19CZhVxbHw2e5+Z59h VgZmg2EAB5gZNlkUZFcUg5gEcUFRI2oUNAoSBPeoL24/L0STGJOY3yd/4hKNiBiJmqgswQ0BlUVZ hYHZ73qWV9V96sy5d+6duYPk5f3fl4Y+Vae7qrq7urt6vWdE4fS5dBBVAT4PfA54D3gneBF8BHwI /EnwjeD3mRDA/wrn6iEXBsRjGVJ13cmLghA9BUEy0ChJ6DQIV8247uiSsPcqOJwCtQNoCsH3AY/1 7wMvgQ+Abzf9UYCHwKMuU3FYdmw/iRzKCCaKSBDmhjDMSyKHbTKVuhDy8vL8ZWVl1ZqmlbS3t2/e vXv3YRI4YsSIi1VVbQmFQvs///zzvRDem7ZCYhLCtLS0HEi3xuPxZIfDYb9hGC5Zljt0XW+LRCKN x44d29nY2NiWkLlrINYT+kSuNzqN58f2ju3wdDiWj2QNP9UEsCFWo5cG1A5VxpxTIfcpyHNm5ucY LrdHdDldgiGJqh4OK+FwKHqy8WS08Uijvv29fdo/3v4E+HaZvjcNFlhOq8sFadf3IBGV9Sj4lh7o MBoraFk3dD+HuCPdxFPUCEBm0Usc3ALvr5phdQBnxsWfrlfsYPd1I6wvxA0DPzh/kC+/dEhWn4LK jDxvhsMnKKIUblMD7Y3B9pbGcPu+7Y1HGz8PYj3vBP8++CbwSV1dXd00URRvSUQA4fu2bNmyMFFc fBjI+SnQD4oPx/fW1tYfgCHBNpjMybW1tTMURZkCxmAIGANJlCRB9vtvBwbLMIke30JnOOgEJzQ0 NITASG0C2a/u2bPn02SCuwtHI1hcXDwX0p0qSI6icHF/f4cn3al53IrmcEpCR1BzRoOqo7Up3N/n 6wDDtR/SfP3gwYOvHD9+HAeChG7YsGHzQOZViSJBR/tBp1ckiuspDHS8AvjH9USXYnzr1q1bzz9V w1QCiUyRGyaNTRt9dnWkbGC1KzMHO3iMwx6KQ7uHDaDQY3ML+3kHDhX0sVMFrb29NXJo167Ilnd2 aX/98xZB1zcA6ecxAv4HXirHZ02/aHX9bYYu4syOO0QNsEVmkA6Ze+bm97d/83Hby0SSFF4kyLdf Ow0bLogwQITIIL4jvvvvB99+4cc7ejRMCx6vm108OPdW5It3x79qefWXC99nhmnBE3UXFNfkLkGa LulhGPjOgiGVzcWV0yozlh2cCiPKw9M3JjJM2RA9C4zRiNFz+9eWjymo9Wc4MSzGkQoR6qJhtBwK Hv76oxOfvfXkrg/aj6uvA/F74LGJdHHHZsw/u6PfUNbYNZjTyDDnQYjOHWipEbbM5S89PA9995bz Ipm5BYnIvJ9vHSw8tCyhYaqpqRnp9XpvMty+kkD54OyO0oE5wdzi9Kg73Z25d/twYdu2D0jm3vnL J+lQRldbY8DXeKTFv3tbbtaRPdOhw37Q1NR0/969e1MZ0Ji44cOHf0f2eq9sGzi6vHXA0PxATp80 p+joMhtBpWGgGlFVz8lDZ/h3fzSyn/L+pSUlJU9s3779NcqbHTbOumJGR/EAplMdKkWC+keIjuv0 e3bylPHjMxZMDvlzBjMGFIcNjovtxJM2wthkZF0NCVuvSLpUiKXufMMl2mR5wqyZnlmXTPHk5hVp kAEfy4jBWhgZI2TBYPL4zrQL9Ahlvy9dHlg30g8+OOfK8yIbXpwYeeWZP0PUG+APgv+fcMrwGX0v k50KTveTOsOQhFHn97/y5Y8/eQWIup36V7YLouKScGqb0CmSyNSQMNIWqDskJZkcRRadRCrLYlI6 okkV2g1JNzzl6YXOyy9Ycca5xcNyh8kGWjc0ityTrcMwxHXQFkIJ6LKKvMXoqycWjt3yhz2DN/18 7x+B7DnwKtLbnSE4FNAAGzhpHUbQCPFwO30y3JBlSw7RGJAfEfLrUBws7xRO8Iwzzpju8Pluax4x ue+xwQ2lisNjLX8oD0SLEDu30+VQVGd+entuQXp79fC+RtM3bfnbNmVnf/lhBRTjJpiZ4WyxOyfB bOuG9kH1i74ZMblC8KVBmxShr0C/MkCJkN/YDi9COAx8DlkJ5Zdmo2+sm1he+P5rxfWyXA6zjjXx iYmyoojQrjBcNoVxCO8hxSpjPF9P71FvRpaa0we3csChSimzvK+TEWXRcQ8NaDEPCNGFQ2HWd3oz YxorDxs133vBoqmOkn6VKAQVwyEDHIdnRIOGJhpHZVE6rOuGqAtGkaHoBbIhyfZMEO7x+rM8F1wy Nzh+2tjgq89O1N5+9XkQgyNqwtGUJXR6HjV9h+WNNIthSaTOhRAdgoqGnLEIwH8BPqlTg4ycx1P9 cDWZqk/K2jUijt+qcxslblhZLo4+7tUiIyRROTEuYW/lTMXlZ+YsmXPH8AWedEc6M9HUDokPErX4 CceMgKP8OD2K78xLBs7pNzy37JlrN6NRwvqOMfjQ2UFMHKP5Khhd7BiKT+6wghPkM1HrguXOMDE9 d8WBWZedoWb3SVOs0lDuKRP25MB4MAuMszozPiMv7avJ84ZmF/XP6vP2q3cB9Q/A49I4oYN9qqVN DdOvPTp8fJkTy87ExKVlf0VjhXlj0BTp9bu/mnTR0D6ZuTcPU9V9H374IQ70nQ5ZLBkmv6l23fgW XQ1lJpFLUgl2ZoYwNEqUl045qRgmpDnXdc0dP3TUT5jowPHPUgav7YguNYL4l8Ee/TGgBrYJxz87 Jqxahal1ulWrJKFwaL4seEa5RWmOIhrngj2F6T+faSGhJzu3yLngh5dFRk2ubX90RaEQDfweglPd 5OxMK0Vs4tXll3vTnKaltzGBkrFkpGzUuS/LkztmQf9L3vvt/jttlF1QNa/EbFQYhQ0HuZk0eCKe mpOg7zFOeBAXjvIMpwAQxYY56z1RemLydIEvvpxWGl2z6SqvS7t+3urhVyouh5tnJDY9A9qFKEpQ ZF71tIxFiA5zgiXgUBD6Ds4ZdumjDbf85qYt+yDCWhohLbQPMWIfMawyghx7OBJ342BfCGY0kB8b vwQ5oCVMHKskZuTddnD2VSO0tAwPEGFiNhKOq7Btagvka0wHxNn6F5bWCfsD7dUNfSXdmDxcC10I SyycHXZxsHybcHLCnMVNg+r7OXXqwlxXKAdd/MyCD+q6OfMBApPcCcptHnF2hbP15N3lra1/i11G gmzSHaPH8jHxEB7bXc3QFAHq19RTF7mYBgRSPGXUSpiUZqYvcsvek2HyC5J/geu2+6/3lVXV8Mol QTD/1tU3dU26t31T09vCunmk0cSF4YbqCBC92AFeOOssRfjekklpumulQ9bPRCYS4KyqGea98//c GXjizgLhmwNPQtTxxEK/VWh29egC2DQGcxHXoVjDhcIiZA4Ui52helzB+WCYHoawpKcg+Vm4m8Ir Caf9MFu0joTMVQ+XmcITOzDrhNSxUa6JE3vRwOwD3aWH/JRuquWkNESn1DnC+4SzZq+su0pxOd1k eOLLJ4BREjVN1RXpkKHrzbJq5IuKXED08YYK811al1d39jUVN7/18z1XQZmsjVsDDBMalUQupbWw yYh2Belxdi+busPZDeIRq8VxYtgTGnxs1PQJmj/DY0D/YPk1Z0DISupvLBm6LyZfihgC+X6eDo9h SzBAETbWNJS6Dn2+SNi+vcvMEDe6I1XD72scVF+K63OqSzs/SlQFoyMoGp8qhtwESu4vGVI1lsHS LWvHnUb/8OgZQysO7jtf2Lv3N8iPjg06pgGSQS/Y3xB+ewetPIlc3LqVcHBABYJDOt4OeN0ynYGO qE6jToW1ge4Mk0soG/RD/6Lli505uYUgEcUy4RFR26mr6q2B685nG7AssLePTZtUYdOmDdDDN/jX vHKhKIkPOCSpisS48wr7yj965NaO3/40T//w76shvNtTHOJLFVaMSp/apzK9CssFqmOtDiF0qEgg EHnL63dNo0aCMx0sfnFV+uDcKteExi/CScsdDWrQPbHLklGSmHHCd4npELHUHBoVrD7iQ/b4WVdL S/PvPelpf0WJUVjXOWAKhRAd4u1Hwj/O65fGdort5cT4jrbwMdgNm2KnJxkYH9FksgzyBUuG3JgG s0auL97IWN4gR4xI19uh0a34rDn6zItzNjUjP7ofvTFhkOLyroQR/rsWr6kHNLwY1nB+v1kfrj84 oemrsLX0MFTN2owjo0JbB+agyhPo8QlKw07Dmi57xOI2/lBB6czmippCl9nJWKWb8biXip2nQzP+ FNI7PraxCWFZftitaksMWUpnM1gzkrcC2FiGpVn7oNG1dXWflW3btm2PnbdP9eBzj46aWoc0PGNc t1TmEASqunZvSG15SLjxklaL9/H1I9Ol6LNORarEMFxCysBKS0mn0+1oHjJ6vvD2Rssw4ShDBoDk k04Vs04s+b1BNDFpXSnfHHz9eGbuLVnWrqgT1Io54QGNkYiQBaeZCJlT+GiRzDBJQm7ppWmLVy5x +TOy2foZuNBCB6LC3YE/PHQXGJXY6SwXe0rP9sXn/lG4+uo/uUac90CaIC+hmZPL583QF/7omvDa e5rVz7beD8LDp5RAVyZ5xKyKy6EtmG3HbLDQegPtkXcUh7IWjBIeV3NOqDSGw0bq2Isqrnz5vs+w AxFTjHQtLReHMRaGpx6IM4gMLDyZymPEwAtkDcSQDIxleTBlE/WaCzefABx9Qrdw7aiTnSyUZQ4N zYg+fM5fYjpZQiGCUFU+On8c5gc5Ta1YUNW0DugRY38yZeMn8fwPTnkbrwd87/a/TjukCNJSPlry 9FEOYi6fyz/h8gGLXlr9iWWYUA61O0yIdTgzYZg9YHTKjnjpVI/L68oeKBt8jqLrIrW/WApDaDOE G8PXnftEbDhc0LtmxurQmpee9avau9BE8jGezUZMiO8thf2z/X1KRwtCrGEKVdffYHj8bhgQkQxc p25wuRnVjJvDN5z3GI+zPW+cvrl1zfMzMiKenQ7Y0GYzaygYh5i+IYRyi/hJmcmGpiBMMxvAmXGy iTxlVDagKfH8syUm4AjROULBFuGHF37a3awiURwZ9fg8TUlbvGKZy5eejR0LrSxsYodbNeGSwA0z V55Oo2Ql/OST0fC15y1t0bSrdVVQoaxMcR6Hw+257Kbrhey8+UBrNk2L61SR6r512aNYG4B2wISa UFHkZ/btaHtdjxphZmAgH6ypmLD/KOigfBM8YdqZIZXtSth5SI7Z5hLyJQq0y7DjiWi7DUNm8PZy 9kaRExaWf9fn53txdj4Lj+rLEhkle57eePb1ZdGw+jVbIkEE8RIsHZINnZZd0GRs0MHAwONMBzwu pwiypRV1YnsKSXCoNy6HoE1enAUKZRcMwI6NHtsfQcT1YHBrIqNkpbr4/C+hjyzC/Sz0yEsQcdiU EyMlA0ZZ9IAMGDCguLli0FCsHN4ROSRcCAX2JzRKJGTxvC8F3E5hRg3SMyHKwyVaILvAnDtzBph5 CTjbRI9GjyDbb0P9nqqLQnktuRwnPXAT3XvBiQzTYM/199zlyC8sI+FRLdIeEYxzIotnPtv7JHrH EV183lMthjpNjUZhSc0r2ZGWkeO9bvVykDSxd9ISU09cWHYZdLRMnNCYkxoGo7oWbTwUeHHdkveC wWD0b1B3zKbwtTnQwntGhjtvzLy+SS98qGGY10Ky1OEIYk5YROIsdQ2F4d2eLrN2KAPy0Dkh78rW NQT2feLKSeVCmIqrqM+bjoaN0ZuQZOiqHoFrp7/pSc62J4WoGtbXkL7tslB2Zp63oP6ivmicLCdB OHrWcU2IeO+czpbCOCPBJTFfFhOMtUwRb3oGDSLY9hGnPqBFQr/oKd3AHx99TYiE2lmnAn6CaBjB aAlaZnaFXYZSO2qikZaXxtJBo4LpAWQGBo2HGv2dnT4R3qJHr++QtOkdEfAEAW8T1ekthnZFVx7U H3h7BTC8K2XvQk6v3Ph1heKec9Uyz6AzRvKM86yFDu37Vfj+Je/2LqPfgvqG2X9Rb3/saXff8uv4 5rEoeAv7VmiLlt0Zfur+f4DkzrV275PJqhyTfy7VC7KzzgLtItyuvfOrK//GNradDvE56M+TMB5V jgYGITq4hzPnvecPPAKotVnLIuCh+nRrGyiezxJAxD3AeH5mqHrgiY/G5RD2MXRUToTsnYOens6s Yl+5nd5kZ3xtbZF3Hpv3VkoXCDsU7XmvIfyE+OPL13dA5pitwoGXmWDY/KaMU78h2NsN2/j9FDYb T1Bq1eXy4qYzc5gYq3T+3i5pGxOwxAbB9oY8dcFhTXEOQGUzA0PKhkqIOl0xJ8DBnKKz7P2Mkiah zW7514QnhUvmfQn7G18mjbdH4JUAUqI9HPAoGMJTdXzGlZjf1d5cBVch7k1VdkdHxxr8uU+sYfLn jnONmTSbMo95laByxH1780BwBng0CKjcLPBp4H3g8XIielzCojz0OFhgtbJBAyB2Dcw5QtX0uNuF e0Z4HQAO6thJF26aYiNP077+PFMv6Q8oMvFCKzWjxukDh8+N7t7+KxZxCo/S4f7JReWZA+2NgNog XFx8hkQ2RdUX8t3KWpiBwzYPFAWIGASCPtUZQ/IGusYd3x1eT/SxEIsOCmCCO9f9LBBLn6KjzWFK F/eoGG5tlvQsCJfhPDe8TSKO2UrVwRifD1cqcnhRcPusUw+IB78JpzxgrZ2xaf+tf57S6vIoVge1 ZyW9wFNt5YsdJfJYNkuEjNOs0V53Fn0yBPYF0NjgjAU3h9Fhu2Z47IQJfj0lwwESt+K4t4c47Q8K ysnDnLv7pxiOBrCjkvFDSA5CsZ9YTk3LHsBmZhDCj/+BwqytaCgYEm793h6L+DQhrE2eJlkkRoJM J5MbTs/u21J79kVE2xPUWo+9LMQZJrdz7mW3Kx5vGu3UY6PG0wjn0IZpkS1D79b37WiUKofli8VF mUpGfproT/MJfp9bUjxuzeVwOiS8retQopKE10slFfbkMCOKruKVODZHldWoGtWjqqRGI0YoEBY6 AkEj0NahNjW2GUcPNOu7P/xGKqvJVYaMnBZfWIciKMbMi24Aw/QnEHsqVwjk+tllV8BYjPaWdTLM H3b2KMybTxwJvoDv6J48b1Pj0j9N/sjjcwxnAbYGBnuNjpEXVF756k92vA5xnS0PXjyyKtFGJspF 3G5YmKxUHtA/2Ga5LV1kY2Gp8NtoiIflB+RZ+YnrmDYWCx19waBqYGADDMkhiESqrh+yiFNA9KB6 QrQZJjuLN82Ra73DHhMZCUifGRaEzFl3fSzq5AjUDsnpnBSAvDi9ogAH3P+ltk/xCKO4zbhkCQ6g PTpJD0eZMYKsspmdmWVkhN/Z2d4gX4o7i1oPXTLkEGjVCA7Wp9dBm4rZpIbEyRB+q4RAacnkBvJL c8P5/XLpZne8AY6/n+XduaMQ84KzG+76DpzmPmPUWdDyIbOdDnE5LT0764d33RDSNN0tyxIlwpSK qqauaeIuU/1d5MBogLwuawzvTIewKLQiuDbAOgIWlgqC8YiL/atrpVFTLtU/eOM/iKcXsKpseO4Y qHYsJjhqJ6IQblWtZRzJ0zTjD5AiGKZOOhYHzFV1fcYLwo4yeN9L9AibIrCUM+kpDQ4xluQg3rNj MyZGZvKhIOycfFDvWQBQcNtD6cZCeutOUEauO92iQ53hi62+IwH1m+744+PCEb3DQ/xxkTBh8VIQ DEpUSWiNeTBBIkoF4siKjnTHcFAg6lGLtcy47LXuOgEdG5iRvlcOD21APmSfKYpBFID5iC14WFE8 NKPCnNDdJaR2adEu2wQY/q2dqUPqwwStJewpJYDl5YzxBhYbDIUxCtqPoOkv8lEYI8BFVOdSS/DO nHsdDBjwNQCuWFRuvHfAb6u50kEaK6ANQs742p9DxOM98mJYvFx6R5kwpuDVYS4/AXRAjXvHT18I ebdfGWGF6ekxfkH/S73pnizs8PZ/OANwuuRn4vlFUf8vngV+jwnpuBcEf647v+E7JfPiefDdLjse T0SfLCxB8bnakzEkCMfORXnAvCNulSMFA6ekOYOsk5GdMCHl7cRhuEDSC6dF4DzeylEsBr9qsywF XAlgywOcNcd7uDuQcoqszGBwcOYDUzDmCe8iBJqd1faAluEI0ffSUUdHyD0KiJUD4y9YL5YoAIzD dg8QvB4Kmhd7eplwd+Rsj4knw8uJuNnXuuPrIY6XisqBxCiTA6YHwEkPzPhDJEL0WFaCXAc8MdoD yhVLyusxk2R4YOLyshHV71ANYy1cdmvHOGZ4TGjHqXDES4YmHtrlE48d2vlhdnYkrAuPdujaSlXT 37HzSgV9BwhFRUN4EVJ+ZgwYUzib6sEONTiNW//0rjdBEu59WP6R8zcdCwYiX2H36eze8AajLf6r nliMa2fcZ7NcCRzskMM00NnTguNyFNaz0+F7MYwfUgLIPKYLOHbZngVwCuzplD7bT2ayUE5q2fD4 XAcsfkgfcVZ+wNFguZxSUap5QTr4akc6yYuHasRoJFmQP2y18Gq2SRNiG1FR+Sk6UcLDCGy7IMv0 hOuiYhlCLg5ocL1Haz6CKabFyGDdhzedUQZBxHFJr8MU3C5K1LUODKelP0GkAeMZsx9l5/t2OGaB TiU5xJNKuL91ymJFFWSauoV9DIYjRI9lIkjlI4gJ0uVhOjEN5Rcew3C+lJswY5rLz+8sYWCLGr1Y u/X7eH2eu4effSBDk7fA5mAeWjJccqMjnKwbCzwNj0hU/bgjcHSisOpmukF8t/vh51Z6ZXk1Sxd+ Ue85c/a84Lq121NNru/QtLP7VGRUY4tmoyjrZLwcJ/d17P1o3cFFiWTt33zs60Fn9e2Hisf5BnNY CcBfUpE5JKPcPaZlb2gj8boK1ChGo+NpcYjvmK4W0j2Ip+D4igfTRSOADjsK4jq7OsvDenjKWEtW hoAXcVMedv2enOzUD5qpQyMDDQAvQnQMV4zSnmTY4h2yU04jebZwhsKvCaxlIdz7jrKRFGLYch4a OULmNNW+S8DDkjzhcqWDDXgJ4sMC/OTa5rBzmFXHdM30jkbFCrQR94CSJIKJyHVJOgrhtSwODQOr W14p8GUFHOFObTWZKDEMA9GyaYCwbIhT21K/zY94RV1MJldrbd4V8ae/lCxL8eFBj3s/hjHD5Cqv uZA2msMtTe9qq6/pNEpIdcv8ryIrnrjbm53/OL4mczj82DVJOLUidq0f2haNG+wKPVQ6nZaQ3JAa udZmlFhwcOnFd/vueWaRx+ctYcNcUclUiLgDPK9JYk4MpRGzyxbCJ0JgsOStjCCS55WlD1yxccad xMo6HNAhZA55EKcWauKKW3aOn1Nx5Sv/scMyTIf27A7V1pVzNniiBJ4iCxI0yUhphqFrRiHVSXy6 cFKU8v4DdW5MncpM0Doz5VlL+PzZ/Heblq+fHoFzDDx1tWQQMRx44IfiUnJOv1AJG9wZVvpxXM50 104Kgu32Nio37YMQlB0O/DQGGvgYw0K8BLOzs9Nl2eHhG9qdNYH7SBim+f1sdCZ6S98QwNos0CDs 3Iglyu4gXGJE2VDp1L6RGnG0A3ZOuAO2DU4Lp1E83lCnvhDxZnlr6+vP+Gjr1m4H38rKypKMjIwq u1w7Dj+B+Su8d+kjdEqJEB31UfbS24fN+sbL9bWe+LjlzkW391YkTnYkR3FZA6oMK0s/dmhPIiHR D7fuxkbOKhQheMZjQsT5FJlDO86qA+NRsAkRR1kEEceGGNTgfGztj3fF5wGMhKE3HfuKywLawtJB sNOTE0+X5L2y/4jcsRRHKwGEif5hx8FwhNzzDmlmMSasYnjORJDbn2Szi4QhNYi06AgijjK1iJFK R3Z5M5wDO9Pjcigv8NP7IygvNYc/RU78r2tzTSxRDUY3owR0BBHH/BQPyGgAtBjfe3IjZpZezA9s OSXJotwJHof1kxRYEe2j6X08hHtCjtoRYyb3lF5Jbe1ZIafHxflxScGXLnx5YQgt/owDdhm8DUII 1B3hCOlI307bLQ48fG+FQ9pncWIGbK7Dl77B2nsxeTA99BKsCsL9an5gI0+IusorL48UVTycyIdK yu6Br3zY+DRrbwfTxT5M6Z/KPpol2Mwz5pvJNSHip+oUwe/PkTJzimB+zowEyBoIwvCOErtoSIKl cMdkMiII0eEUmYd1GmQVVtKwltkNo8xRxZCaNFhIQ2vOgp+D5gFS45JEJy3sGS8kaMkDmR5RlgLp OfBJlK9/yxLpfBSLDqWY0vQqTk90xvTh6mv/b0MnSWJs7Pf7LUjPcLO7OEjBDTwYHviXyMXPmJAK uyVRE44wvcBf2HB+37lbXjqAXx1gLhrWjsGnQfpB/XA+kxFBYXXmRAA4wn3BiBM8ZK8wLbcorYLM gMlu5cHpVD5KwJY0CA3It3GqLP4Bpijj7IYE5aHY9HxfYdW4nMu/ePfE/RDU2RC6Jjiw+szCC2Ky YlMkGL9Dj5/3xqfE1pLTZxvbjKIAO4TrC6Ga2qXZ+3e/ffLkyYSXbfPz833hPpW34YFNzOhgyomq alhYvjD2ygnLHNcVb5O8bZqTCnsOkuLYtrGDxjvspGrcHbZQZO8HXrEiIDoccLGTO4Iop6OoctbI kSPrN2/evDVeHr5jGVsrG65oKx/Mp+g2IpQT1iJR4bZLO+sEsmXJhzwiTnn9VntMKMssM+abybXy Qr3dCkgJUZTxM4ZCzZn3N6Cz9qsYLpVULtUPfonLuYPgca073jF87DxuEaEwNtGIq7pxGOL+y5D0 F0KBPdvASsMFpQTuoouc+ohZQ3XJcR4sq74HC9wBSEXyqECesTNvDH66HfeXsEJwul4h1024ypFT 2M/eyNRMtgHek2FKrxlTeD7qDfsBOj4f4pCHxD65jnHG1BmeCKewgeMK5oFhWgPUbGkBp9BbgLWf rd8xQSiusF961aBJfZbs/MuxZ+B1G3j7qRYuTybNWTJsiexWvGgJbVngOAQ405wfAF1KjjdEKnlK LF2IjFDHOtUtPwgrYTgUhWhTHCsfvJ91yXGsHh4AAAzuSURBVIBLvnj3vX0Q8yL4QJwAJBs6dFrh 9UU1WWeQsUUa1B8aO4S6LP5fO1/0y3c+iVZNjMJSscvpK3byk/0HjygbMOjxwrbmR3fs2PGhnbeq qmqQJ6fP0sOD6odRx7PHs2sn4eBeexjDISPUFu1xvTFM+B0tLBC1Za5/3sbNj1h1il61KqTf84vn BUW+nCsVFGEqF/maivsXZOSVPDpsWGQ1fPQNtwuQgLny8vKMjIKilV8WlpWgUUiUnisY6vIBcNJH V3pJHDx4cCqzeZY+fFv8CPwhhG94bvhKCvF4ufArZO+gQYOSLjU5f+xz586dBxQjs7AfO2GDOLxv 4JUdLmXRj5aE//5mrfr5x4fkPsXp8rjp4x25Bf1R4faKg9nLF/Dlu7s6Qvt+3+XDcLFp8bd16yKB deuwM6Jf5bv/t9NhQFstS+JIJOB7CHDSU1lTLy5edW/0vTe3au0tQceQkRVSw5lngQFlrRinYOgc af4Se69mgXGPokrvxIKq9EFYpVSrtF0E37RucyjiNjgvge0W/DwJDS64wkUcIX/CMXe97JL9+A5q iNlyKoYO5y9QRrcfVd/CeK9Peg4Sm0vpEeRS4St5S2ovz+7zRfbfntv/JJC/iTzoZKcwZ8b1NQsr xxaMw3fKJ0J0mG4krO55ePaGfTyk5ydcPTO11TXfPXNzikfmvXfojj9PXQObLTdRnjCGypVfnlV9 6UMjl/zujs1V8Mu5zyCqEXwEPA5qFbXT+4yYft3QebARz7dxzPIwAYhrWnP78eYHAOt08KNu465R 72pe+ezOQI5h449m5qUfnnTx7LzNr1cPEcWLPv300wMmnSetpP+vvho1pQq+qwTGPVGnZcul5+Ll Mlozb2wQhsKyjkwF7cKQJMDWTzr7C8yY8Jeuca5ZcT2aLoiXwcmVyPe+eBvENq7Axa6vRp/b0Hfr xgfrFWUnzOS/hD860Ayf6s1TZWdd4+jpQ2WXG78jYvVLe3pBRXovLjmotE75WINWXzI0p1xYurYL fZKAotzCXx47tvEpKzqJXE/byRqfz4ftPGUHP27+gSK5HDBj5vrCjoPLG4fbl+aYNPsCYdJ5EII1 BfFmwig9Aie2cCHk7vZtf7pXWLeuUxcY2QvXsWwB/qRjvefep7/rEKSfgcIzWVoQ6C6tGuourRxq pY9yWT5tdSspzFBgVBInNcwpXyjCZ7H5yMx58YkH7mog+tQDc99cmoQ3JnjpukmP+5zyDSwQeFn7 5eIE+Nyza/zFlVe99tiutzD+3oc2vrT8R9OOwF/WKKRlIYazq5fAIzkd7rMX1swbMKog+PTN71uG 6er/HPeDvNKMcaw+WBqQCpSZllCIw2fpU248mCaaVbN6mZnFrsHNLY9N9XmyKXBXVq5/ARjwHOIn 042wtDa3bsnz5wz8eseJT5q+am+Mho2IL8+d0X9wVkVWcVoZpkP5IMgMNeQHfvVz25rL2OdbYrKj O1wPwGdmz8YGRiMxElCDC2blZe2desnI7F3v9xNMw5Q2eY53z9S5DYoEP5SghGKkQr3DD4/bJenp uGC2B0UsrC/ACysr5DFlhzwkJJ4Jj9Xj3e0LPgo/8NtHPJK0hJ12soYFdQT9DT+u5oA/vXJ07Mxh SlNDmav5ZJsEjVaD78R09CnJEd0+tn8WL9J6lxVcXnc62415ko8QneZN9x6cMn+U2bJZW8UfUGOb RYdlYh99M8vmPrJvv/BXbphsY5+Vb5LbOKCuX2PFkL6pysU0MvZ9VqNEPR4nHadS5SNEx08WcHTh 73hiEFK1PVFJnx9dtjDl5QTnTv4M3nHFc8FVT70L+0/P+ERpknX6gnkAo2jlx8QpXnSx05nkggWh vH993plkHBAyB8pFXFbEX3fHbI+D+zdPAxszTOyIFV7oqBU7QEV9wUT4a1SlwPO1sElQjdvEayG1 l9DImKlCD4w1aDBEspMuSge+9uiiQQLDCCcIn63YteHFDY8RfSoQdUe3nrAaMS9mdabCbtHg6dyP Xj7nXPj2/V8MRcIlpyWLygdNyT+wvnCMUM/ZLL2bGmC6AL1TeTAfhqb96p4Z65+6t1NLnBmeHcsu 2aDf/+wfXLL4HTJGPLKzJLCXKWqwoUdMRrofJh4ibE1gh+M5I16CcMHoHmHZwgPEQ5D2L/k7aQpg rxTG1nJYeZA88hIzw+mFkmQwfGjzck9B3VQB/kQU/NEGFsbaPVBTW4fPL2SEM3Mz6MoEa8osDVNU XHoRQ38psOzybTEJwQt85pcHQTIM58lBH4f9YhBq2QKWNgwIVo7h5rNNp3adwF+wgV2cxHKhCuBg 1G391Ac+zc3KhJA5ACwN8xVxQ3bCsOLLPsTWqFgw8Pw0zcQxdQwDiL5D1X4ZUPcPiy4/fUaJ5w6e qxYdDN2+4JxWNboUlBPGfLDptAntOOVXdLhijnstWSYyem7p/PRMT57VEUAWwwGGOiL7f3JB1w+b xcug9wfO37g9FFK/xnc6VkZIeFYfb/GI6YXfIfr7Z6x/WRO1u5gKIRA0yR4EeQBRc0gGGOuI1RMQ E4RTky9DAW0WnvrFcnX/xjoj5NPqIIifontw9sb3oaNcCEfCbWY7ouaVUCJdpENoxy1ibpSuAkOS NFPB8ImFsHz5GDsFeZxZII6Q7lRZMgHB0T3W04kcTPU17RftW1++z05v4TG6oSwhJNyiTIqgOUx0 Yo0ykpbyiSfCzXp0dhS2RnhaNBhziO2CjCZC5kGnFgQcHUsXIMwIN7efODafBdofUAzix1M4xNlp HPJj2QkiHu9RB0w/AOP0oatw2mfmC+/acbkoz+QhGC8zwTvjhRQkLdP1Dl7RT+xhOglxYTVysj0S vTC84vKrYC8p5Ts0dp2kgmMDVVdc8dNINNQQ1tRP+feYsELx20RceQRRSWFfzhvdyHUPnVg8A3VI nRshw2FkEDWj6x5DN8JYVFT9fUJ5ZhpDp5fOBjpr9L536ut3wl9uvVjU9FZqlAQTJcUuBpt1iXWG DqEmGq8dP9565iPf2biXh6b+dDmkKBu52egNfARBLnxR0ppEpCrxgVmvrw+3ddRCI343vq3G6xln TEznCM10OdSPRVX9+3fPfP1KrPNu0151Y2v78e2jw5r+S9YfgBoHSfxPA2Y8PzcM0EGw85mdLair HQE9srx92YJFybYfcJDBH/wm8vFpJHvHpR8aErxiQJBdNwC5EvygKxmf8ONF+9rCJ0aHItpvEqXP ZIEMhIk859H0kKb/rFVtmSI8fGtHfFpQF2HixTjEyRFO8djvEbfsAurFph+URbzQOyPEhxXDZWFR O3GKj9cLyzfoisqMdCFNikrCisUHImr012xdDAkT5BYTv/tsbOgItg2J3HnlC5SRfzaMrLzmk45v ttdDY3wE/lKGQR8MQ8WwdS/AaDT6niAewJOKhG7+70Y5C6oyyuDEsA0+T9pGkOGw6a26pGcSMnYT qEni0/DzGC7PhEyeiZcOyKo8a1V/Xi+mnPtmvf780eZQBWxCLwXlfwZZZ50KO5Y/x3XSnpw/3dGM 76wz61o7dKoXAtHI+PumvTZz7aV/73Z2aJdjx3P7+/fhJj/zoAsLAu7JcBy206aKPzzv7X13zVg/ EcbJc6GhbaBNXVY2EIIwkYdBZRfMWG5pPNFe/cCs9TGncN2m/cgjwY7ll16lRYL1IU1dG4IfJ/AR ms+qw2l+68pAu8cf5YYLOwVkRNf2dUT1h4KhQHlo2cLYPZf4RCHTaFj4vhLHWUHg6xjxpMneT2bl f4KdDFY3rLMhZDjIbs4u/DgZHwtfteRkYOXllzVrwQFBVXsQftAOdY5Jc1kEYzozdOqwGj0ZMYSf tQnRusAdl10ngDFPlM43eUUfhbVoG/OqDSIe53VdY2EIE3mURWl0ZOTuQYPDPDNgnYaGDA5B1C3X D9cz6ZsgxgVLKw9jH4Cb3bf4XBmDHvOK0pVcEfBre0M/AJfc7ov8eOHPexzVmJB/zsO9cu1kRVYe d8hSDa7eYU8B8iauD2iBK4RVi/FK//9X7ubnp1W6PEYd/A3NMlgSffzQeW9YlwqXvzLlEvgDjT49 ou98Zfuh93es2oEnW//r3dW/Pys3J8s9Fubwo2RJKoS/E5MNB3BOWROaoFsdh98ibodTzc0PXrhx 52kpzA03uBx5w4elyVJtSDdKAsGWNcJ9N/Gja4hzFoy4HTYyvogokX8Iy6/ZlWqa3tVPLaY5pA7D C+IIo6JiRFde8Z+pyHGv+Nkk+PbPoES0qiTsCK1atClRXMIw/JNnckGFLDhrHZJaJUSlDE2SPNAH IjAFbYT51wH4Psr28Iqrdv0r+6iw6qfwCW74k16nyx355Dg3TCTwjjV9vQ7H8IAkNgnHP94swNqX ov6lECrIYRSP1BWhSFPDnwqrr//8X5qffyf+bw38WwP/VA38N+jNHj2BlaDDAAAAAElFTkSuQmCC --Apple-Mail=_BE59B16D-0314-4D22-9919-D24757751487-- --Apple-Mail=_BCCB4ADA-6425-4260-B6F8-AB4F871F1AA7--