Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bAPzy-0005k4-IC for pgsql-performance@arkaria.postgresql.org; Tue, 07 Jun 2016 23:01:26 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.84_2) (envelope-from ) id 1bAPzx-0007i8-UO for pgsql-performance@arkaria.postgresql.org; Tue, 07 Jun 2016 23:01:25 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1b8Qe2-0006Xg-RQ for pgsql-performance@postgresql.org; Thu, 02 Jun 2016 11:18:34 +0000 Received: from mail-lf0-x243.google.com ([2a00:1450:4010:c07::243]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.84_2) (envelope-from ) id 1b8Qdx-0002MA-Vw for pgsql-performance@postgresql.org; Thu, 02 Jun 2016 11:18:33 +0000 Received: by mail-lf0-x243.google.com with SMTP id 65so4777796lfq.1 for ; Thu, 02 Jun 2016 04:18:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date:cc:message-id:references :to; bh=4qygrIJ3MslQsfhWZnLIvVXMeW0iZ6/DQlSPc5ffH08=; b=C9qb/OGa2Qt6N0LJLyQNvIo3nbUcceSWiUC70cfJoC/XUFZHFdQ+6MilFgMeIbS1C5 ye6feESbwx3NOeCx+hUvdlwgLgPGY+vLQg+cK5kqfQLAsmtxvNwkLe73V9BebDTXM/bc nGDcYT1OKANhlZ4sFhMGVeGKlqVMXK3m8q+UsDi/FtAUTBOyHMZdTt5Hk0LXCi+fTEq+ 3ueOcJ0AEK2j7fcP1bVynqsAutjBEaHU/0kDhy1jNCWeZ7XQxyXGY3KU3+QF1hnqfBxT GJ/MbvTsPvW1RJC08Q3XONse7f1Et5CVdkyXEBirWzxNIazgxSxBDjNtarz7I2NX1SlK olCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=4qygrIJ3MslQsfhWZnLIvVXMeW0iZ6/DQlSPc5ffH08=; b=UJUcI2RrHGunEkOQAuUzG/y0rzWKkO7WRKnrvIHAmELlQr6bGS7obk43gmZADZV35o qcMOAHNdIv+IU4tFSoF1amgEvc4kTIgTevnKWxzWgc2XD/dSVmYLVt1GgJIpC2flNjRp aq4v/bD1iySRQHrdogc3OJK2boBM8HEisH+OqVy8rHQt0YiHhmeJm1Bn6byAdtbCa0mW k5KDJgs6VA5EfVxV/nuboJLfc+qW/I88FAKCqeY6f1Hqkxk5kiwogsXkS5ycbO4guoWD IacG5PIvMAiHZm1N359jDEturRnm2mLNHXxIdn4CegNU9+muQIsX93a6207hlZgu/3Cn 5Vzw== X-Gm-Message-State: ALyK8tLoAxEg6GNaQAmaT+r8XGSd7ncJIvObc/bTO8xG6RVwKGfj9oq+m5q98zGjlhzJWg== X-Received: by 10.46.5.67 with SMTP id 64mr3707762ljf.12.1464866308276; Thu, 02 Jun 2016 04:18:28 -0700 (PDT) Received: from [10.77.58.255] ([82.196.139.3]) by smtp.gmail.com with ESMTPSA id h96sm4006160lji.21.2016.06.02.04.18.26 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 02 Jun 2016 04:18:27 -0700 (PDT) Content-Type: multipart/signed; boundary="Apple-Mail=_A6BB809F-79F1-4397-A0BB-73B958C4E7C4"; protocol="application/pkcs7-signature"; micalg=sha1 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: 9.4 -> 9.5 regression with queries through pgbouncer on RHEL 6 From: =?utf-8?B?0JDQvdGC0L7QvSDQkdGD0YjQvNC10LvQtdCy?= In-Reply-To: <26239E39-A838-4611-B7BA-0E96D2DF3253@simply.name> Date: Thu, 2 Jun 2016 14:18:26 +0300 Cc: =?utf-8?B?0JDQvdGC0L7QvSDQkdGD0YjQvNC10LvQtdCy?= , pgsql-performance@postgresql.org Message-Id: References: <26239E39-A838-4611-B7BA-0E96D2DF3253@simply.name> To: Vladimir Borodin X-Mailer: Apple Mail (2.3124) X-Pg-Spam-Score: -2.7 (--) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-performance Precedence: bulk Sender: pgsql-performance-owner@postgresql.org --Apple-Mail=_A6BB809F-79F1-4397-A0BB-73B958C4E7C4 Content-Type: multipart/alternative; boundary="Apple-Mail=_8E7E3C84-AA10-43F1-8990-877A935ECAF8" --Apple-Mail=_8E7E3C84-AA10-43F1-8990-877A935ECAF8 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 UP. repeat tests on local vm.. reults are discouraging OS PG TPS AVG latency Centos 7 9.5.3 23.711023 168.421 Centos 7 9.5.3 26.609271 150.188 Centos 7 9.5.3 25.220044 158.416 Centos 7 9.5.3 25.598977 156.047 Centos 7 9.4.8 278.572191 14.077 Centos 7 9.4.8 247.237755 16.177 Centos 7 9.4.8 240.007524 16.276 Centos 7 9.4.8 237.862238 16.596 ps: latest update on centos 7 +xfs + lates database version from repo, = no pgbouncer=20 > On 25 May 2016, at 17:33, Vladimir Borodin wrote: >=20 > Hi all. >=20 > We have found that queries through PgBouncer 1.7.2 (with transaction = pooling) to local PostgreSQL are almost two times slower in 9.5.3 than = in 9.4.8 on RHEL 6 hosts (all packages are updated to last versions). = Meanwhile the problem can=E2=80=99t be reproduced i.e. on Ubuntu 14.04 = (also fully-updated). >=20 > Here is how the results look like for 9.4, 9.5 and 9.6. All are built = from latest commits on yesterday in > * REL9_4_STABLE (a0cc89a28141595d888d8aba43163d58a1578bfb), > * REL9_5_STABLE (e504d915bbf352ecfc4ed335af934e799bf01053), > * master (6ee7fb8244560b7a3f224784b8ad2351107fa55d). >=20 > All of them are build on the host where testing is done (with stock = gcc versions). Sysctls, pgbouncer config and everything we found are the = same, postgres configs are default, PGDATA is in tmpfs. All numbers are = reproducible, they are stable between runs. >=20 > Shortly: >=20 > OS PostgreSQL version TPS = Avg. latency > RHEL 6 9.4 44898 = 1.425 ms > RHEL 6 9.5 26199 = 2.443 ms > RHEL 6 9.5 43027 = 1.487 ms > Ubuntu 14.04 9.4 67458 = 0.949 ms > Ubuntu 14.04 9.5 64065 = 0.999 ms > Ubuntu 14.04 9.6 64350 = 0.995 ms >=20 > You could see that the difference between major versions on Ubuntu is = not significant, but on RHEL 9.5 is 70% slower than 9.4 and 9.6. >=20 > Below are more details. >=20 > RHEL 6: >=20 > postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres = -T 60 -j 64 -c 64 -S -n 'host=3Dlocalhost port=3D6432 dbname=3Dpg94' > transaction type: SELECT only > scaling factor: 100 > query mode: simple > number of clients: 64 > number of threads: 64 > duration: 60 s > number of transactions actually processed: 2693962 > latency average: 1.425 ms > tps =3D 44897.461518 (including connections establishing) > tps =3D 44898.763258 (excluding connections establishing) > postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres = -T 60 -j 64 -c 64 -S -n 'host=3Dlocalhost port=3D6432 dbname=3Dpg95' > transaction type: SELECT only > scaling factor: 100 > query mode: simple > number of clients: 64 > number of threads: 64 > duration: 60 s > number of transactions actually processed: 1572014 > latency average: 2.443 ms > tps =3D 26198.928627 (including connections establishing) > tps =3D 26199.803363 (excluding connections establishing) > postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres = -T 60 -j 64 -c 64 -S -n 'host=3Dlocalhost port=3D6432 dbname=3Dpg96' > transaction type: SELECT only > scaling factor: 100 > query mode: simple > number of clients: 64 > number of threads: 64 > duration: 60 s > number of transactions actually processed: 2581645 > latency average: 1.487 ms > tps =3D 43025.676995 (including connections establishing) > tps =3D 43027.038275 (excluding connections establishing) > postgres@pgload05g ~ $ >=20 > Ubuntu 14.04 (the same hardware): >=20 > postgres@pgloadpublic02:~$ /usr/lib/postgresql/9.4/bin/pgbench -U = postgres -T 60 -j 64 -c 64 -S -n 'host=3Dlocalhost port=3D6432 = dbname=3Dpg94' > transaction type: SELECT only > scaling factor: 100 > query mode: simple > number of clients: 64 > number of threads: 64 > duration: 60 s > number of transactions actually processed: 4047653 > latency average: 0.949 ms > tps =3D 67458.361515 (including connections establishing) > tps =3D 67459.983480 (excluding connections establishing) > postgres@pgloadpublic02:~$ /usr/lib/postgresql/9.4/bin/pgbench -U = postgres -T 60 -j 64 -c 64 -S -n 'host=3Dlocalhost port=3D6432 = dbname=3Dpg95' > transaction type: SELECT only > scaling factor: 100 > query mode: simple > number of clients: 64 > number of threads: 64 > duration: 60 s > number of transactions actually processed: 3844084 > latency average: 0.999 ms > tps =3D 64065.447458 (including connections establishing) > tps =3D 64066.943627 (excluding connections establishing) > postgres@pgloadpublic02:~$ /usr/lib/postgresql/9.4/bin/pgbench -U = postgres -T 60 -j 64 -c 64 -S -n 'host=3Dlocalhost port=3D6432 = dbname=3Dpg96' > transaction type: SELECT only > scaling factor: 100 > query mode: simple > number of clients: 64 > number of threads: 64 > duration: 60 s > number of transactions actually processed: 3861088 > latency average: 0.995 ms > tps =3D 64348.573126 (including connections establishing) > tps =3D 64350.195750 (excluding connections establishing) > postgres@pgloadpublic02:~$ >=20 > In both tests (RHEL and Ubuntu) the bottleneck is performance of singe = CPU core which is 100% consumed by PgBouncer. If pgbench connects to = postgres directly I get the following (expected) numbers: >=20 > postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres = -T 60 -j 64 -c 64 -S -n 'host=3Dlocalhost port=3D5432' > transaction type: SELECT only > scaling factor: 100 > query mode: simple > number of clients: 64 > number of threads: 64 > duration: 60 s > number of transactions actually processed: 10010710 > latency average: 0.384 ms > tps =3D 166835.937859 (including connections establishing) > tps =3D 166849.730224 (excluding connections establishing) > postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres = -T 60 -j 64 -c 64 -S -n 'host=3Dlocalhost port=3D5433' > transaction type: SELECT only > scaling factor: 100 > query mode: simple > number of clients: 64 > number of threads: 64 > duration: 60 s > number of transactions actually processed: 13373890 > latency average: 0.287 ms > tps =3D 222888.311289 (including connections establishing) > tps =3D 222951.470125 (excluding connections establishing) > postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U postgres = -T 60 -j 64 -c 64 -S -n 'host=3Dlocalhost port=3D5434' > transaction type: SELECT only > scaling factor: 100 > query mode: simple > number of clients: 64 > number of threads: 64 > duration: 60 s > number of transactions actually processed: 12989816 > latency average: 0.296 ms > tps =3D 216487.458399 (including connections establishing) > tps =3D 216548.069976 (excluding connections establishing) > postgres@pgload05g ~ $ >=20 > Compilation options look almost the same: > # RHEL 6 > CFLAGS =3D -Wall -Wmissing-prototypes -Wpointer-arith = -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute = -Wformat-security -fno-strict-aliasing -fwrapv -g -O2 > # Ubuntu > CFLAGS =3D -Wall -Wmissing-prototypes -Wpointer-arith = -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute = -Wformat-security -fno-strict-aliasing -fwrapv = -fexcess-precision=3Dstandard -g -O2 >=20 > Attached are a simple script to deploy the testing environment = (PgBouncer should be installed) and pgbouncer config. I could provide = any other needed information like backtraces or perf reports or anything = else. >=20 > > >=20 > -- > May the force be with you=E2=80=A6 > https://simply.name --Apple-Mail=_8E7E3C84-AA10-43F1-8990-877A935ECAF8 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
UP. repeat tests on local = vm.. reults are discouraging
OS   =       PG     TPS AVG latency
Centos 7 9.5.3 23.711023 = = 168.421
Centos 7 9.5.3 = = 26.609271 = 150.188
Centos 7 9.5.3 = = 25.220044 = 158.416
Centos 7 9.5.3 = = 25.598977 = 156.047
Centos 7 9.4.8 = = 278.572191 14.077
Centos = 7 = 9.4.8 = 247.237755 16.177
Centos = 7 = 9.4.8 = 240.007524 16.276
Centos = 7 = 9.4.8 = 237.862238 16.596

ps: latest update on centos 7 +xfs + = lates database version from repo, no pgbouncer 


On 25 May 2016, at 17:33, = Vladimir Borodin <root@simply.name> wrote:

Hi all.

We have found that = queries through PgBouncer 1.7.2 (with transaction pooling) to local = PostgreSQL are almost two times slower in 9.5.3 than in 9.4.8 on RHEL 6 = hosts (all packages are updated to last versions). Meanwhile the problem = can=E2=80=99t be reproduced i.e. on Ubuntu 14.04 (also = fully-updated).

Here is how the results look like for 9.4, 9.5 and 9.6. All = are built from latest commits on yesterday in
= * REL9_4_STABLE = (a0cc89a28141595d888d8aba43163d58a1578bfb),
= * REL9_5_STABLE = (e504d915bbf352ecfc4ed335af934e799bf01053),
= * master = (6ee7fb8244560b7a3f224784b8ad2351107fa55d).

All of them are build on the host where = testing is done (with stock gcc versions). Sysctls, pgbouncer config and = everything we found are the same, postgres configs are default, PGDATA = is in tmpfs. All numbers are reproducible, they are stable between = runs.

Shortly:

OS PostgreSQL = version = TPS = Avg. latency
RHEL 6 = 9.4 = 44898 1.425 ms
RHEL 6 9.5 = 26199 2.443 ms
RHEL 6 = 9.5 = 43027 1.487 ms
Ubuntu 14.04 9.4 = 67458 = 0.949 ms
Ubuntu 14.04 9.5 = 64065 0.999 ms
Ubuntu 14.04 9.6 = 64350 = 0.995 ms

You could see that the difference between major versions on = Ubuntu is not significant, but on RHEL 9.5 is 70% slower than 9.4 and = 9.6.

Below are = more details.

RHEL 6:

postgres@pgload05g ~ $ = /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n = 'host=3Dlocalhost port=3D6432 dbname=3Dpg94'
transaction type: SELECT only
scaling = factor: 100
query mode: simple
number of clients: 64
number of threads: = 64
duration: 60 s
number of = transactions actually processed: 2693962
latency = average: 1.425 ms
tps =3D 44897.461518 (including = connections establishing)
tps =3D 44898.763258 = (excluding connections establishing)
postgres@pgload05g ~ $ /usr/lib/postgresql/9.4/bin/pgbench -U = postgres -T 60 -j 64 -c 64 -S -n 'host=3Dlocalhost port=3D6432 = dbname=3Dpg95'
transaction type: SELECT = only
scaling factor: 100
query = mode: simple
number of clients: 64
number of threads: 64
duration: 60 = s
number of transactions actually processed: = 1572014
latency average: 2.443 ms
tps =3D 26198.928627 (including connections = establishing)
tps =3D 26199.803363 (excluding = connections establishing)
postgres@pgload05g ~ $ = /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n = 'host=3Dlocalhost port=3D6432 dbname=3Dpg96'
transaction type: SELECT only
scaling = factor: 100
query mode: simple
number of clients: 64
number of threads: = 64
duration: 60 s
number of = transactions actually processed: 2581645
latency = average: 1.487 ms
tps =3D 43025.676995 (including = connections establishing)
tps =3D 43027.038275 = (excluding connections establishing)
postgres@pgload05g ~ $

Ubuntu 14.04 (the same = hardware):

postgres@pgloadpublic02:~$ = /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n = 'host=3Dlocalhost port=3D6432 dbname=3Dpg94'
transaction type: SELECT only
scaling = factor: 100
query mode: simple
number of clients: 64
number of threads: = 64
duration: 60 s
number of = transactions actually processed: 4047653
latency = average: 0.949 ms
tps =3D 67458.361515 (including = connections establishing)
tps =3D 67459.983480 = (excluding connections establishing)
postgres@pgloadpublic02:~$ = /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n = 'host=3Dlocalhost port=3D6432 dbname=3Dpg95'
transaction type: SELECT only
scaling = factor: 100
query mode: simple
number of clients: 64
number of threads: = 64
duration: 60 s
number of = transactions actually processed: 3844084
latency = average: 0.999 ms
tps =3D 64065.447458 (including = connections establishing)
tps =3D 64066.943627 = (excluding connections establishing)
postgres@pgloadpublic02:~$ = /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n = 'host=3Dlocalhost port=3D6432 dbname=3Dpg96'
transaction type: SELECT only
scaling = factor: 100
query mode: simple
number of clients: 64
number of threads: = 64
duration: 60 s
number of = transactions actually processed: 3861088
latency = average: 0.995 ms
tps =3D 64348.573126 (including = connections establishing)
tps =3D 64350.195750 = (excluding connections establishing)
postgres@pgloadpublic02:~$

In both tests (RHEL and Ubuntu) the = bottleneck is performance of singe CPU core which is 100% consumed by = PgBouncer. If pgbench connects to postgres directly I get the following = (expected) numbers:

postgres@pgload05g ~ $ = /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n = 'host=3Dlocalhost port=3D5432'
transaction type: = SELECT only
scaling factor: 100
query mode: simple
number of clients: = 64
number of threads: 64
duration: 60 s
number of transactions = actually processed: 10010710
latency average: 0.384 = ms
tps =3D 166835.937859 (including connections = establishing)
tps =3D 166849.730224 (excluding = connections establishing)
postgres@pgload05g ~ $ = /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n = 'host=3Dlocalhost port=3D5433'
transaction type: = SELECT only
scaling factor: 100
query mode: simple
number of clients: = 64
number of threads: 64
duration: 60 s
number of transactions = actually processed: 13373890
latency average: 0.287 = ms
tps =3D 222888.311289 (including connections = establishing)
tps =3D 222951.470125 (excluding = connections establishing)
postgres@pgload05g ~ $ = /usr/lib/postgresql/9.4/bin/pgbench -U postgres -T 60 -j 64 -c 64 -S -n = 'host=3Dlocalhost port=3D5434'
transaction type: = SELECT only
scaling factor: 100
query mode: simple
number of clients: = 64
number of threads: 64
duration: 60 s
number of transactions = actually processed: 12989816
latency average: 0.296 = ms
tps =3D 216487.458399 (including connections = establishing)
tps =3D 216548.069976 (excluding = connections establishing)
postgres@pgload05g ~ = $

Compilation = options look almost the same:
# = RHEL 6
CFLAGS =3D -Wall -Wmissing-prototypes = -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels = -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing = -fwrapv -g -O2
# Ubuntu
CFLAGS =3D -Wall -Wmissing-prototypes -Wpointer-arith = -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute = -Wformat-security -fno-strict-aliasing -fwrapv = -fexcess-precision=3Dstandard -g -O2

Attached are a = simple script to deploy the testing environment (PgBouncer should be = installed) and pgbouncer config. I could provide any other needed = information like backtraces or perf reports or anything else.

<pgbouncer.ini>
<deploy.sh>

--
May = the force be with you=E2=80=A6
https://simply.name


= --Apple-Mail=_8E7E3C84-AA10-43F1-8990-877A935ECAF8-- --Apple-Mail=_A6BB809F-79F1-4397-A0BB-73B958C4E7C4 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIGNDCCBjAw ggUYoAMCAQICAw/8IDANBgkqhkiG9w0BAQUFADCBjDELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0 YXJ0Q29tIEx0ZC4xKzApBgNVBAsTIlNlY3VyZSBEaWdpdGFsIENlcnRpZmljYXRlIFNpZ25pbmcx ODA2BgNVBAMTL1N0YXJ0Q29tIENsYXNzIDEgUHJpbWFyeSBJbnRlcm1lZGlhdGUgQ2xpZW50IENB MB4XDTE1MTEwNTIyNTA0NFoXDTE2MTEwNTE5NDY1NlowSDEfMB0GA1UEAwwWYnVzaG1lbGV2LmFh QGdtYWlsLmNvbTElMCMGCSqGSIb3DQEJARYWYnVzaG1lbGV2LmFhQGdtYWlsLmNvbTCCASIwDQYJ KoZIhvcNAQEBBQADggEPADCCAQoCggEBANg1oXn6dMqWBNfhPXZwUKeIa46wGB9okLKtoRrdGWVR /qAQeBG5G4TpqdAcf/rTGIXFLk88kIrG7rD7gUN558S8RMFHkcxHtZ2dtDO+wnj/mmb4OLCgmRdz 8zxIbHOYqgdKgGh8medTQ7m2DTYtHB03ZxYpO0Eov8O1Rys13+KT7iOdGCZr7WeicBCX8mwrS68B 2uxJH5J+8kHf0oo/afSZrXuMxaS68iXkcWV0cLRnXOt6PmI/3Lt8uBR3VB4hcLPY8bB1tInuOOTf GvKxsDUQTlrJURLmdKpdKYl1gGsmJOVntrP//lA0ALn6yOC45DLZKQCfmuNppEztjivm/N0CAwEA AaOCAtwwggLYMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgSwMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggr BgEFBQcDBDAdBgNVHQ4EFgQUVvfRuKFUn/OUCENPdJxqxnW6MN4wHwYDVR0jBBgwFoAUU3Ltkpzg 2ssBXHx+ljVO8tS4UYIwIQYDVR0RBBowGIEWYnVzaG1lbGV2LmFhQGdtYWlsLmNvbTCCAUwGA1Ud IASCAUMwggE/MIIBOwYLKwYBBAGBtTcBAgMwggEqMC4GCCsGAQUFBwIBFiJodHRwOi8vd3d3LnN0 YXJ0c3NsLmNvbS9wb2xpY3kucGRmMIH3BggrBgEFBQcCAjCB6jAnFiBTdGFydENvbSBDZXJ0aWZp Y2F0aW9uIEF1dGhvcml0eTADAgEBGoG+VGhpcyBjZXJ0aWZpY2F0ZSB3YXMgaXNzdWVkIGFjY29y ZGluZyB0byB0aGUgQ2xhc3MgMSBWYWxpZGF0aW9uIHJlcXVpcmVtZW50cyBvZiB0aGUgU3RhcnRD b20gQ0EgcG9saWN5LCByZWxpYW5jZSBvbmx5IGZvciB0aGUgaW50ZW5kZWQgcHVycG9zZSBpbiBj b21wbGlhbmNlIG9mIHRoZSByZWx5aW5nIHBhcnR5IG9ibGlnYXRpb25zLjA2BgNVHR8ELzAtMCug KaAnhiVodHRwOi8vY3JsLnN0YXJ0c3NsLmNvbS9jcnR1MS1jcmwuY3JsMIGOBggrBgEFBQcBAQSB gTB/MDkGCCsGAQUFBzABhi1odHRwOi8vb2NzcC5zdGFydHNzbC5jb20vc3ViL2NsYXNzMS9jbGll bnQvY2EwQgYIKwYBBQUHMAKGNmh0dHA6Ly9haWEuc3RhcnRzc2wuY29tL2NlcnRzL3N1Yi5jbGFz czEuY2xpZW50LmNhLmNydDAjBgNVHRIEHDAahhhodHRwOi8vd3d3LnN0YXJ0c3NsLmNvbS8wDQYJ KoZIhvcNAQEFBQADggEBAFpFATZjdd3MOOyexrz9TZFHzrn/6/d30fRYFJ2kPvMOKDZUtUL1gQUe KQdIg2S91nEX2hIg2EITYAsfRC89FVZb7A1XG7pmgyS3YoF0gufai+bgNQgshs68tkFreXK6IG3H Pf0bdGIJLtUvEo+FpqWUQmdmua7vcg2vsl9HiN6GggnK6KXqDPObwjLkZ5jIq6EF+hD/bWUZBDQU kaZG2K4I5/quqAxJiP9JVDPmmo/z/Ub0hIAQtoPiEYzyPm7RMPtW2qHE+VoX7rJmz7Y9r2majEMZ 22ziUPo5RmcroBVrohX+5us+nu+zPlxJsJYcdoZ8mg9sNUhMPMl0EF9qnAUxggNvMIIDawIBATCB lDCBjDELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKzApBgNVBAsTIlNlY3Vy ZSBEaWdpdGFsIENlcnRpZmljYXRlIFNpZ25pbmcxODA2BgNVBAMTL1N0YXJ0Q29tIENsYXNzIDEg UHJpbWFyeSBJbnRlcm1lZGlhdGUgQ2xpZW50IENBAgMP/CAwCQYFKw4DAhoFAKCCAa8wGAYJKoZI hvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTYwNjAyMTExODI2WjAjBgkqhkiG 9w0BCQQxFgQUayYcZXpBhVKA7k1XT4vsABQ6BCgwgaUGCSsGAQQBgjcQBDGBlzCBlDCBjDELMAkG A1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKzApBgNVBAsTIlNlY3VyZSBEaWdpdGFs IENlcnRpZmljYXRlIFNpZ25pbmcxODA2BgNVBAMTL1N0YXJ0Q29tIENsYXNzIDEgUHJpbWFyeSBJ bnRlcm1lZGlhdGUgQ2xpZW50IENBAgMP/CAwgacGCyqGSIb3DQEJEAILMYGXoIGUMIGMMQswCQYD VQQGEwJJTDEWMBQGA1UEChMNU3RhcnRDb20gTHRkLjErMCkGA1UECxMiU2VjdXJlIERpZ2l0YWwg Q2VydGlmaWNhdGUgU2lnbmluZzE4MDYGA1UEAxMvU3RhcnRDb20gQ2xhc3MgMSBQcmltYXJ5IElu dGVybWVkaWF0ZSBDbGllbnQgQ0ECAw/8IDANBgkqhkiG9w0BAQEFAASCAQCVPLBttxqHdlLoLhVs Sa+4AA6ToXKpqwMHkEd09XIk3Fq03Uv7A3+xz8GG3osVMj/UnQkkNHmS/pb63iUIRUNHfJ+u4Az1 RLi+O/oSdyEop1nFV/0bdq/GbBfETKuQswz0148RFNessMGFRUG0xVbBYg1gNd+KxYz/tViHeftm gQ3im7RhCtO5kkUNPxEsOWGXXqSyee+tVXbbJYKWdR5Sq1VZoDA6dVspqcCN+QcWLMG/lhzhofCJ jSaHm5I0moB7d3/OXGntxiPvZwJU76CK1WzrildANmpjArR54Q2UKrl7jczJZyG312wFgFjPhaEf cUPZSZ33Gp3vXxqDWEcDAAAAAAAA --Apple-Mail=_A6BB809F-79F1-4397-A0BB-73B958C4E7C4--