From henk53602@hotmail.com Tue Mar  10 14:09:10 2009
Received: from localhost (unknown [200.46.208.211])
	by mail.postgresql.org (Postfix) with ESMTP id EC53D634B26
	for <pgsql-performance-postgresql.org@mail.postgresql.org>;
	Tue, 10 Mar 2009 11:09:14 -0300 (ADT)
Received: from mail.postgresql.org ([200.46.204.86])
	by localhost (mx1.hub.org [200.46.208.211]) (amavisd-maia, port 10024)
	with ESMTP id 56498-05
	for <pgsql-performance-postgresql.org@mail.postgresql.org>;
	Tue, 10 Mar 2009 11:09:04 -0300 (ADT)
X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6
Received: from col0-omc4-s17.col0.hotmail.com (col0-omc4-s17.col0.hotmail.com
	[65.55.34.219])
	by mail.postgresql.org (Postfix) with ESMTP id 1E922634B18
	for <pgsql-performance@postgresql.org>;
	Tue, 10 Mar 2009 11:09:12 -0300 (ADT)
Received: from COL104-W43 ([65.55.34.201]) by col0-omc4-s17.col0.hotmail.com
	with Microsoft SMTPSVC(6.0.3790.3959);
	Tue, 10 Mar 2009 07:09:11 -0700
Message-ID: <COL104-W4301D2DDC445A99E55E086F5A10@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_7cf50bb5-99b5-4ab4-848b-b49b4877551b_"
X-Originating-IP: [87.215.139.10]
From: henk de wit <henk53602@hotmail.com>
To: <pgsql-performance@postgresql.org>
Subject: When does sequential performance matter in PG?
Date: Tue, 10 Mar 2009 15:09:10 +0100
Importance: Normal
MIME-Version: 1.0
X-OriginalArrivalTime: 10 Mar 2009 14:09:11.0200 (UTC)
	FILETIME=[C9066200:01C9A189]
X-Virus-Scanned: Maia Mailguard 1.0.1
X-Spam-Status: No, hits=0.001 tagged_above=0 required=5
	tests=HTML_MESSAGE=0.001
X-Spam-Level: 
X-Archive-Number: 200903/77
X-Sequence-Number: 32999

--_7cf50bb5-99b5-4ab4-848b-b49b4877551b_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


Hi=2C

It is frequently said that for PostgreSQL the number 1 thing to pay attenti=
on to when increasing performance is the amount of IOPS a storage system is=
 capable of. Now I wonder if there is any situation in which sequential IO =
performance comes into play. E.g. perhaps during a tablescan on a non-fragm=
ented table=2C or during a backup or restore?

The reason I'm asking is that we're building a storage array and for some r=
eason are unable to
increase the number of random IOPS beyond a certain threshold when we add m=
ore controllers or more (SSD) disks to the system. However=2C the sequentia=
l performance keeps increasing when we do that.

Would this extra sequential performance be of any benefit to PG or would it=
 just be wasted?

Kind regards

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE!
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/=

--_7cf50bb5-99b5-4ab4-848b-b49b4877551b_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<style>
.hmmessage P
{
margin:0px=3B
padding:0px
}
body.hmmessage
{
font-size: 10pt=3B
font-family:Verdana
}
</style>
</head>
<body class=3D'hmmessage'>
Hi=2C<br><br>It is frequently said that for PostgreSQL the number 1 thing t=
o pay attention to when increasing performance is the amount of IOPS a stor=
age system is capable of. Now I wonder if there is any situation in which s=
equential IO performance comes into play. E.g. perhaps during a tablescan o=
n a non-fragmented table=2C or during a backup or restore?<br><br>The reaso=
n I'm asking is that we're building a storage array and for some reason are=
 unable to
increase the number of random IOPS beyond a certain threshold when we add m=
ore controllers or more (SSD) disks to the system. However=2C the sequentia=
l performance keeps increasing when we do that.<br><br>Would this extra seq=
uential performance be of any benefit to PG or would it just be wasted?<br>=
<br>Kind regards<br><br /><hr />Express yourself instantly with MSN Messeng=
er! <a href=3D'http://clk.atdmt.com/AVE/go/onm00200471ave/direct/01/' targe=
t=3D'_new'>MSN Messenger</a></body>
</html>=

--_7cf50bb5-99b5-4ab4-848b-b49b4877551b_--



From matthew@flymine.org Tue Mar  10 14:28:07 2009
Received: from localhost (unknown [200.46.208.211])
	by mail.postgresql.org (Postfix) with ESMTP id C5DF86330F9
	for <pgsql-performance-postgresql.org@mail.postgresql.org>;
	Tue, 10 Mar 2009 11:28:13 -0300 (ADT)
Received: from mail.postgresql.org ([200.46.204.86])
	by localhost (mx1.hub.org [200.46.208.211]) (amavisd-maia, port 10024)
	with ESMTP id 60946-02
	for <pgsql-performance-postgresql.org@mail.postgresql.org>;
	Tue, 10 Mar 2009 11:28:01 -0300 (ADT)
X-Greylist: from auto-whitelisted by SQLgrey-1.7.6
Received: from ppsw-0.csi.cam.ac.uk (ppsw-0.csi.cam.ac.uk [131.111.8.130])
	by mail.postgresql.org (Postfix) with ESMTP id 627496327B7
	for <pgsql-performance@postgresql.org>;
	Tue, 10 Mar 2009 11:28:09 -0300 (ADT)
X-Cam-AntiVirus: no malware found
X-Cam-SpamDetails: not scanned
X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/
Received: from flymine.sysbiol.cam.ac.uk ([131.111.60.120]:37358
	helo=flymine.flymine.org)
	by ppsw-0.csi.cam.ac.uk (ppsw.cam.ac.uk [131.111.8.130]:25)
	with esmtp id 1Lh2wB-0000hG-0R (Exim 4.70)
	(return-path <matthew@flymine.org>); Tue, 10 Mar 2009 14:28:07 +0000
Received: from host103
	([192.168.128.103] helo=aragorn.flymine.org ident=Debian-exim)
	by flymine.flymine.org with esmtp (Exim 4.12)
	id 1Lh2wB-0004rM-00; Tue, 10 Mar 2009 14:28:07 +0000
Received: from mnw21 (helo=localhost)
	by aragorn.flymine.org with local-esmtp (Exim 4.69)
	(envelope-from <matthew@flymine.org>)
	id 1Lh2wB-0001Jt-1q; Tue, 10 Mar 2009 14:28:07 +0000
Date: Tue, 10 Mar 2009 14:28:07 +0000 (GMT)
From: Matthew Wakeling <matthew@flymine.org>
X-X-Sender: mnw21@aragorn.flymine.org
To: henk de wit <henk53602@hotmail.com>
cc: pgsql-performance@postgresql.org
Subject: Re: When does sequential performance matter in PG?
In-Reply-To: <COL104-W4301D2DDC445A99E55E086F5A10@phx.gbl>
Message-ID: <alpine.DEB.2.00.0903101425050.559@aragorn.flymine.org>
References: <COL104-W4301D2DDC445A99E55E086F5A10@phx.gbl>
User-Agent: Alpine 2.00 (DEB 1167 2008-08-23)
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
X-Virus-Scanned: Maia Mailguard 1.0.1
X-Archive-Number: 200903/78
X-Sequence-Number: 33000

On Tue, 10 Mar 2009, henk de wit wrote:
> It is frequently said that for PostgreSQL the number 1 thing to pay 
> attention to when increasing performance is the amount of IOPS a storage 
> system is capable of. Now I wonder if there is any situation in which 
> sequential IO performance comes into play. E.g. perhaps during a 
> tablescan on a non-fragmented table, or during a backup or restore?

Yes, up to a point. That point is when a single CPU can no longer handle 
the sequential transfer rate. Yes, there are some parallel restore 
possibilities which will get you further. Generally it only takes a few 
discs to max out a single CPU though.

> The reason I'm asking is that we're building a storage array and for 
> some reason are unable to increase the number of random IOPS beyond a 
> certain threshold when we add more controllers or more (SSD) disks to 
> the system. However, the sequential performance keeps increasing when we 
> do that.

Are you sure you're measuring the maximum IOPS, rather than measuring the 
IOPS capable in a single thread? The advantage of having more discs is 
that you can perform more operations in parallel, so if you have lots of 
simultaneous requests they can be spread over the disc array.

Matthew

-- 
 [About NP-completeness] These are the problems that make efficient use of
 the Fairy Godmother.                    -- Computer Science Lecturer



From henk53602@hotmail.com Tue Mar  10 14:57:03 2009
Received: from localhost (unknown [200.46.208.211])
	by mail.postgresql.org (Postfix) with ESMTP id 6DFC06339F3
	for <pgsql-performance-postgresql.org@mail.postgresql.org>;
	Tue, 10 Mar 2009 11:57:09 -0300 (ADT)
Received: from mail.postgresql.org ([200.46.204.86])
	by localhost (mx1.hub.org [200.46.208.211]) (amavisd-maia, port 10024)
	with ESMTP id 67796-02
	for <pgsql-performance-postgresql.org@mail.postgresql.org>;
	Tue, 10 Mar 2009 11:56:57 -0300 (ADT)
X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6
Received: from col0-omc4-s17.col0.hotmail.com (col0-omc4-s17.col0.hotmail.com
	[65.55.34.219])
	by mail.postgresql.org (Postfix) with ESMTP id 820E1632A91
	for <pgsql-performance@postgresql.org>;
	Tue, 10 Mar 2009 11:57:06 -0300 (ADT)
Received: from COL104-W31 ([65.55.34.199]) by col0-omc4-s17.col0.hotmail.com
	with Microsoft SMTPSVC(6.0.3790.3959);
	Tue, 10 Mar 2009 07:57:04 -0700
Message-ID: <COL104-W31005A94BBFDA5DA8B7ED0F5A10@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_57593139-db1d-4bfe-936b-d25c6f39984c_"
X-Originating-IP: [87.215.139.10]
From: henk de wit <henk53602@hotmail.com>
To: <matthew@flymine.org>
CC: <pgsql-performance@postgresql.org>
Subject: Re: When does sequential performance matter in PG?
Date: Tue, 10 Mar 2009 15:57:03 +0100
Importance: Normal
In-Reply-To: <alpine.DEB.2.00.0903101425050.559@aragorn.flymine.org>
References: <COL104-W4301D2DDC445A99E55E086F5A10@phx.gbl>
	<alpine.DEB.2.00.0903101425050.559@aragorn.flymine.org>
MIME-Version: 1.0
X-OriginalArrivalTime: 10 Mar 2009 14:57:04.0141 (UTC)
	FILETIME=[796E63D0:01C9A190]
X-Virus-Scanned: Maia Mailguard 1.0.1
X-Spam-Status: No, hits=0.001 tagged_above=0 required=5
	tests=HTML_MESSAGE=0.001
X-Spam-Level: 
X-Archive-Number: 200903/80
X-Sequence-Number: 33002

--_57593139-db1d-4bfe-936b-d25c6f39984c_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


Hi=2C

> On Tue=2C 10 Mar 2009=2C henk de wit wrote:
> > Now I wonder if there is any situation in which=20
> > sequential IO performance comes into play. E.g. perhaps during a=20
> > tablescan on a non-fragmented table=2C or during a backup or restore?
>=20
> Yes=2C up to a point. That point is when a single CPU can no longer handl=
e=20
> the sequential transfer rate. Yes=2C there are some parallel restore=20
> possibilities which will get you further. Generally it only takes a few=20
> discs to max out a single CPU though.

I see=2C but I take it you are only referring to a backup or a restore? It'=
s of course unlikely (even highly undesirable) that multiple processes are =
doing a backup=2C but it doesn't seem unlikely that multiple queries are do=
ing a table scan =3B)

> Are you sure you're measuring the maximum IOPS=2C rather than measuring t=
he=20
> IOPS capable in a single thread?

I'm pretty sure we're not testing the number of IOPS for a single thread=2C=
 as we're testing with 1=2C 10 and 40 threads. There is a significant (2x) =
increase in the total number of IOPS when going from 1 to 10 threads=2C but=
 no increase when going from 10 to 40 threads. You can read more details ab=
out the setup I used and the problems I ran into here: http://www.xtremesys=
tems.org/forums/showthread.php?p=3D3707365

Henk

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE!
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/=

--_57593139-db1d-4bfe-936b-d25c6f39984c_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<style>
.hmmessage P
{
margin:0px=3B
padding:0px
}
body.hmmessage
{
font-size: 10pt=3B
font-family:Verdana
}
</style>
</head>
<body class=3D'hmmessage'>
Hi=2C<br><br>&gt=3B On Tue=2C 10 Mar 2009=2C henk de wit wrote:<br>&gt=3B &=
gt=3B Now I wonder if there is any situation in which <br>&gt=3B &gt=3B seq=
uential IO performance comes into play. E.g. perhaps during a <br>&gt=3B &g=
t=3B tablescan on a non-fragmented table=2C or during a backup or restore?<=
br>&gt=3B <br>&gt=3B Yes=2C up to a point. That point is when a single CPU =
can no longer handle <br>&gt=3B the sequential transfer rate. Yes=2C there =
are some parallel restore <br>&gt=3B possibilities which will get you furth=
er. Generally it only takes a few <br>&gt=3B discs to max out a single CPU =
though.<br><br>I see=2C but I take it you are only referring to a backup or=
 a restore? It's of course unlikely (even highly undesirable) that multiple=
 processes are doing a backup=2C but it doesn't seem unlikely that multiple=
 queries are doing a table scan =3B)<br><br>&gt=3B Are you sure you're meas=
uring the maximum IOPS=2C rather than measuring the <br>&gt=3B IOPS capable=
 in a single thread?<br><br>I'm pretty sure we're not testing the number of=
 IOPS for a single thread=2C as we're testing with 1=2C 10 and 40 threads. =
There is a significant (2x) increase in the total number of IOPS when going=
 from 1 to 10 threads=2C but no increase when going from 10 to 40 threads. =
You can read more details about the setup I used and the problems I ran int=
o here: http://www.xtremesystems.org/forums/showthread.php?p=3D3707365<br><=
br>Henk<br><br /><hr />Express yourself instantly with MSN Messenger! <a hr=
ef=3D'http://clk.atdmt.com/AVE/go/onm00200471ave/direct/01/' target=3D'_new=
'>MSN Messenger</a></body>
</html>=

--_57593139-db1d-4bfe-936b-d25c6f39984c_--



From gsmith@gregsmith.com Tue Mar  10 17:50:05 2009
Received: from localhost (unknown [200.46.208.211])
	by mail.postgresql.org (Postfix) with ESMTP id 46649634863
	for <pgsql-performance-postgresql.org@mail.postgresql.org>;
	Tue, 10 Mar 2009 14:50:12 -0300 (ADT)
Received: from mail.postgresql.org ([200.46.204.86])
	by localhost (mx1.hub.org [200.46.208.211]) (amavisd-maia, port 10024)
	with ESMTP id 10790-06
	for <pgsql-performance-postgresql.org@mail.postgresql.org>;
	Tue, 10 Mar 2009 14:50:09 -0300 (ADT)
X-Greylist: from auto-whitelisted by SQLgrey-1.7.6
Received: from westnet.com (westnet.com [216.187.52.2])
	by mail.postgresql.org (Postfix) with ESMTP id 2AE9E633744
	for <pgsql-performance@postgresql.org>;
	Tue, 10 Mar 2009 14:50:07 -0300 (ADT)
Received: from westnet.com (localhost [127.0.0.1])
	by westnet.com (8.14.0/8.14.0) with ESMTP id n2AHo5lm020941;
	Tue, 10 Mar 2009 13:50:05 -0400 (EDT)
Received: from localhost (gsmith@localhost)
	by westnet.com (8.14.0/8.13.2/Submit) with ESMTP id n2AHo5NC020938;
	Tue, 10 Mar 2009 13:50:05 -0400 (EDT)
X-Authentication-Warning: westnet.com: gsmith owned process doing -bs
Date: Tue, 10 Mar 2009 13:50:05 -0400 (EDT)
From: Greg Smith <gsmith@gregsmith.com>
X-X-Sender: gsmith@westnet.com
To: henk de wit <henk53602@hotmail.com>
cc: pgsql-performance@postgresql.org
Subject: Re: When does sequential performance matter in PG?
In-Reply-To: <COL104-W4301D2DDC445A99E55E086F5A10@phx.gbl>
Message-ID: <alpine.GSO.2.01.0903101338560.14397@westnet.com>
References: <COL104-W4301D2DDC445A99E55E086F5A10@phx.gbl>
User-Agent: Alpine 2.01 (GSO 1184 2008-12-16)
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
X-Virus-Scanned: Maia Mailguard 1.0.1
X-Spam-Status: No, hits=0 tagged_above=0 required=5 tests=none
X-Spam-Level: 
X-Archive-Number: 200903/82
X-Sequence-Number: 33004

On Tue, 10 Mar 2009, henk de wit wrote:

> Now I wonder if there is any situation in which sequential IO 
> performance comes into play. E.g. perhaps during a tablescan on a 
> non-fragmented table, or during a backup or restore?

If you're doing a sequential scan of data that was loaded in a fairly 
large batch, you can approach reading at the sequential I/O rate of the 
drives.  Doing a backup using pg_dump is one situation where you might 
actually do that.

Unless your disk performance is really weak, restores in PostgreSQL are 
usually CPU bound right now.  There's a new parallel restore feature in 
8.4 that may make sequential write performance a more likely upper bound 
to run into, assuming your table structure is amenable to loading in 
parallel (situations with just one giant table won't benefit as much).

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD



From scott@richrelevance.com Tue Mar  10 18:01:00 2009
Received: from localhost (unknown [200.46.208.211])
	by mail.postgresql.org (Postfix) with ESMTP id 6C2C06348BC
	for <pgsql-performance-postgresql.org@mail.postgresql.org>;
	Tue, 10 Mar 2009 15:01:18 -0300 (ADT)
Received: from mail.postgresql.org ([200.46.204.86])
	by localhost (mx1.hub.org [200.46.208.211]) (amavisd-maia, port 10024)
	with ESMTP id 14478-01
	for <pgsql-performance-postgresql.org@mail.postgresql.org>;
	Tue, 10 Mar 2009 15:01:10 -0300 (ADT)
X-Greylist: from auto-whitelisted by SQLgrey-1.7.6
Received: from EXHUB018-2.exch018.msoutlookonline.net
	(exhub018-2.exch018.msoutlookonline.net [64.78.17.17])
	by mail.postgresql.org (Postfix) with ESMTP id 24BA4634B8C
	for <pgsql-performance@postgresql.org>;
	Tue, 10 Mar 2009 15:01:02 -0300 (ADT)
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
	EXHUB018-2.exch018.msoutlookonline.net ([64.78.17.17]) with mapi;
	Tue, 10 Mar 2009 11:01:01 -0700
From: Scott Carey <scott@richrelevance.com>
To: Matthew Wakeling <matthew@flymine.org>,
 henk de wit <henk53602@hotmail.com>
CC: "pgsql-performance@postgresql.org" <pgsql-performance@postgresql.org>
Date: Tue, 10 Mar 2009 11:01:00 -0700
Subject: Re: When does sequential performance matter in PG?
Thread-Topic: [PERFORM] When does sequential performance matter in PG?
Thread-Index: AcmhjH1/oq84KqVuRK6YpbrBokQKzQAHawIJ
Message-ID: <C5DBF8E9.3248%scott@richrelevance.com>
In-Reply-To: <alpine.DEB.2.00.0903101425050.559@aragorn.flymine.org>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C5DBF8E93248scottrichrelevancecom_"
MIME-Version: 1.0
X-Virus-Scanned: Maia Mailguard 1.0.1
X-Spam-Status: No, hits=0.001 tagged_above=0 required=5
	tests=HTML_MESSAGE=0.001
X-Spam-Level: 
X-Archive-Number: 200903/83
X-Sequence-Number: 33005

--_000_C5DBF8E93248scottrichrelevancecom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable




On 3/10/09 6:28 AM, "Matthew Wakeling" <matthew@flymine.org> wrote:

On Tue, 10 Mar 2009, henk de wit wrote:
> It is frequently said that for PostgreSQL the number 1 thing to pay
> attention to when increasing performance is the amount of IOPS a storage
> system is capable of. Now I wonder if there is any situation in which
> sequential IO performance comes into play. E.g. perhaps during a
> tablescan on a non-fragmented table, or during a backup or restore?

Yes, up to a point. That point is when a single CPU can no longer handle
the sequential transfer rate. Yes, there are some parallel restore
possibilities which will get you further. Generally it only takes a few
discs to max out a single CPU though.

This is not true if  you have concurrent sequential scans.  Then an array c=
an be tuned for total throughput with concurrent access.  Single thread seq=
uential measurements are similarly useful to single thread random i/o measu=
rement - not really a test like the DB will act, but useful as a starting p=
oint for tuning.
I'm past the point where a single thread can not keep up with the disk on a=
 sequential scan.  For the most simple select * queries, this is ~ 800MB/se=
c for me.
For any queries those with more complicated processing/filtering, its much =
less, usually 400MB/sec is a pretty good rate for a single thread.
However our raw array does about 1200MB/sec, and can get 75% efficiency on =
this or so with between 4 and 8 concurrent sequential scans.  It took some =
significant tuning and testing time to make sure this worked, and to balanc=
e that with random i/o requirements.

Furthermore, higher sequential rates help your random IOPS when you have se=
quential access concurrent with random access.  You can tune OS parameters =
(readahead in linux, I/O scheduler types) to bias throughput or latency tow=
ards random iops throughput or sequential MB/sec throughput.  Having faster=
 sequential disk access means less % of time doing sequential I/O, meaning =
more time left for random I/O.  It only goes so far, but it does help with =
mixed loads.

Overall, it depends a lot on how important sequential scans are to your use=
 case.




--_000_C5DBF8E93248scottrichrelevancecom_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<HTML>
<HEAD>
<TITLE>Re: [PERFORM] When does sequential performance matter in PG?</TITLE>
</HEAD>
<BODY>
<FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'font-size:=
11pt'><BR>
<BR>
<BR>
On 3/10/09 6:28 AM, &quot;Matthew Wakeling&quot; &lt;<a href=3D"matthew@fly=
mine.org">matthew@flymine.org</a>&gt; wrote:<BR>
<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"=
><SPAN STYLE=3D'font-size:11pt'>On Tue, 10 Mar 2009, henk de wit wrote:<BR>
&gt; It is frequently said that for PostgreSQL the number 1 thing to pay<BR=
>
&gt; attention to when increasing performance is the amount of IOPS a stora=
ge<BR>
&gt; system is capable of. Now I wonder if there is any situation in which<=
BR>
&gt; sequential IO performance comes into play. E.g. perhaps during a<BR>
&gt; tablescan on a non-fragmented table, or during a backup or restore?<BR=
>
<BR>
Yes, up to a point. That point is when a single CPU can no longer handle<BR=
>
the sequential transfer rate. Yes, there are some parallel restore<BR>
possibilities which will get you further. Generally it only takes a few<BR>
discs to max out a single CPU though.<BR>
<BR>
</SPAN></FONT></BLOCKQUOTE><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial=
"><SPAN STYLE=3D'font-size:11pt'>This is not true if &nbsp;you have concurr=
ent sequential scans. &nbsp;Then an array can be tuned for total throughput=
 with concurrent access. &nbsp;Single thread sequential measurements are si=
milarly useful to single thread random i/o measurement &#8212; not really a=
 test like the DB will act, but useful as a starting point for tuning.<BR>
I&#8217;m past the point where a single thread can not keep up with the dis=
k on a sequential scan. &nbsp;For the most simple select * queries, this is=
 ~ 800MB/sec for me.<BR>
For any queries those with more complicated processing/filtering, its much =
less, usually 400MB/sec is a pretty good rate for a single thread. &nbsp;<B=
R>
However our raw array does about 1200MB/sec, and can get 75% efficiency on =
this or so with between 4 and 8 concurrent sequential scans. &nbsp;It took =
some significant tuning and testing time to make sure this worked, and to b=
alance that with random i/o requirements.<BR>
<BR>
Furthermore, higher sequential rates help your random IOPS when you have se=
quential access concurrent with random access. &nbsp;You can tune OS parame=
ters (readahead in linux, I/O scheduler types) to bias throughput or latenc=
y towards random iops throughput or sequential MB/sec throughput. &nbsp;Hav=
ing faster sequential disk access means less % of time doing sequential I/O=
, meaning more time left for random I/O. &nbsp;It only goes so far, but it =
does help with mixed loads. &nbsp;<BR>
<BR>
Overall, it depends a lot on how important sequential scans are to your use=
 case.<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"=
><SPAN STYLE=3D'font-size:11pt'><BR>
<BR>
<BR>
</SPAN></FONT></BLOCKQUOTE>
</BODY>
</HTML>


--_000_C5DBF8E93248scottrichrelevancecom_--




From mariel.cherkassky@gmail.com Thu Mar  28 10:10:05 2019
Received: from malur.postgresql.org ([217.196.149.56])
	by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256)
	(Exim 4.89)
	(envelope-from <pgsql-performance-owner+archive@lists.postgresql.org>)
	id 1h9RzL-00005T-Oq
	for pgsql-performance@arkaria.postgresql.org; Thu, 28 Mar 2019 10:10:23 +0000
Received: from localhost ([127.0.0.1] helo=malur.postgresql.org)
	by malur.postgresql.org with esmtp (Exim 4.89)
	(envelope-from <pgsql-performance-owner+archive@lists.postgresql.org>)
	id 1h9RzK-0003uu-Ah
	for pgsql-performance@arkaria.postgresql.org; Thu, 28 Mar 2019 10:10:22 +0000
Received: from makus.postgresql.org ([2001:4800:3e1:1::229])
	by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256)
	(Exim 4.89)
	(envelope-from <mariel.cherkassky@gmail.com>)
	id 1h9RzJ-0003un-SK
	for pgsql-performance@lists.postgresql.org; Thu, 28 Mar 2019 10:10:22 +0000
Received: from mail-lf1-x142.google.com ([2a00:1450:4864:20::142])
	by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256)
	(Exim 4.89)
	(envelope-from <mariel.cherkassky@gmail.com>)
	id 1h9RzH-0004AY-AD
	for pgsql-performance@postgresql.org; Thu, 28 Mar 2019 10:10:20 +0000
Received: by mail-lf1-x142.google.com with SMTP id 5so13541306lft.12
        for <pgsql-performance@postgresql.org>;
 Thu, 28 Mar 2019 03:10:18 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20161025;
        h=mime-version:from:date:message-id:subject:to;
        bh=ZREZcGcputvr9FGHYnIC7Aur9CQwh4odUrd3JGpA2CQ=;
        b=CXpk7aBX9ytOSBdb1/EY9BaPR85S5sXGnaOk/y3X4+0tQPba2sidCRPCCga5PnR6Ur
         BSnfZbF4T5GZnolH8ToWV5wOOQ0t5sniF54gZ+mptutrBWobv5QJ6X1uzJsD7JpoXvMS
         DoKw70fXU6yiu3jt52GoRX9UtlW3/wssvefznbgJhT2njRy6uxrViyZH7b8v0RvRl8GL
         d6PSSaUyITt34VCo2zCdVxO5ckn2bqZHmGf90nvq6bQqCfD0gIkQRYb8vXpfDOfpdtat
         H6dciT4sNNpy4zIky/s8qsa7LqYAmGL8A+QDVZlwroqCiEGJwg/LIjiZriSxdujPvEDL
         xvgQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:mime-version:from:date:message-id:subject:to;
        bh=ZREZcGcputvr9FGHYnIC7Aur9CQwh4odUrd3JGpA2CQ=;
        b=aqlQIuxdz39oOjroWR0WqHRzfmHjtqrmDUkgC7731HjV80g6duvyDPcoYj7ENk1Bfi
         ZmhZDbQ6WSZdeGgd8dyV+ZU/qHxVD5NodseG9IFWeG0rPK4bgrwyeebaHRvNPJvDDggl
         ApzC7Y64FW1dIOqCWSOqavxLhhw8uMNDEsgsN2qRQcfZjj5UwYJmSLUSQkwd0ijwzO11
         PAuJXo9V8AZnUa9qtplzcgdEGCbBT6bJDLzI6Q+Y6CBS2v5CT8g98GTrEJpKPJC2zs7p
         uNBbi2akgWoo2z3MMKT2IeoOWlkGUdNAwNv+5A4qIAUcKWf3orR+WAcahGyU/BI8d2Ab
         Ttdg==
X-Gm-Message-State: APjAAAVYNSFfHsHDVFAO2bUA/FzetPf2h7AV1eWSpgaRhQ4Afi3QYyt3
	TtL7QGzXgO3RirHair8wv0Afiy4WZR54TuBxMy1KzA==
X-Google-Smtp-Source: 
 APXvYqwxgBKtGTDZMC/1mRjH2QiPSi8cF+ceXuWvcGGfEcIb+w+YxmuJei8s/TouPObT/QVURsfEQey60HqDYnro50o=
X-Received: by 2002:ac2:5205:: with SMTP id a5mr22453550lfl.102.1553767817113;
 Thu, 28 Mar 2019 03:10:17 -0700 (PDT)
MIME-Version: 1.0
From: Mariel Cherkassky <mariel.cherkassky@gmail.com>
Date: Thu, 28 Mar 2019 12:10:05 +0200
Message-ID: 
 <CA+t6e1nLGxZAGT0bPrGY1r7ag+nGfv+PGcYt1WKsEMNhNNkQrQ@mail.gmail.com>
Subject: Scale out postgresql
To: PostgreSQL mailing lists <pgsql-performance@postgresql.org>
Content-Type: multipart/alternative; boundary="000000000000cd7774058524c2fc"
List-Id: <pgsql-performance.lists.postgresql.org>
List-Help: <https://lists.postgresql.org/manage/>
List-Subscribe: <https://lists.postgresql.org/manage/>
List-Post: <mailto:pgsql-performance@lists.postgresql.org>
List-Owner: <mailto:pgsql-performance-owner@lists.postgresql.org>
List-Archive: <https://www.postgresql.org/list/pgsql-performance>
Precedence: bulk

--000000000000cd7774058524c2fc
Content-Type: text/plain; charset="UTF-8"

Hey,
I was searching for a solution to scale my postgresql instance in the
cloud. I'm aware of that that I can create many read only replicas in the
cloud and it would improve my reading performance. I wanted to hear what
solution are you familiar with ? Are there any sharding solution that are
commonly used (citus ? pg_shard ?) My instance has many dbs (one per
customer) and big customers can generate a load of load on others..



Thanks.

--000000000000cd7774058524c2fc
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"rtl"><div style=3D"" dir=3D"ltr">Hey,</div><div style=3D"" dir=
=3D"ltr">I was searching for a solution to scale my postgresql instance in =
the cloud. I&#39;m aware of that that I can create many read only replicas =
in the cloud and it would improve my reading performance. I wanted to hear =
what solution are you familiar with ? Are there any sharding solution that =
are commonly used (citus ? pg_shard ?) My instance has many dbs (one per cu=
stomer) and big customers can generate a load of load on others..</div><div=
 style=3D"" dir=3D"ltr"><br></div><div style=3D"" dir=3D"ltr"><br></div><di=
v style=3D"" dir=3D"ltr"><br></div><div style=3D"" dir=3D"ltr">Thanks.</div=
></div>

--000000000000cd7774058524c2fc--





From samruohola@yahoo.com Thu Mar  28 16:10:26 2019
Received: from malur.postgresql.org ([217.196.149.56])
	by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256)
	(Exim 4.89)
	(envelope-from <pgsql-performance-owner+archive@lists.postgresql.org>)
	id 1h9Xby-0007GH-US
	for pgsql-performance@arkaria.postgresql.org; Thu, 28 Mar 2019 16:10:39 +0000
Received: from localhost ([127.0.0.1] helo=malur.postgresql.org)
	by malur.postgresql.org with esmtp (Exim 4.89)
	(envelope-from <pgsql-performance-owner+archive@lists.postgresql.org>)
	id 1h9Xbx-0001zL-Na
	for pgsql-performance@arkaria.postgresql.org; Thu, 28 Mar 2019 16:10:37 +0000
Received: from makus.postgresql.org ([2001:4800:3e1:1::229])
	by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256)
	(Exim 4.89)
	(envelope-from <samruohola@yahoo.com>)
	id 1h9Xbx-0001yc-5u
	for pgsql-performance@lists.postgresql.org; Thu, 28 Mar 2019 16:10:37 +0000
Received: from sonic303-2.consmr.mail.bf2.yahoo.com ([74.6.131.41])
	by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256)
	(Exim 4.89)
	(envelope-from <samruohola@yahoo.com>)
	id 1h9Xbu-0004Wx-MW
	for pgsql-performance@postgresql.org; Thu, 28 Mar 2019 16:10:36 +0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048;
 t=1553789432; bh=TOgGGQiKX8L6CcadnyMLUHjKc3BvIxtWCN27pyI6yFM=;
 h=Date:From:Reply-To:To:In-Reply-To:References:Subject:From:Subject;
 b=JU5jumJ7T4I3+g8lIXQL0Yf7vKZY0q02LDUasOiI+6TMQ2YFVAzTmU9Kw2X+H8yjf0RYEL4A3RBKdNuNIDdxmnWqvt3uM8VZVRs6wivtFjWBlwC/f2rHRvfruJPSMMp/jtBirLncgFX17x1YLmLUYRpElAOcrBUL9+e7FXQegQbgstHVT0MauCVmkWN2jXfauaRGjb/8aldynJGdXDhm3MaQDF5GrFpGUCP6hh8l/+Ihk0+IozP4PyqQ2sUakT3SOT/FNWeyz9S/rAMbD/tZ0BVbTvIr6TGE98shT0VovGQcswGsIN+rX9/ozqMOscUoCKqB4pR2oXeKAIGnuTCgJw==
X-YMail-OSG: ClZXimYVM1kMTGiYDbl2zjTF6mJSaE3E2DBZ_jfxABm2FsxP2q3kz_bz3cEX184
 fLjDcaFfH.6dm0veFLfgocC1QY_KTGxGCXGKO08wWiBvKfN2Y42N5Ss1YRCQhjXbK4m_ZpeM4Tcl
 wl4pCsg.ON9aHkUgsQn7Tkbb14WtvjsFk4SYE6LkYtMPY0GSceCCyM31ojOOALL1.ZtuwtKwOOlq
 .Af4yEVWxlxq1trwXJvW_YXPdKixO.mTaHKQp0L2ywJRm1QmpGn5hK6i0UsVymLtAsiFlodvYeTf
 tfNCee6fmCXEyyZ_fACn82cInrnycJg2WxCNBA6Jo2xFX1X1qINjis2KQLoXabFdE10wAQpqC5Q6
 GUv5GQPKDa8yCgzZmAGK8QhUuL37TQmLhh3PnIZ9E.fGEtIzFUUy2j0V8Ueo8MvdgNhP.FvdFpLr
 DcqotjqU4gTx30f8e0cYXmNF.8voiWnMKtSnxa7ow9jBHXox0dU13l6K9urz3YifM7GbKQJBXu.z
 GSbT46fm4gGhBBoIUZSMZ5RTlyZIJAhPZmJ9dLv7aMGk.Y1thT9aDZ0QzQckKrSNovwTTolxKk87
 YwU7M2HhNyw.HWtEnsms2pfl3o0IHuyyPfFD8wM2.c84gsBzk10KDmgKHtjX5nYRAfyWC81_tpy1
 HesRlXGmFSv.yML.0Dyp5Is6vaLnrkfLSvJycpbxJzewYLGuxnIJyL1Z.BB_kMYW1xEWwglS7zWP
 509N7ADA3HZ9ljfLjgvJoIbNXPHJhiBlAf5PD6883ffJnYMvxLyyCpfu9c8EtzTVwSR93Zb2y8zo
 WjQr7R6PaWMLPZt98s3uuH7yDNxGqtc3fl2i0Ii8xBRaZfFnPzq5Mbe4p9AgP2ThG8a6uM_XmOsY
 rO_qnBjpJhI7Ap1Hl.KGFfEsQY130MSlseqkrUDrlwbDnLoOYtfoLEH1cpVmtHYYKAiDAGvBy5q_
 WMYDaQMlmCHfQCwMoo6n0b.RAYcJwUgEBEv_8OopjZctgSo5o__p8S6vb8OSFoO4Lw4QM5FT1ILG
 sL3uURsjKER72BoVTQ2nCyFJBY5E6gvNdvkSwwHNhofoLO8a8hWExnSR.NiJXeri0xXTgwsQ59_k
 cN1FE
Received: from sonic.gate.mail.ne1.yahoo.com by
 sonic303.consmr.mail.bf2.yahoo.com with HTTP; Thu, 28 Mar 2019 16:10:32 +0000
Date: Thu, 28 Mar 2019 16:10:26 +0000 (UTC)
From: "Sam R." <samruohola@yahoo.com>
Reply-To: "samruohola@yahoo.com" <samruohola@yahoo.com>
To: "mariel.cherkassky@gmail.com" <mariel.cherkassky@gmail.com>,
	PostgreSQL mailing lists <pgsql-performance@postgresql.org>
Message-ID: <2064609649.12651682.1553789426776@mail.yahoo.com>
In-Reply-To: 
 <CA+t6e1nLGxZAGT0bPrGY1r7ag+nGfv+PGcYt1WKsEMNhNNkQrQ@mail.gmail.com>
References: 
 <CA+t6e1nLGxZAGT0bPrGY1r7ag+nGfv+PGcYt1WKsEMNhNNkQrQ@mail.gmail.com>
Subject: Re: Scale out postgresql
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_12651681_843879007.1553789426774"
X-Mailer: WebService/1.1.13212 YahooMailAndroidMobile YMobile/1.0
 (com.yahoo.mobile.client.android.mail/5.38.6; Android/9; PPR1.180610.011;
 B2N_sprout; HMD Global; Nokia 7 plus; 5.71; 2034x1080;)
Content-Length: 4179
List-Id: <pgsql-performance.lists.postgresql.org>
List-Help: <https://lists.postgresql.org/manage/>
List-Subscribe: <https://lists.postgresql.org/manage/>
List-Post: <mailto:pgsql-performance@lists.postgresql.org>
List-Owner: <mailto:pgsql-performance-owner@lists.postgresql.org>
List-Archive: <https://www.postgresql.org/list/pgsql-performance>
Precedence: bulk

------=_Part_12651681_843879007.1553789426774
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi!
With following kinds of keywords, it is possible to find / search for cloud=
 native (SQL) implementations e.g. with google:=C2=A0
cloud native sql database
E.g. CockroachDB, YugaByteDB.
I do not know are you planning to do it by other means (by yourself).
I myself would be interested, has someone had experiences with such? Is HA =
provided "ready made? Is HA working fine and does it recover/handle all sit=
uations well, or is additional algorithms needed to be implemented in addit=
ion on top e.g. for automatic recovery (by "myself").
I could start an other email chain, if this chain is meant more for somethi=
ng else.
Best RegardsSam

=20
=20
  On to, maalisk. 28, 2019 at 12:10, Mariel Cherkassky<mariel.cherkassky@gm=
ail.com> wrote:   Hey,I was searching for a solution to scale my postgresql=
 instance in the cloud. I'm aware of that that I can create many read only =
replicas in the cloud and it would improve my reading performance. I wanted=
 to hear what solution are you familiar with ? Are there any sharding solut=
ion that are commonly used (citus ? pg_shard ?) My instance has many dbs (o=
ne per customer) and big customers can generate a load of load on others..


Thanks. =20

------=_Part_12651681_843879007.1553789426774
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi!<div id=3D"yMail_cursorElementTracker_1553788793985"><br></div><div id=
=3D"yMail_cursorElementTracker_1553788794353">With following kinds of keywo=
rds, it is possible to find / search for cloud native (SQL) implementations=
 e.g. with google:&nbsp;</div><div id=3D"yMail_cursorElementTracker_1553788=
844243"><br>cloud native sql database</div><div id=3D"yMail_cursorElementTr=
acker_1553788847928"><br></div><div id=3D"yMail_cursorElementTracker_155378=
8848427">E.g. CockroachDB, YugaByteDB.</div><div id=3D"yMail_cursorElementT=
racker_1553788973346"><br></div><div id=3D"yMail_cursorElementTracker_15537=
88973867">I do not know are you planning to do it by other means (by yourse=
lf).</div><div id=3D"yMail_cursorElementTracker_1553789037001"><br></div><d=
iv id=3D"yMail_cursorElementTracker_1553789037207">I myself would be intere=
sted, has someone had experiences with such? Is HA provided "ready made? Is=
 HA working fine and does it recover/handle all situations well, or is addi=
tional algorithms needed to be implemented in addition on top e.g. for auto=
matic recovery (by "myself").</div><div id=3D"yMail_cursorElementTracker_15=
53789168891"><br></div><div id=3D"yMail_cursorElementTracker_1553789169449"=
>I could start an other email chain, if this chain is meant more for someth=
ing else.</div><div id=3D"yMail_cursorElementTracker_1553788938238"><br></d=
iv><div id=3D"yMail_cursorElementTracker_1553788956490">Best Regards</div><=
div id=3D"yMail_cursorElementTracker_1553788962317">Sam<br><div id=3D"ymail=
_android_signature"><br></div> <br> <blockquote style=3D"margin: 0 0 20px 0=
;"> <div style=3D"font-family:Roboto, sans-serif; color:#6D00F6;"> <div>On =
to, maalisk. 28, 2019 at 12:10, Mariel Cherkassky</div><div>&lt;mariel.cher=
kassky@gmail.com&gt; wrote:</div> </div> <div style=3D"padding: 10px 0 0 20=
px; margin: 10px 0 0 0; border-left: 1px solid #6D00F6;"> <div id=3D"yiv549=
4098850"><div dir=3D"rtl"><div style=3D"" dir=3D"ltr">Hey,</div><div style=
=3D"" dir=3D"ltr">I was searching for a solution to scale my postgresql ins=
tance in the cloud. I'm aware of that that I can create many read only repl=
icas in the cloud and it would improve my reading performance. I wanted to =
hear what solution are you familiar with ? Are there any sharding solution =
that are commonly used (citus ? pg_shard ?) My instance has many dbs (one p=
er customer) and big customers can generate a load of load on others..</div=
><div style=3D"" dir=3D"ltr"><br></div><div style=3D"" dir=3D"ltr"><br></di=
v><div style=3D"" dir=3D"ltr"><br></div><div style=3D"" dir=3D"ltr">Thanks.=
</div></div></div> </div> </blockquote></div>
------=_Part_12651681_843879007.1553789426774--





From mariel.cherkassky@gmail.com Thu Mar  28 16:19:00 2019
Received: from malur.postgresql.org ([217.196.149.56])
	by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256)
	(Exim 4.89)
	(envelope-from <pgsql-performance-owner+archive@lists.postgresql.org>)
	id 1h9XkM-0007dc-TI
	for pgsql-performance@arkaria.postgresql.org; Thu, 28 Mar 2019 16:19:19 +0000
Received: from localhost ([127.0.0.1] helo=malur.postgresql.org)
	by malur.postgresql.org with esmtp (Exim 4.89)
	(envelope-from <pgsql-performance-owner+archive@lists.postgresql.org>)
	id 1h9XkL-0006tn-5U
	for pgsql-performance@arkaria.postgresql.org; Thu, 28 Mar 2019 16:19:17 +0000
Received: from makus.postgresql.org ([2001:4800:3e1:1::229])
	by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256)
	(Exim 4.89)
	(envelope-from <mariel.cherkassky@gmail.com>)
	id 1h9XkK-0006tg-Mu
	for pgsql-performance@lists.postgresql.org; Thu, 28 Mar 2019 16:19:16 +0000
Received: from mail-lj1-x241.google.com ([2a00:1450:4864:20::241])
	by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256)
	(Exim 4.89)
	(envelope-from <mariel.cherkassky@gmail.com>)
	id 1h9XkI-0004kJ-5T
	for pgsql-performance@postgresql.org; Thu, 28 Mar 2019 16:19:15 +0000
Received: by mail-lj1-x241.google.com with SMTP id j89so18222512ljb.1
        for <pgsql-performance@postgresql.org>;
 Thu, 28 Mar 2019 09:19:13 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20161025;
        h=mime-version:references:in-reply-to:from:date:message-id:subject:to
         :cc;
        bh=URX8cTJj3NIKNcVXhwl/l0BQbHa5F/T2YVMHTA+17FQ=;
        b=hgIYLxje6ddjl8+imHV+94j/LXs3YlaTAunRofyN/TqfVK1ASBq6l/bYwC1PXd1yEq
         jQkX3ZV3OlaNnh4MNpowhynxE15izuL1gdb0wgemYHLjJb1nwmOE4/OIw3889QvPTvfW
         A1Vqma1LbrvCsio/QcYXzpT7RbSZXiNOEKOROvSUe02U8RyC7dcBzH/catlzlSJzWT59
         kWPkacGh8yppQblrnpMnGi4X/bA7sf0RK1WSI9J9l9hdM6/whL97ypWP52c7afrtofEZ
         n1wpxlmq0SSaVhIeGjRyygBZZG+Olshg4JPIclaE9Nbsk24Mw0YohDsja+EC6XXHpY4B
         lIFg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:mime-version:references:in-reply-to:from:date
         :message-id:subject:to:cc;
        bh=URX8cTJj3NIKNcVXhwl/l0BQbHa5F/T2YVMHTA+17FQ=;
        b=hVHRx05JYOLapyGd7VX3qH6DYoKiAptjiP8mKVK1vYdZLvZjHYuvsdrNV1TJE2E4ki
         3qMAjuknRLT26vurjFsHn4ikOF53O9kez7AIQawqGtM8A0ljwGge7y2nN8s83vHERm2L
         cFzwfCQMLosNgw3zrvB7EEke0ikp/J4gV+l4TukB95xbR22L74M+d6Ae0SV7mUAcgh0R
         wfChphAPIo7HgY1ixbO/i1Gq8wW/wDmKwAniMtSpONG1kHDoaF2o8wUG3dWHVGdYU2/H
         tTSV7DLd/juD52fPw4xTLaoUVCAjhAcibMlxe6Vl7R4FqoWWrl6mcnopfqUdVk68x8Px
         M32g==
X-Gm-Message-State: APjAAAXx+9WiQ+PWhp/UJM4NF8RwvWVSqgPFOTyV+y1Fnk5l/NGZGGKP
	e/oHbzV/PfRZ6kU3QfHDHbt+oJzhf5+yJ84Akh0=
X-Google-Smtp-Source: 
 APXvYqxl2HjX60cx8Yj+VTxBreJJLh+BbU+g9b03+9EMglotHE5eGcKBIsusfYK2kg9NyK+zXTf129AQYnzuJRHltLk=
X-Received: by 2002:a2e:240a:: with SMTP id k10mr23712443ljk.31.1553789952433;
 Thu, 28 Mar 2019 09:19:12 -0700 (PDT)
MIME-Version: 1.0
References: 
 <CA+t6e1nLGxZAGT0bPrGY1r7ag+nGfv+PGcYt1WKsEMNhNNkQrQ@mail.gmail.com>
 <2064609649.12651682.1553789426776@mail.yahoo.com>
In-Reply-To: <2064609649.12651682.1553789426776@mail.yahoo.com>
From: Mariel Cherkassky <mariel.cherkassky@gmail.com>
Date: Thu, 28 Mar 2019 18:19:00 +0200
Message-ID: 
 <CA+t6e1m4H-P0s=s1Vthh6b4ePROxnkPQhLHUq3s0OC+aBnEM=Q@mail.gmail.com>
Subject: Re: Scale out postgresql
To: "samruohola@yahoo.com" <samruohola@yahoo.com>
Cc: PostgreSQL mailing lists <pgsql-performance@postgresql.org>
Content-Type: multipart/alternative; boundary="0000000000002ba6da058529ea7a"
List-Id: <pgsql-performance.lists.postgresql.org>
List-Help: <https://lists.postgresql.org/manage/>
List-Subscribe: <https://lists.postgresql.org/manage/>
List-Post: <mailto:pgsql-performance@lists.postgresql.org>
List-Owner: <mailto:pgsql-performance-owner@lists.postgresql.org>
List-Archive: <https://www.postgresql.org/list/pgsql-performance>
Precedence: bulk

--0000000000002ba6da058529ea7a
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hey Sam,
Are you familiar with scale solutions that arent in the cloud ?

=E2=80=AB=D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A =D7=99=D7=95=D7=9D =D7=94=D7=
=B3, 28 =D7=91=D7=9E=D7=A8=D7=A5 2019 =D7=91-18:10 =D7=9E=D7=90=D7=AA =E2=
=80=AASam R.=E2=80=AC=E2=80=8F <=E2=80=AAsamruohola@yahoo.com
=E2=80=AC=E2=80=8F>:=E2=80=AC

> Hi!
>
> With following kinds of keywords, it is possible to find / search for
> cloud native (SQL) implementations e.g. with google:
>
> cloud native sql database
>
> E.g. CockroachDB, YugaByteDB.
>
> I do not know are you planning to do it by other means (by yourself).
>
> I myself would be interested, has someone had experiences with such? Is H=
A
> provided "ready made? Is HA working fine and does it recover/handle all
> situations well, or is additional algorithms needed to be implemented in
> addition on top e.g. for automatic recovery (by "myself").
>
> I could start an other email chain, if this chain is meant more for
> something else.
>
> Best Regards
> Sam
>
>
> On to, maalisk. 28, 2019 at 12:10, Mariel Cherkassky
> <mariel.cherkassky@gmail.com> wrote:
> Hey,
> I was searching for a solution to scale my postgresql instance in the
> cloud. I'm aware of that that I can create many read only replicas in the
> cloud and it would improve my reading performance. I wanted to hear what
> solution are you familiar with ? Are there any sharding solution that are
> commonly used (citus ? pg_shard ?) My instance has many dbs (one per
> customer) and big customers can generate a load of load on others..
>
>
>
> Thanks.
>
>

--0000000000002ba6da058529ea7a
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"rtl"><div style=3D"" dir=3D"ltr">Hey Sam,</div><div style=3D"" =
dir=3D"ltr">Are you familiar with scale solutions that arent in the cloud ?=
</div></div><br><div class=3D"gmail_quote"><div dir=3D"rtl" class=3D"gmail_=
attr">=E2=80=AB=D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A =D7=99=D7=95=D7=9D =D7=
=94=D7=B3, 28 =D7=91=D7=9E=D7=A8=D7=A5 2019 =D7=91-18:10 =D7=9E=D7=90=D7=AA=
 =E2=80=AASam R.=E2=80=AC=E2=80=8F &lt;=E2=80=AA<a href=3D"mailto:samruohol=
a@yahoo.com">samruohola@yahoo.com</a>=E2=80=AC=E2=80=8F&gt;:=E2=80=AC<br></=
div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;bor=
der-left:1px solid rgb(204,204,204);padding-left:1ex">Hi!<div id=3D"gmail-m=
_5163663321805382429yMail_cursorElementTracker_1553788793985"><br></div><di=
v id=3D"gmail-m_5163663321805382429yMail_cursorElementTracker_1553788794353=
">With following kinds of keywords, it is possible to find / search for clo=
ud native (SQL) implementations e.g. with google:=C2=A0</div><div id=3D"gma=
il-m_5163663321805382429yMail_cursorElementTracker_1553788844243"><br>cloud=
 native sql database</div><div id=3D"gmail-m_5163663321805382429yMail_curso=
rElementTracker_1553788847928"><br></div><div id=3D"gmail-m_516366332180538=
2429yMail_cursorElementTracker_1553788848427">E.g. CockroachDB, YugaByteDB.=
</div><div id=3D"gmail-m_5163663321805382429yMail_cursorElementTracker_1553=
788973346"><br></div><div id=3D"gmail-m_5163663321805382429yMail_cursorElem=
entTracker_1553788973867">I do not know are you planning to do it by other =
means (by yourself).</div><div id=3D"gmail-m_5163663321805382429yMail_curso=
rElementTracker_1553789037001"><br></div><div id=3D"gmail-m_516366332180538=
2429yMail_cursorElementTracker_1553789037207">I myself would be interested,=
 has someone had experiences with such? Is HA provided &quot;ready made? Is=
 HA working fine and does it recover/handle all situations well, or is addi=
tional algorithms needed to be implemented in addition on top e.g. for auto=
matic recovery (by &quot;myself&quot;).</div><div id=3D"gmail-m_51636633218=
05382429yMail_cursorElementTracker_1553789168891"><br></div><div id=3D"gmai=
l-m_5163663321805382429yMail_cursorElementTracker_1553789169449">I could st=
art an other email chain, if this chain is meant more for something else.</=
div><div id=3D"gmail-m_5163663321805382429yMail_cursorElementTracker_155378=
8938238"><br></div><div id=3D"gmail-m_5163663321805382429yMail_cursorElemen=
tTracker_1553788956490">Best Regards</div><div id=3D"gmail-m_51636633218053=
82429yMail_cursorElementTracker_1553788962317">Sam<br><div id=3D"gmail-m_51=
63663321805382429ymail_android_signature"><br></div> <br> <blockquote style=
=3D"margin:0px 0px 20px"> <div style=3D"font-family:Roboto,sans-serif;color=
:rgb(109,0,246)"> <div>On to, maalisk. 28, 2019 at 12:10, Mariel Cherkassky=
</div><div>&lt;<a href=3D"mailto:mariel.cherkassky@gmail.com" target=3D"_bl=
ank">mariel.cherkassky@gmail.com</a>&gt; wrote:</div> </div> <div style=3D"=
padding:10px 0px 0px 20px;margin:10px 0px 0px;border-left:1px solid rgb(109=
,0,246)"> <div id=3D"gmail-m_5163663321805382429yiv5494098850"><div dir=3D"=
rtl"><div dir=3D"ltr">Hey,</div><div dir=3D"ltr">I was searching for a solu=
tion to scale my postgresql instance in the cloud. I&#39;m aware of that th=
at I can create many read only replicas in the cloud and it would improve m=
y reading performance. I wanted to hear what solution are you familiar with=
 ? Are there any sharding solution that are commonly used (citus ? pg_shard=
 ?) My instance has many dbs (one per customer) and big customers can gener=
ate a load of load on others..</div><div dir=3D"ltr"><br></div><div dir=3D"=
ltr"><br></div><div dir=3D"ltr"><br></div><div dir=3D"ltr">Thanks.</div></d=
iv></div> </div> </blockquote></div></blockquote></div>

--0000000000002ba6da058529ea7a--





From Sumedh.Pathak@microsoft.com Thu Mar  28 16:26:23 2019
Received: from malur.postgresql.org ([217.196.149.56])
	by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256)
	(Exim 4.89)
	(envelope-from <pgsql-performance-owner+archive@lists.postgresql.org>)
	id 1h9cM2-000075-PV
	for pgsql-performance@arkaria.postgresql.org; Thu, 28 Mar 2019 21:14:31 +0000
Received: from localhost ([127.0.0.1] helo=malur.postgresql.org)
	by malur.postgresql.org with esmtp (Exim 4.89)
	(envelope-from <pgsql-performance-owner+archive@lists.postgresql.org>)
	id 1h9cLz-00030o-Bw
	for pgsql-performance@arkaria.postgresql.org; Thu, 28 Mar 2019 21:14:27 +0000
Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29])
	by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256)
	(Exim 4.89)
	(envelope-from <Sumedh.Pathak@microsoft.com>)
	id 1h9XrL-0001nd-P6
	for pgsql-performance@lists.postgresql.org; Thu, 28 Mar 2019 16:26:32 +0000
Received: from mail-eopbgr780123.outbound.protection.outlook.com
 ([40.107.78.123] helo=NAM03-BY2-obe.outbound.protection.outlook.com)
	by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256)
	(Exim 4.89)
	(envelope-from <Sumedh.Pathak@microsoft.com>)
	id 1h9XrH-0002vM-Vz
	for pgsql-performance@postgresql.org; Thu, 28 Mar 2019 16:26:31 +0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=selector1;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=YOvKf8Foy3FR1guXwvwEqGKHb7BjHXMBaINSUI2RJ70=;
 b=fY0QPdX0sihXW4nvl3UFP0YseGnM3Jgitth8NtI9SAUCv5ZWH1NG74T5LZiMsUtjwtBghfnlVXZj2VlcXrRhsKdw0C+ty7mxpLrIJctpTIjCyyG3nHKdPnAa+J2oI2a51AHY/v5CtmDl3W3YfhEi4bdUYk4JrixGOJSq4a5DMmY=
ARC-Seal: i=1; a=rsa-sha256; s=testarcselector01; d=microsoft.com; cv=none;
 b=BfP8beL5yRCS2v7EvNpS+akH66ejddMm+gQz3ZBYO4hJwEEl3DNB4tzquTCWUVrmwGTEL7tCIsnherRAOW9PRzM1asYkYdMeQcfRc+fNtrY8Al8Cs2C9aYEIFKWwbSbkgEWnYiCEzBjwHSjY6akgSWc/BNaErPMDajzzbb1mpEk=
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=testarcselector01;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=YOvKf8Foy3FR1guXwvwEqGKHb7BjHXMBaINSUI2RJ70=;
 b=UlZaFXNwSfundwO8e7mwUCDMoaPvWENOn9haYh17v34qjdNT6M5B1YpzcqV7XjHAMyUa9jxx5wlwgNz3QSf/AEDtcRSvxseioyBZ5PGK+Gg6YyA6o0zUnwksHWmw3R1iWbxMWhhRaTr2XU/mCy78iP0vcD0rQGXfMj2OIRlx7hc=
ARC-Authentication-Results: 
	=?utf-8?B?aT0xOyB0ZXN0Lm9mZmljZTM2NS5jb20gMTsNCgkJZG1hcmM9bm9uZSBhY3Rp?=
 =?utf-8?B?b249bm9uZSBoZWFkZXIuZnJvbT1taWNyb3NvZnQuY29tOw0KCQlhcmM9bm9u?=
 =?utf-8?Q?e?=
Received: from MN2PR21MB1200.namprd21.prod.outlook.com (20.179.20.80) by
 MN2PR21MB1264.namprd21.prod.outlook.com (20.179.21.30) with Microsoft SMTP
 Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
 15.20.1771.2; Thu, 28 Mar 2019 16:26:24 +0000
Received: from MN2PR21MB1200.namprd21.prod.outlook.com
 ([fe80::1d9a:d1f8:4ef5:6482]) by MN2PR21MB1200.namprd21.prod.outlook.com
 ([fe80::1d9a:d1f8:4ef5:6482%7]) with mapi id 15.20.1750.002; Thu, 28 Mar 2019
 16:26:24 +0000
From: Sumedh Pathak <Sumedh.Pathak@microsoft.com>
To: Mariel Cherkassky <mariel.cherkassky@gmail.com>, "samruohola@yahoo.com"
	<samruohola@yahoo.com>
CC: PostgreSQL mailing lists <pgsql-performance@postgresql.org>
Subject: Re: Scale out postgresql
Thread-Topic: Scale out postgresql
Thread-Index: AQHU5U54AALa9wunPEC04P8f2YS7m6YhNuUAgAACZQCAAAGXUw==
Date: Thu, 28 Mar 2019 16:26:23 +0000
Message-ID: 
 <MN2PR21MB120014E7EDB452FE85965719EA590@MN2PR21MB1200.namprd21.prod.outlook.com>
References: 
 <CA+t6e1nLGxZAGT0bPrGY1r7ag+nGfv+PGcYt1WKsEMNhNNkQrQ@mail.gmail.com>
 <2064609649.12651682.1553789426776@mail.yahoo.com>,<CA+t6e1m4H-P0s=s1Vthh6b4ePROxnkPQhLHUq3s0OC+aBnEM=Q@mail.gmail.com>
In-Reply-To: 
 <CA+t6e1m4H-P0s=s1Vthh6b4ePROxnkPQhLHUq3s0OC+aBnEM=Q@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [2604:5500:c0a1:fa00:a151:f73b:715b:703a]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: b2e3d6dd-61b8-4fcf-84a6-08d6b39a1eb8
x-ms-office365-filtering-ht: Tenant
x-microsoft-antispam: 
 BCL:0;PCL:0;RULEID:(2390118)(7020095)(4652040)(8989299)(5600127)(711020)(4605104)(4618075)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(2017052603328)(7193020);SRVR:MN2PR21MB1264;
x-ms-traffictypediagnostic: MN2PR21MB1264:
x-ms-exchange-purlcount: 1
x-microsoft-antispam-prvs: 
 <MN2PR21MB12641409684DB3A4A7F418C7EA590@MN2PR21MB1264.namprd21.prod.outlook.com>
x-ms-oob-tlc-oobclassifiers: OLM:8882;
x-forefront-prvs: 0990C54589
x-forefront-antispam-report: 
 SFV:NSPM;SFS:(10019020)(366004)(346002)(396003)(136003)(376002)(39860400002)(199004)(189003)(2906002)(6436002)(53386004)(81166006)(106356001)(229853002)(256004)(6306002)(55016002)(54896002)(4326008)(7116003)(53936002)(6246003)(81156014)(68736007)(8990500004)(9686003)(10090500001)(71200400001)(19627405001)(8676002)(6116002)(236005)(71190400001)(105586002)(5070765005)(478600001)(52536014)(105004)(606006)(186003)(86362001)(72206003)(46003)(2501003)(74316002)(110136005)(5660300002)(14454004)(316002)(22452003)(25786009)(76176011)(3480700005)(33656002)(10290500003)(7696005)(6346003)(102836004)(6506007)(99286004)(11346002)(476003)(97736004)(446003)(8936002)(7736002)(53546011)(86612001)(486006);DIR:OUT;SFP:1102;SCL:1;SRVR:MN2PR21MB1264;H:MN2PR21MB1200.namprd21.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1;MX:1;
received-spf: None (protection.outlook.com: microsoft.com does not designate
 permitted sender hosts)
authentication-results: spf=none (sender IP is )
 smtp.mailfrom=Sumedh.Pathak@microsoft.com;
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam-message-info: 
 SLmNTCE2kq8IPRFW55LZWwX8ysNCb4W52nwGSV4L0zwYNOzKB9F2lA1imkelUtLC72ae3egjD3H/Yn3Jgo6P5F/t+eGt/eOSF7TB1U5JNPFBVu/0oQT8Njr7S/XZFFl3PwTUzsqt1OnkR17NpRYibEawAtTNzVBR4Wjfn6qb69ocy2kpgSJ322/WdXJCNRn2MrxhwVayWtetVkfmL1G58+mxfUASIGHfn5tGBtKmMSWc1T3PK6vVz7ICl4etxXS8mgGEtHqXxhM9CYEZCXVAjubRNeFVY/OPV8kupy9AMJofrzrVW0DiW6RMpMeQ6MaIRxeycLy36QJxl3R0iVCdPg3W2ftjORiUGVadcmQpUY4JknXACXNBT8mHnN7SbhjPzdnYKMk/uig/aRaTC7WUFspb8+SOHXh2IfYxyX6n4AY=
Content-Type: multipart/alternative;
	boundary="_000_MN2PR21MB120014E7EDB452FE85965719EA590MN2PR21MB1200namp_"
MIME-Version: 1.0
X-OriginatorOrg: microsoft.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 
 b2e3d6dd-61b8-4fcf-84a6-08d6b39a1eb8
X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Mar 2019 16:26:23.9367
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR21MB1264
List-Id: <pgsql-performance.lists.postgresql.org>
List-Help: <https://lists.postgresql.org/manage/>
List-Subscribe: <https://lists.postgresql.org/manage/>
List-Post: <mailto:pgsql-performance@lists.postgresql.org>
List-Owner: <mailto:pgsql-performance-owner@lists.postgresql.org>
List-Archive: <https://www.postgresql.org/list/pgsql-performance>
Precedence: bulk

--_000_MN2PR21MB120014E7EDB452FE85965719EA590MN2PR21MB1200namp_
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64

RGlzY2xhaW1lciwgSSB3b3JrIGZvciBDaXR1cyBEYXRhLg0KDQpZb3UgY2FuIGNoZWNrIG91dCB3
d3cuY2l0dXNkYXRhLmNvbTxodHRwOi8vd3d3LmNpdHVzZGF0YS5jb20+LiBXZSBhaW0gdG8gc2Nh
bGUgb3V0IFBvc3RncmVTUUwsIGFuZCBvZmZlciBpdCBib3RoIGluIHRoZSBjbG91ZCBhbmQgb24t
cHJlbS4gSGFwcHkgdG8gcHV0IHlvdSBpbiB0b3VjaCB3aXRoIHRoZSByaWdodCBwZW9wbGUgaWYg
eW91IGhhdmUgbW9yZSBxdWVzdGlvbnMuDQoNClRoYW5rcywNClN1bWVkaA0KDQpfX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fXw0KRnJvbTogTWFyaWVsIENoZXJrYXNza3kgPG1hcmllbC5j
aGVya2Fzc2t5QGdtYWlsLmNvbT4NClNlbnQ6IFRodXJzZGF5LCBNYXJjaCAyOCwgMjAxOSA5OjE5
IEFNDQpUbzogc2FtcnVvaG9sYUB5YWhvby5jb20NCkNjOiBQb3N0Z3JlU1FMIG1haWxpbmcgbGlz
dHMNClN1YmplY3Q6IFJlOiBTY2FsZSBvdXQgcG9zdGdyZXNxbA0KDQpIZXkgU2FtLA0KQXJlIHlv
dSBmYW1pbGlhciB3aXRoIHNjYWxlIHNvbHV0aW9ucyB0aGF0IGFyZW50IGluIHRoZSBjbG91ZCA/
DQoNCuKAq9eR16rXkNeo15nXmiDXmdeV150g15TXsywgMjgg15HXnteo16UgMjAxOSDXkS0xODox
MCDXnteQ16og4oCqU2FtIFIu4oCs4oCPIDzigKpzYW1ydW9ob2xhQHlhaG9vLmNvbTxtYWlsdG86
c2FtcnVvaG9sYUB5YWhvby5jb20+4oCs4oCPPjrigKwNCkhpIQ0KDQpXaXRoIGZvbGxvd2luZyBr
aW5kcyBvZiBrZXl3b3JkcywgaXQgaXMgcG9zc2libGUgdG8gZmluZCAvIHNlYXJjaCBmb3IgY2xv
dWQgbmF0aXZlIChTUUwpIGltcGxlbWVudGF0aW9ucyBlLmcuIHdpdGggZ29vZ2xlOg0KDQpjbG91
ZCBuYXRpdmUgc3FsIGRhdGFiYXNlDQoNCkUuZy4gQ29ja3JvYWNoREIsIFl1Z2FCeXRlREIuDQoN
CkkgZG8gbm90IGtub3cgYXJlIHlvdSBwbGFubmluZyB0byBkbyBpdCBieSBvdGhlciBtZWFucyAo
YnkgeW91cnNlbGYpLg0KDQpJIG15c2VsZiB3b3VsZCBiZSBpbnRlcmVzdGVkLCBoYXMgc29tZW9u
ZSBoYWQgZXhwZXJpZW5jZXMgd2l0aCBzdWNoPyBJcyBIQSBwcm92aWRlZCAicmVhZHkgbWFkZT8g
SXMgSEEgd29ya2luZyBmaW5lIGFuZCBkb2VzIGl0IHJlY292ZXIvaGFuZGxlIGFsbCBzaXR1YXRp
b25zIHdlbGwsIG9yIGlzIGFkZGl0aW9uYWwgYWxnb3JpdGhtcyBuZWVkZWQgdG8gYmUgaW1wbGVt
ZW50ZWQgaW4gYWRkaXRpb24gb24gdG9wIGUuZy4gZm9yIGF1dG9tYXRpYyByZWNvdmVyeSAoYnkg
Im15c2VsZiIpLg0KDQpJIGNvdWxkIHN0YXJ0IGFuIG90aGVyIGVtYWlsIGNoYWluLCBpZiB0aGlz
IGNoYWluIGlzIG1lYW50IG1vcmUgZm9yIHNvbWV0aGluZyBlbHNlLg0KDQpCZXN0IFJlZ2FyZHMN
ClNhbQ0KDQoNCk9uIHRvLCBtYWFsaXNrLiAyOCwgMjAxOSBhdCAxMjoxMCwgTWFyaWVsIENoZXJr
YXNza3kNCjxtYXJpZWwuY2hlcmthc3NreUBnbWFpbC5jb208bWFpbHRvOm1hcmllbC5jaGVya2Fz
c2t5QGdtYWlsLmNvbT4+IHdyb3RlOg0KSGV5LA0KSSB3YXMgc2VhcmNoaW5nIGZvciBhIHNvbHV0
aW9uIHRvIHNjYWxlIG15IHBvc3RncmVzcWwgaW5zdGFuY2UgaW4gdGhlIGNsb3VkLiBJJ20gYXdh
cmUgb2YgdGhhdCB0aGF0IEkgY2FuIGNyZWF0ZSBtYW55IHJlYWQgb25seSByZXBsaWNhcyBpbiB0
aGUgY2xvdWQgYW5kIGl0IHdvdWxkIGltcHJvdmUgbXkgcmVhZGluZyBwZXJmb3JtYW5jZS4gSSB3
YW50ZWQgdG8gaGVhciB3aGF0IHNvbHV0aW9uIGFyZSB5b3UgZmFtaWxpYXIgd2l0aCA/IEFyZSB0
aGVyZSBhbnkgc2hhcmRpbmcgc29sdXRpb24gdGhhdCBhcmUgY29tbW9ubHkgdXNlZCAoY2l0dXMg
PyBwZ19zaGFyZCA/KSBNeSBpbnN0YW5jZSBoYXMgbWFueSBkYnMgKG9uZSBwZXIgY3VzdG9tZXIp
IGFuZCBiaWcgY3VzdG9tZXJzIGNhbiBnZW5lcmF0ZSBhIGxvYWQgb2YgbG9hZCBvbiBvdGhlcnMu
Lg0KDQoNCg0KVGhhbmtzLg0K

--_000_MN2PR21MB120014E7EDB452FE85965719EA590MN2PR21MB1200namp_
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: base64

PGh0bWw+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIgY29udGVudD0i
dGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjxzdHlsZSB0eXBlPSJ0ZXh0L2NzcyIgc3R5bGU9
ImRpc3BsYXk6bm9uZTsiPiBQIHttYXJnaW4tdG9wOjA7bWFyZ2luLWJvdHRvbTowO30gPC9zdHls
ZT4NCjwvaGVhZD4NCjxib2R5IGRpcj0ibHRyIj4NCjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OiBU
YWhvbWEsIEdlbmV2YSwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxMHB0OyBjb2xvcjogcmdiKDAs
IDAsIDApOyI+DQpEaXNjbGFpbWVyLCBJIHdvcmsgZm9yIENpdHVzIERhdGEuPC9kaXY+DQo8ZGl2
IHN0eWxlPSJmb250LWZhbWlseTogVGFob21hLCBHZW5ldmEsIHNhbnMtc2VyaWY7IGZvbnQtc2l6
ZTogMTBwdDsgY29sb3I6IHJnYigwLCAwLCAwKTsiPg0KPGJyPg0KPC9kaXY+DQo8ZGl2IHN0eWxl
PSJmb250LWZhbWlseTogVGFob21hLCBHZW5ldmEsIHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTogMTBw
dDsgY29sb3I6IHJnYigwLCAwLCAwKTsiPg0KWW91IGNhbiBjaGVjayBvdXQgPGEgaHJlZj0iaHR0
cDovL3d3dy5jaXR1c2RhdGEuY29tIiBpZD0iTFBOb0xQMjkwMzExIj53d3cuY2l0dXNkYXRhLmNv
bTwvYT4uIFdlIGFpbSB0byBzY2FsZSBvdXQgUG9zdGdyZVNRTCwgYW5kIG9mZmVyIGl0IGJvdGgg
aW4gdGhlIGNsb3VkIGFuZCBvbi1wcmVtLiBIYXBweSB0byBwdXQgeW91IGluIHRvdWNoIHdpdGgg
dGhlIHJpZ2h0IHBlb3BsZSBpZiB5b3UgaGF2ZSBtb3JlIHF1ZXN0aW9ucy48L2Rpdj4NCjxkaXYg
c3R5bGU9ImZvbnQtZmFtaWx5OiBUYWhvbWEsIEdlbmV2YSwgc2Fucy1zZXJpZjsgZm9udC1zaXpl
OiAxMHB0OyBjb2xvcjogcmdiKDAsIDAsIDApOyI+DQo8YnI+DQo8L2Rpdj4NCjxkaXYgc3R5bGU9
ImZvbnQtZmFtaWx5OiBUYWhvbWEsIEdlbmV2YSwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxMHB0
OyBjb2xvcjogcmdiKDAsIDAsIDApOyI+DQpUaGFua3MsPC9kaXY+DQo8ZGl2IHN0eWxlPSJmb250
LWZhbWlseTogVGFob21hLCBHZW5ldmEsIHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTogMTBwdDsgY29s
b3I6IHJnYigwLCAwLCAwKTsiPg0KU3VtZWRoPC9kaXY+DQo8YnI+DQo8ZGl2IGlkPSJhcHBlbmRv
bnNlbmQiPjwvZGl2Pg0KPGhyIHN0eWxlPSJkaXNwbGF5OmlubGluZS1ibG9jazt3aWR0aDo5OCUi
IHRhYmluZGV4PSItMSI+DQo8ZGl2IGlkPSJkaXZScGx5RndkTXNnIiBkaXI9Imx0ciI+PGZvbnQg
ZmFjZT0iQ2FsaWJyaSwgc2Fucy1zZXJpZiIgc3R5bGU9ImZvbnQtc2l6ZToxMXB0IiBjb2xvcj0i
IzAwMDAwMCI+PGI+RnJvbTo8L2I+IE1hcmllbCBDaGVya2Fzc2t5ICZsdDttYXJpZWwuY2hlcmth
c3NreUBnbWFpbC5jb20mZ3Q7PGJyPg0KPGI+U2VudDo8L2I+IFRodXJzZGF5LCBNYXJjaCAyOCwg
MjAxOSA5OjE5IEFNPGJyPg0KPGI+VG86PC9iPiBzYW1ydW9ob2xhQHlhaG9vLmNvbTxicj4NCjxi
PkNjOjwvYj4gUG9zdGdyZVNRTCBtYWlsaW5nIGxpc3RzPGJyPg0KPGI+U3ViamVjdDo8L2I+IFJl
OiBTY2FsZSBvdXQgcG9zdGdyZXNxbDwvZm9udD4NCjxkaXY+Jm5ic3A7PC9kaXY+DQo8L2Rpdj4N
CjxkaXY+DQo8ZGl2IGRpcj0icnRsIj4NCjxkaXYgZGlyPSJsdHIiIHN0eWxlPSIiPkhleSBTYW0s
PC9kaXY+DQo8ZGl2IGRpcj0ibHRyIiBzdHlsZT0iIj5BcmUgeW91IGZhbWlsaWFyIHdpdGggc2Nh
bGUgc29sdXRpb25zIHRoYXQgYXJlbnQgaW4gdGhlIGNsb3VkID88L2Rpdj4NCjwvZGl2Pg0KPGJy
Pg0KPGRpdiBjbGFzcz0ieF9nbWFpbF9xdW90ZSI+DQo8ZGl2IGRpcj0icnRsIiBjbGFzcz0ieF9n
bWFpbF9hdHRyIj7igKvXkdeq15DXqNeZ15og15nXldedINeU17MsIDI4INeR157XqNelIDIwMTkg
15EtMTg6MTAg157XkNeqIOKAqlNhbSBSLuKArOKAjyAmbHQ74oCqPGEgaHJlZj0ibWFpbHRvOnNh
bXJ1b2hvbGFAeWFob28uY29tIj5zYW1ydW9ob2xhQHlhaG9vLmNvbTwvYT7igKzigI8mZ3Q7OuKA
rDxicj4NCjwvZGl2Pg0KPGJsb2NrcXVvdGUgY2xhc3M9InhfZ21haWxfcXVvdGUiIHN0eWxlPSJt
YXJnaW46MHB4IDBweCAwcHggMC44ZXg7IGJvcmRlci1sZWZ0OjFweCBzb2xpZCByZ2IoMjA0LDIw
NCwyMDQpOyBwYWRkaW5nLWxlZnQ6MWV4Ij4NCkhpIQ0KPGRpdiBpZD0ieF9nbWFpbC1tXzUxNjM2
NjMzMjE4MDUzODI0Mjl5TWFpbF9jdXJzb3JFbGVtZW50VHJhY2tlcl8xNTUzNzg4NzkzOTg1Ij48
YnI+DQo8L2Rpdj4NCjxkaXYgaWQ9InhfZ21haWwtbV81MTYzNjYzMzIxODA1MzgyNDI5eU1haWxf
Y3Vyc29yRWxlbWVudFRyYWNrZXJfMTU1Mzc4ODc5NDM1MyI+V2l0aCBmb2xsb3dpbmcga2luZHMg
b2Yga2V5d29yZHMsIGl0IGlzIHBvc3NpYmxlIHRvIGZpbmQgLyBzZWFyY2ggZm9yIGNsb3VkIG5h
dGl2ZSAoU1FMKSBpbXBsZW1lbnRhdGlvbnMgZS5nLiB3aXRoIGdvb2dsZTombmJzcDs8L2Rpdj4N
CjxkaXYgaWQ9InhfZ21haWwtbV81MTYzNjYzMzIxODA1MzgyNDI5eU1haWxfY3Vyc29yRWxlbWVu
dFRyYWNrZXJfMTU1Mzc4ODg0NDI0MyI+PGJyPg0KY2xvdWQgbmF0aXZlIHNxbCBkYXRhYmFzZTwv
ZGl2Pg0KPGRpdiBpZD0ieF9nbWFpbC1tXzUxNjM2NjMzMjE4MDUzODI0Mjl5TWFpbF9jdXJzb3JF
bGVtZW50VHJhY2tlcl8xNTUzNzg4ODQ3OTI4Ij48YnI+DQo8L2Rpdj4NCjxkaXYgaWQ9InhfZ21h
aWwtbV81MTYzNjYzMzIxODA1MzgyNDI5eU1haWxfY3Vyc29yRWxlbWVudFRyYWNrZXJfMTU1Mzc4
ODg0ODQyNyI+RS5nLiBDb2Nrcm9hY2hEQiwgWXVnYUJ5dGVEQi48L2Rpdj4NCjxkaXYgaWQ9Inhf
Z21haWwtbV81MTYzNjYzMzIxODA1MzgyNDI5eU1haWxfY3Vyc29yRWxlbWVudFRyYWNrZXJfMTU1
Mzc4ODk3MzM0NiI+PGJyPg0KPC9kaXY+DQo8ZGl2IGlkPSJ4X2dtYWlsLW1fNTE2MzY2MzMyMTgw
NTM4MjQyOXlNYWlsX2N1cnNvckVsZW1lbnRUcmFja2VyXzE1NTM3ODg5NzM4NjciPkkgZG8gbm90
IGtub3cgYXJlIHlvdSBwbGFubmluZyB0byBkbyBpdCBieSBvdGhlciBtZWFucyAoYnkgeW91cnNl
bGYpLjwvZGl2Pg0KPGRpdiBpZD0ieF9nbWFpbC1tXzUxNjM2NjMzMjE4MDUzODI0Mjl5TWFpbF9j
dXJzb3JFbGVtZW50VHJhY2tlcl8xNTUzNzg5MDM3MDAxIj48YnI+DQo8L2Rpdj4NCjxkaXYgaWQ9
InhfZ21haWwtbV81MTYzNjYzMzIxODA1MzgyNDI5eU1haWxfY3Vyc29yRWxlbWVudFRyYWNrZXJf
MTU1Mzc4OTAzNzIwNyI+SSBteXNlbGYgd291bGQgYmUgaW50ZXJlc3RlZCwgaGFzIHNvbWVvbmUg
aGFkIGV4cGVyaWVuY2VzIHdpdGggc3VjaD8gSXMgSEEgcHJvdmlkZWQgJnF1b3Q7cmVhZHkgbWFk
ZT8gSXMgSEEgd29ya2luZyBmaW5lIGFuZCBkb2VzIGl0IHJlY292ZXIvaGFuZGxlIGFsbCBzaXR1
YXRpb25zIHdlbGwsIG9yIGlzIGFkZGl0aW9uYWwNCiBhbGdvcml0aG1zIG5lZWRlZCB0byBiZSBp
bXBsZW1lbnRlZCBpbiBhZGRpdGlvbiBvbiB0b3AgZS5nLiBmb3IgYXV0b21hdGljIHJlY292ZXJ5
IChieSAmcXVvdDtteXNlbGYmcXVvdDspLjwvZGl2Pg0KPGRpdiBpZD0ieF9nbWFpbC1tXzUxNjM2
NjMzMjE4MDUzODI0Mjl5TWFpbF9jdXJzb3JFbGVtZW50VHJhY2tlcl8xNTUzNzg5MTY4ODkxIj48
YnI+DQo8L2Rpdj4NCjxkaXYgaWQ9InhfZ21haWwtbV81MTYzNjYzMzIxODA1MzgyNDI5eU1haWxf
Y3Vyc29yRWxlbWVudFRyYWNrZXJfMTU1Mzc4OTE2OTQ0OSI+SSBjb3VsZCBzdGFydCBhbiBvdGhl
ciBlbWFpbCBjaGFpbiwgaWYgdGhpcyBjaGFpbiBpcyBtZWFudCBtb3JlIGZvciBzb21ldGhpbmcg
ZWxzZS48L2Rpdj4NCjxkaXYgaWQ9InhfZ21haWwtbV81MTYzNjYzMzIxODA1MzgyNDI5eU1haWxf
Y3Vyc29yRWxlbWVudFRyYWNrZXJfMTU1Mzc4ODkzODIzOCI+PGJyPg0KPC9kaXY+DQo8ZGl2IGlk
PSJ4X2dtYWlsLW1fNTE2MzY2MzMyMTgwNTM4MjQyOXlNYWlsX2N1cnNvckVsZW1lbnRUcmFja2Vy
XzE1NTM3ODg5NTY0OTAiPkJlc3QgUmVnYXJkczwvZGl2Pg0KPGRpdiBpZD0ieF9nbWFpbC1tXzUx
NjM2NjMzMjE4MDUzODI0Mjl5TWFpbF9jdXJzb3JFbGVtZW50VHJhY2tlcl8xNTUzNzg4OTYyMzE3
Ij5TYW08YnI+DQo8ZGl2IGlkPSJ4X2dtYWlsLW1fNTE2MzY2MzMyMTgwNTM4MjQyOXltYWlsX2Fu
ZHJvaWRfc2lnbmF0dXJlIj48YnI+DQo8L2Rpdj4NCjxicj4NCjxibG9ja3F1b3RlIHN0eWxlPSJt
YXJnaW46MHB4IDBweCAyMHB4Ij4NCjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OlJvYm90byxzYW5z
LXNlcmlmOyBjb2xvcjpyZ2IoMTA5LDAsMjQ2KSI+DQo8ZGl2Pk9uIHRvLCBtYWFsaXNrLiAyOCwg
MjAxOSBhdCAxMjoxMCwgTWFyaWVsIENoZXJrYXNza3k8L2Rpdj4NCjxkaXY+Jmx0OzxhIGhyZWY9
Im1haWx0bzptYXJpZWwuY2hlcmthc3NreUBnbWFpbC5jb20iIHRhcmdldD0iX2JsYW5rIj5tYXJp
ZWwuY2hlcmthc3NreUBnbWFpbC5jb208L2E+Jmd0OyB3cm90ZTo8L2Rpdj4NCjwvZGl2Pg0KPGRp
diBzdHlsZT0icGFkZGluZzoxMHB4IDBweCAwcHggMjBweDsgbWFyZ2luOjEwcHggMHB4IDBweDsg
Ym9yZGVyLWxlZnQ6MXB4IHNvbGlkIHJnYigxMDksMCwyNDYpIj4NCjxkaXYgaWQ9InhfZ21haWwt
bV81MTYzNjYzMzIxODA1MzgyNDI5eWl2NTQ5NDA5ODg1MCI+DQo8ZGl2IGRpcj0icnRsIj4NCjxk
aXYgZGlyPSJsdHIiPkhleSw8L2Rpdj4NCjxkaXYgZGlyPSJsdHIiPkkgd2FzIHNlYXJjaGluZyBm
b3IgYSBzb2x1dGlvbiB0byBzY2FsZSBteSBwb3N0Z3Jlc3FsIGluc3RhbmNlIGluIHRoZSBjbG91
ZC4gSSdtIGF3YXJlIG9mIHRoYXQgdGhhdCBJIGNhbiBjcmVhdGUgbWFueSByZWFkIG9ubHkgcmVw
bGljYXMgaW4gdGhlIGNsb3VkIGFuZCBpdCB3b3VsZCBpbXByb3ZlIG15IHJlYWRpbmcgcGVyZm9y
bWFuY2UuIEkgd2FudGVkIHRvIGhlYXIgd2hhdCBzb2x1dGlvbiBhcmUgeW91IGZhbWlsaWFyDQog
d2l0aCA/IEFyZSB0aGVyZSBhbnkgc2hhcmRpbmcgc29sdXRpb24gdGhhdCBhcmUgY29tbW9ubHkg
dXNlZCAoY2l0dXMgPyBwZ19zaGFyZCA/KSBNeSBpbnN0YW5jZSBoYXMgbWFueSBkYnMgKG9uZSBw
ZXIgY3VzdG9tZXIpIGFuZCBiaWcgY3VzdG9tZXJzIGNhbiBnZW5lcmF0ZSBhIGxvYWQgb2YgbG9h
ZCBvbiBvdGhlcnMuLjwvZGl2Pg0KPGRpdiBkaXI9Imx0ciI+PGJyPg0KPC9kaXY+DQo8ZGl2IGRp
cj0ibHRyIj48YnI+DQo8L2Rpdj4NCjxkaXYgZGlyPSJsdHIiPjxicj4NCjwvZGl2Pg0KPGRpdiBk
aXI9Imx0ciI+VGhhbmtzLjwvZGl2Pg0KPC9kaXY+DQo8L2Rpdj4NCjwvZGl2Pg0KPC9ibG9ja3F1
b3RlPg0KPC9kaXY+DQo8L2Jsb2NrcXVvdGU+DQo8L2Rpdj4NCjwvZGl2Pg0KPC9ib2R5Pg0KPC9o
dG1sPg0K

--_000_MN2PR21MB120014E7EDB452FE85965719EA590MN2PR21MB1200namp_--