Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bgWdB-0004uq-0Y for pgsql-performance@arkaria.postgresql.org; Sun, 04 Sep 2016 12:34:37 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.84_2) (envelope-from ) id 1bgWdA-0000HQ-GX for pgsql-performance@arkaria.postgresql.org; Sun, 04 Sep 2016 12:34:36 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1bgWdA-0000HK-3t for pgsql-performance@postgresql.org; Sun, 04 Sep 2016 12:34:36 +0000 Received: from aibo.runbox.com ([91.220.196.211]) by magus.postgresql.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:256) (Exim 4.84_2) (envelope-from ) id 1bgWd1-000644-A3 for pgsql-performance@postgresql.org; Sun, 04 Sep 2016 12:34:35 +0000 Received: from [10.9.9.211] (helo=mailfront11.runbox.com) by bars.runbox.com with esmtp (Exim 4.71) (envelope-from ) id 1bgWcz-0000ot-MY for pgsql-performance@postgresql.org; Sun, 04 Sep 2016 14:34:25 +0200 Received: from cpe-66-27-125-35.san.res.rr.com ([66.27.125.35] helo=seasyslap4) by mailfront11.runbox.com with esmtpsa (uid:561468 ) (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) id 1bgWcf-0007hj-2J for pgsql-performance@postgresql.org; Sun, 04 Sep 2016 14:34:05 +0200 From: "Mike Sofen" To: "'postgres performance list'" References: <04b701d204c2$0bd7e330$2387a990$@runbox.com> In-Reply-To: Subject: Re: Postgres bulk insert/ETL performance on high speed servers - test results Date: Sun, 4 Sep 2016 05:34:01 -0700 Message-ID: <001401d206a8$9eda1350$dc8e39f0$@runbox.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0015_01D2066D.F27CC1F0" X-Mailer: Microsoft Outlook 16.0 Thread-Index: AQGpF9i869D43eNvo08BeUkSb4OtBwJxyO9fAcliwO6gmUXpAA== Content-Language: en-us X-Pg-Spam-Score: -4.3 (----) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-performance Precedence: bulk Sender: pgsql-performance-owner@postgresql.org This is a multipart message in MIME format. ------=_NextPart_000_0015_01D2066D.F27CC1F0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable From: Claudio Freire Sent: Friday, September 02, 2016 1:27 PM On Thu, Sep 1, 2016 at 11:30 PM, Mike Sofen < = msofen@runbox.com> wrote: > It's obvious the size of the batch exceeded the AWS server memory,=20 > resulting in a profoundly slower processing time. This was a true,=20 > apples to apples comparison between Pass 1 and Pass 2: average row=20 > lengths were within 7% of each other (1121 vs 1203) using identical=20 > table structures and processing code, the only difference was the = target server. >=20 > I'm happy to answer questions about these results. =20 Are you sure it's a memory thing and not an EBS bandwidth thing? =20 EBS has significantly less bandwidth than direct-attached flash. =20 You raise a good point. However, other disk activities involving large = data (like backup/restore and pure large table copying), on both = platforms, do not seem to support that notion. I did have both our IT = department and Cisco turn on instrumentation for my last test, capturing = all aspects of both tests on both platforms, and I=E2=80=99m hoping to = see the results early next week and will reply again. =20 Mike ------=_NextPart_000_0015_01D2066D.F27CC1F0 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

From: Claudio = Freire=C2=A0=C2=A0 Sent: Friday, September 02, 2016 1:27 PM
On Thu, = Sep 1, 2016 at 11:30 PM, Mike Sofen <msofen@runbox.com<= /a>> wrote:

> It's obvious the size of the batch = exceeded the AWS server memory,

> resulting in a profoundly slower = processing time.=C2=A0 This was a true,

> apples to apples = comparison between Pass 1 and Pass 2: average row

> lengths were within = 7% of each other (1121 vs 1203) using identical

> table structures = and processing code, the only difference was the target = server.

> 

> I'm happy to answer = questions about these results.

 

Are you sure it's a memory thing and not an = EBS bandwidth thing?

 

 

You raise a good point.=C2=A0 However, other disk = activities involving large data (like backup/restore and pure large = table copying), on both platforms, do not seem to support that = notion.=C2=A0 I did have both our IT department and Cisco turn on = instrumentation for my last test, capturing all aspects of both tests on = both platforms, and I=E2=80=99m hoping to see the results early next = week and will reply again.

 

Mike

------=_NextPart_000_0015_01D2066D.F27CC1F0--