From: Mike Sofen <msofen@runbox.com>
To: 'postgres performance list' <pgsql-performance@postgresql.org>
Subject: Re: Postgres bulk insert/ETL performance on high speed servers - test results
Date: Sun, 4 Sep 2016 05:34:01 -0700
Message-ID: <001401d206a8$9eda1350$dc8e39f0$@runbox.com> (raw)
In-Reply-To: <CAGTBQpa+go_gT2=H7JJROTY-qpo8x12JLNP-L46kaRMuk14zZg@mail.gmail.com>
References: <MWHPR10MB130991B6EE45A43B50E081EFDDE20@MWHPR10MB1309.namprd10.prod.outlook.com>
<04b701d204c2$0bd7e330$2387a990$@runbox.com>
<CAGTBQpa+go_gT2=H7JJROTY-qpo8x12JLNP-L46kaRMuk14zZg@mail.gmail.com>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-performance>
From: Claudio Freire Sent: Friday, September 02, 2016 1:27 PM
On Thu, Sep 1, 2016 at 11:30 PM, Mike Sofen < <mailto:msofen@runbox.com> msofen@runbox.com> wrote:
> It's obvious the size of the batch exceeded the AWS server memory,
> resulting in a profoundly slower processing time. This was a true,
> apples to apples comparison between Pass 1 and Pass 2: average row
> lengths were within 7% of each other (1121 vs 1203) using identical
> table structures and processing code, the only difference was the target server.
>
> I'm happy to answer questions about these results.
Are you sure it's a memory thing and not an EBS bandwidth thing?
EBS has significantly less bandwidth than direct-attached flash.
You raise a good point. However, other disk activities involving large data (like backup/restore and pure large table copying), on both platforms, do not seem to support that notion. I did have both our IT department and Cisco turn on instrumentation for my last test, capturing all aspects of both tests on both platforms, and I’m hoping to see the results early next week and will reply again.
Mike
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: msofen@runbox.com
Subject: Re: Postgres bulk insert/ETL performance on high speed servers - test results
In-Reply-To: <001401d206a8$9eda1350$dc8e39f0$@runbox.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox