From: <bill.poole@ymail.com>
To: "'Laurenz Albe'" <laurenz.albe@cybertec.at>,
	<pgsql-performance@postgresql.org>
References: <008701db93cd$2eb404d0$8c1c0e70$.ref@ymail.com>
	 <008701db93cd$2eb404d0$8c1c0e70$@ymail.com>
 <0942a47d7a89f1b181f7e306f8389d9717eb5fe0.camel@cybertec.at>
In-Reply-To: <0942a47d7a89f1b181f7e306f8389d9717eb5fe0.camel@cybertec.at>
Subject: RE: Bulk DML performance
Date: Thu, 13 Mar 2025 18:13:08 +0800
Message-ID: <00a401db9400$86c240f0$9446c2d0$@ymail.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Thread-Index: AQCh7jmKC8Qb7CsNFahmY0MRK48+bwLkLSPcAykm/9G1s7YUwA==
Content-Language: en-au
Content-Length: 5319
Archived-At: 
 <https://www.postgresql.org/message-id/00a401db9400%2486c240f0%249446c2d0%24%40ymail.com>
Precedence: bulk

Thanks Laurenz for taking the time to look at this.

> That is unavoidable, because PostgreSQL adds a new version for each =
row to the table.  To avoid that kind of bloat, you'd have to update in =
smaller batches and run VACUUM between those to free the "dead" row =
versions.

Yes, I would expect the number of blocks/MBs to double due to executing =
an update, but it is noteworthy that inserting 27 MB of data into a =
newly created table creates 191 MB of data including the index and 127 =
MB of data excluding the index.

> It is the index maintenance that is killing you.

Yes, but as you say I need the index to perform the INSERT ... ON =
CONFLICT (...) DO UPDATE. Can you help me understand why performing 3 =
million lookups on a b-tree index with all pages cached in memory takes =
so long?

> If you don't need crash safety, you could use UNLOGGED tables and be =
somewhat faster (5.8 seconds for the initial INSERT here).

Sadly, that is not an option for me. I'm building a production system.

> Well, that is not a great statement.

Understood, but I was highlighting the performance of deleting 3 million =
rows identified by 3 million IDs, as opposed to deleting rows in a given =
range of IDs or deleting the whole table. It seems like deleting 3 =
million rows identified by 3 million IDs should be faster than updating =
3 million rows (also identified by 3 million IDs).

> With the table as it is you won't get better performance if you want =
the features that a relational database provides.

Sorry to hear that. I had hoped there was room to improve this =
performance.

> To get better performance, the best I can think of is to parallelize =
loading the data until you saturate CPU, disk or hit internal contention =
in the database.

Sadly, I cannot do that because I need all rows to be inserted in a =
single database transaction, which I cannot do over multiple database =
connections.

Regards,
Bill

-----Original Message-----
From: Laurenz Albe <laurenz.albe@cybertec.at>=20
Sent: Thursday, 13 March 2025 5:21 PM
To: bill.poole@ymail.com; pgsql-performance@postgresql.org
Subject: Re: Bulk DML performance

On Thu, 2025-03-13 at 12:05 +0800, bill.poole@ymail.com wrote:

> The following INSERT .. ON CONFLICT (=E2=80=A6) DO UPDATE statement=20
> inserts/updates
> 3 million rows with only 9 bytes per row and takes about 8 seconds on=20
> first run (to insert the rows) and about 14 seconds on subsequent runs =

> (to update the rows),

I can confirm these times on my not so new laptop with local NVME.
That's the time it takes if you have an index on the table and want to =
be crash safe.

>           but is only inserting 27 MB of data (3 million rows with 9=20
> bytes per row); although after the first run, SELECT=20
> pg_size_pretty(pg_total_relation_size('test')) reports the table size=20
> as 191 MB and after the second run reports the table size as 382 MB=20
> (adding another 191 MB).

That is unavoidable, because PostgreSQL adds a new version for each row =
to the table.  To avoid that kind of bloat, you'd have to update in =
smaller batches and run VACUUM between those to free the "dead" row =
versions.

> CREATE TABLE test (
>   id bigint PRIMARY KEY,
>   text1 text
> );
> =20
> INSERT INTO test (id, text1)
> SELECT generate_series, 'x'
> FROM generate_series(1, 3000000)
> ON CONFLICT (id) DO UPDATE
> SET text1 =3D 'x';
> =20
> If PostgreSQL is writing 191 MB on the first run and 382 MB on each=20
> subsequent run, then PostgreSQL is only writing about 28 MB/s.=20
> Although PostgreSQL is also able to write about 4.5 GB in about 35=20
> seconds (as stated above), which is about 128 MB/s, so it seems the=20
> performance constraint depends on the number of rows inserted more =
than the size of each row.

It is the index maintenance that is killing you.
Without the primary key, the first insert takes under 1.8 seconds here.
But you need the primary key index if you want to use INSERT ... ON =
CONFLICT.

The update has to do even more work, so it is slower.

If you don't need crash safety, you could use UNLOGGED tables and be =
somewhat faster (5.8 seconds for the initial INSERT here).

Essentially, the moderate performance is the price you are paying for =
data integrity (crash safety) and consistency (primary key).

> Furthermore, deleting the rows takes about 18 seconds to perform=20
> (about 4 seconds longer than the time taken to update the rows):
> =20
> DELETE FROM test
> WHERE id in (
>   SELECT * FROM generate_series(1, 3000000)
> )

Well, that is not a great statement.

The following takes only 1.6 seconds here:

DELETE FROM test WHERE id BETWEEN 1 AND 3000000;

And if you want to delete all rows, TRUNCATE is very, very fast.

> It seems like it should be possible to do better than this on modern=20
> hardware, but I don=E2=80=99t have enough knowledge of the inner =
workings of=20
> PostgreSQL to know whether my instinct is correct on this, so I=20
> thought I=E2=80=99d raise the question with the experts.

With the table as it is you won't get better performance if you want the =
features that a relational database provides.

To get better performance, the best I can think of is to parallelize =
loading the data until you saturate CPU, disk or hit internal contention =
in the database.

Yours,
Laurenz Albe