Message-ID: <0942a47d7a89f1b181f7e306f8389d9717eb5fe0.camel@cybertec.at>
Subject: Re: Bulk DML performance
From: Laurenz Albe <laurenz.albe@cybertec.at>
To: bill.poole@ymail.com, pgsql-performance@postgresql.org
Date: Thu, 13 Mar 2025 10:21:26 +0100
In-Reply-To: <008701db93cd$2eb404d0$8c1c0e70$@ymail.com>
References: <008701db93cd$2eb404d0$8c1c0e70$.ref@ymail.com>
	 <008701db93cd$2eb404d0$8c1c0e70$@ymail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.54.3 (3.54.3-1.fc41) 
MIME-Version: 1.0
Archived-At: 
 <https://www.postgresql.org/message-id/0942a47d7a89f1b181f7e306f8389d9717eb5fe0.camel%40cybertec.at>
Precedence: bulk

On Thu, 2025-03-13 at 12:05 +0800, bill.poole@ymail.com wrote:

> The following INSERT .. ON CONFLICT (=E2=80=A6) DO UPDATE statement inser=
ts/updates
> 3 million rows with only 9 bytes per row and takes about 8 seconds on fir=
st
> run (to insert the rows) and about 14 seconds on subsequent runs (to upda=
te
> the rows),

I can confirm these times on my not so new laptop with local NVME.
That's the time it takes if you have an index on the table and want to be
crash safe.

>           but is only inserting 27 MB of data (3 million rows with 9 byte=
s
> per row); although after the first run,
> SELECT pg_size_pretty(pg_total_relation_size('test')) reports the table s=
ize
> as 191 MB and after the second run reports the table size as 382 MB (addi=
ng
> another 191 MB).

That is unavoidable, because PostgreSQL adds a new version for each row to =
the
table.  To avoid that kind of bloat, you'd have to update in smaller batche=
s
and run VACUUM between those to free the "dead" row versions.

> CREATE TABLE test (
> =C2=A0 id bigint PRIMARY KEY,
> =C2=A0 text1 text
> );
> =C2=A0
> INSERT INTO test (id, text1)
> SELECT generate_series, 'x'
> FROM generate_series(1, 3000000)
> ON CONFLICT (id) DO UPDATE
> SET text1 =3D 'x';
> =C2=A0
> If PostgreSQL is writing 191 MB on the first run and 382 MB on each subse=
quent
> run, then PostgreSQL is only writing about 28 MB/s. Although PostgreSQL i=
s
> also able to write about 4.5 GB in about 35 seconds (as stated above), wh=
ich
> is about 128 MB/s, so it seems the performance constraint depends on the
> number of rows inserted more than the size of each row.

It is the index maintenance that is killing you.
Without the primary key, the first insert takes under 1.8 seconds here.
But you need the primary key index if you want to use INSERT ... ON CONFLIC=
T.

The update has to do even more work, so it is slower.

If you don't need crash safety, you could use UNLOGGED tables and be somewh=
at
faster (5.8 seconds for the initial INSERT here).

Essentially, the moderate performance is the price you are paying for
data integrity (crash safety) and consistency (primary key).

> Furthermore, deleting the rows takes about 18 seconds to perform (about 4
> seconds longer than the time taken to update the rows):
> =C2=A0
> DELETE FROM test
> WHERE id in (
> =C2=A0 SELECT * FROM generate_series(1, 3000000)
> )

Well, that is not a great statement.

The following takes only 1.6 seconds here:

DELETE FROM test WHERE id BETWEEN 1 AND 3000000;

And if you want to delete all rows, TRUNCATE is very, very fast.

> It seems like it should be possible to do better than this on modern
> hardware, but I don=E2=80=99t have enough knowledge of the inner workings=
 of
> PostgreSQL to know whether my instinct is correct on this, so I thought
> I=E2=80=99d raise the question with the experts.

With the table as it is you won't get better performance if you want the
features that a relational database provides.

To get better performance, the best I can think of is to parallelize
loading the data until you saturate CPU, disk or hit internal contention
in the database.

Yours,
Laurenz Albe