public inbox for [email protected]
help / color / mirror / Atom feedFrom: Tom Lane <[email protected]>
To: jack <[email protected]>
Cc: [email protected] <[email protected]>
Subject: Re: constant crashing
Date: Sun, 14 Apr 2024 11:05:57 -0400
Message-ID: <[email protected]> (raw)
In-Reply-To: <EOK1DtQPZjUx8lfmH2mKBjPejveLIfJ8M_GR-At5A9yhT-Ki5JE9GyvQ657CQZWOmZu3OsByefHlGymD3i60c-eYE7uiSYQDVZsujA3fZZA=@a7q.com>
References: <EOK1DtQPZjUx8lfmH2mKBjPejveLIfJ8M_GR-At5A9yhT-Ki5JE9GyvQ657CQZWOmZu3OsByefHlGymD3i60c-eYE7uiSYQDVZsujA3fZZA=@a7q.com>
jack <[email protected]> writes:
> I am trying to load about 1.4 billion records into a postgresql table.
> The data is in about 100 individual CSV files which I load individually into the table using the COPY command.
> After the data is loaded I perform updates to the data, and this is when I get errors. Some updates work, some crash.
> I am using psql and an SQL script file (update.sql) to perform the updating (UPDATE table SET field=UPPER(field), etc.).
> The errors are all "Server closed the connection unexpectedly"
What appears in the postmaster log when this happens?
It would also be instructive to run "top" while you trigger
the problem, and see if there is obvious distress like the
server process's memory consumption ballooning.
It smells like a resource-exhaustion problem. Which probably
shouldn't be happening on a machine with 128G RAM, but maybe
something is misconfigured to give up way below that. In any
case, you'll never find the cause from the client side.
> Last attempts have been with ubuntu 22.04.04 and postgreSQL 16.
16 dot which?
regards, tom lane
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected]
Subject: Re: constant crashing
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox