Feedback-ID: i76984098:Fastmail
Message-ID: <6f56cb2b-cda0-47b1-a412-719265fd75b9@aklaver.com>
Date: Sun, 14 Apr 2024 07:58:18 -0700
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: constant crashing
To: jack <jack4pg@a7q.com>,
 "pgsql-general@lists.postgresql.org" <pgsql-general@lists.postgresql.org>
References: <EOK1DtQPZjUx8lfmH2mKBjPejveLIfJ8M_GR-At5A9yhT-Ki5JE9GyvQ657CQZWOmZu3OsByefHlGymD3i60c-eYE7uiSYQDVZsujA3fZZA=@a7q.com>
Content-Language: en-US
From: Adrian Klaver <adrian.klaver@aklaver.com>
In-Reply-To: <EOK1DtQPZjUx8lfmH2mKBjPejveLIfJ8M_GR-At5A9yhT-Ki5JE9GyvQ657CQZWOmZu3OsByefHlGymD3i60c-eYE7uiSYQDVZsujA3fZZA=@a7q.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Archived-At: <https://www.postgresql.org/message-id/6f56cb2b-cda0-47b1-a412-719265fd75b9%40aklaver.com>
Precedence: bulk

On 4/14/24 07:24, jack wrote:
> Hello,
> I am trying to load about 1.4 billion records into a postgresql table.
> The data is in about 100 individual CSV files which I load individually 
> into the table using the COPY command.
> After the data is loaded I perform updates to the data, and this is when 
> I get errors. Some updates work, some crash.
> I am using psql and an SQL script file (update.sql) to perform the 
> updating (UPDATE table SET field=UPPER(field), etc.).
> 
> The errors are all "Server closed the connection unexpectedly"
> 


> I have been working on this for just over 1 year now, documenting every 
> step, and I am still unable to get this to work without it crashing 
> somewhere along the way.
> I am beginning to wonder if postgreSQL is bi-polar.

More information needed, to start:

1) Complete example of the code in the script.

2) Table definition(s) of those being copied into.

3) Sample of the data being copied.

4) The error message(s) generated.

5) Database locale

> 
> Any help would be greatly appreciated.
> Thank you

-- 
Adrian Klaver
adrian.klaver@aklaver.com