MIME-Version: 1.0
References: <RFS5d0T3zgpLa3Rd39K96mcUndlPZD7ktJmzEXkPTrcIwO3U0MDFC0StbozM0qW9tl7X1YDVVYGFQhCOMFi9r4kp3OohQCFIh198LV1HZvQ=@a7q.com>
In-Reply-To: <RFS5d0T3zgpLa3Rd39K96mcUndlPZD7ktJmzEXkPTrcIwO3U0MDFC0StbozM0qW9tl7X1YDVVYGFQhCOMFi9r4kp3OohQCFIh198LV1HZvQ=@a7q.com>
From: Francisco Olarte <folarte@peoplecall.com>
Date: Mon, 15 Apr 2024 10:24:55 +0200
Message-ID: <CA+bJJby8zoTFVMYd405We8WF78yCc2NP4uaQf8ekuL8xtmBSYw@mail.gmail.com>
Subject: Re: constant crashing
To: jack <jack4pg@a7q.com>
Cc: "pgsql-general@lists.postgresql.org" <pgsql-general@lists.postgresql.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: <https://www.postgresql.org/message-id/CA%2BbJJby8zoTFVMYd405We8WF78yCc2NP4uaQf8ekuL8xtmBSYw%40mail.gmail.com>
Precedence: bulk

Jack:

On Sun, 14 Apr 2024 at 22:18, jack <jack4pg@a7q.com> wrote:
> The CSV files are being produced by another system, a WIndows app on a Wi=
ndows machine. I then copy them to a USB key and copy them onto the ubuntu =
machine. The data is then imported via the COPY command.
>
> COPY master (field01,field02..fieldX) FROM '/data/file.text' DELIMITER E'=
\t'
> The fields are tab delimited.

Then call them tab-delimited, not CSV. It may be important. You have a
tendency of sending your interpretation of information which hampers
people trying to help you.

As an example, only about three messages above did we see the "signal
11" in the server logs which indicates a probable bug. Signal 11 is
normally due to stray pointers or corrupted memory ( although faulty
ram can lead to it too ), which hints at a bug, not a resource
problem.

> But importing the data works. I can get all the data into a single table =
without any problems. The issue is only when I start to update the single t=
able. And that is why I started using smaller temporary tables for each CSV=
 file, to do the updates in the smaller tables before I move them all to a =
single large table.
> After all the data is loaded and updated, I run php programs on the large=
 table to generate reports. All of which works well EXCEPT for performing t=
he updates on the data. And I do not want to use perl or any outside tool. =
I want it all one in SQL because I am required to document all my steps so =
that someone else can take over, so everything needs to be as simple as pos=
sible.

I suggested perl ( or any similar thing, php should be equally fine )
as it normally makes your importing / debugging much easier / lighter.

On the documentation side, you can have the data and some
transformations in a big sql file, but having some steps in a php file
will have them documented too. But if you want to do text processing
in SQL, go ahead, the only problems are going to be making it harder
to debug and probably harder to document. Now people know the signal
11 stuff and the problem can probably be tracked.

Francisco Olarte.