MIME-Version: 1.0
References: <1453510076.1900935.1753260637232.ref@mail.yahoo.com> <1453510076.1900935.1753260637232@mail.yahoo.com>
In-Reply-To: <1453510076.1900935.1753260637232@mail.yahoo.com>
From: Merlin Moncure <mmoncure@gmail.com>
Date: Wed, 23 Jul 2025 11:49:49 -0600
Message-ID: <CAHyXU0x1uW_349ONE+35KT87Ua-dX8-QaZ5Sj6eENTMwsX=okw@mail.gmail.com>
Subject: Re: Is there any limit on the number of rows to import using copy command
To: "sivapostgres@yahoo.com" <sivapostgres@yahoo.com>
Cc: Pgsql-general <pgsql-general@postgresql.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: <https://www.postgresql.org/message-id/CAHyXU0x1uW_349ONE%2B35KT87Ua-dX8-QaZ5Sj6eENTMwsX%3Dokw%40mail.gmail.com>
Precedence: bulk

On Wed, Jul 23, 2025 at 2:51=E2=80=AFAM sivapostgres@yahoo.com
<sivapostgres@yahoo.com> wrote:
>
> Tried in PostgreSQL 11.11 , PostgreSQL 15.2 in Windows 10
>
> Here we try to transfer data from one database to another (remote) databa=
se.
>
> Tables do have records ranging from 85000 to 3600000 along with smaller s=
ized tables.
> No issues while transferring smaller sized tables.
>
> I here take one particular table [table1] which has 85000 records.
> The table got Primary Key, Foreign Key(s), Triggers.  Trigger updates ano=
ther table [table2]
> Table2 have 2 triggers, one to arrive a closing value and other to delete=
, if the closing value is zero.
>
> 1.  Transfer the data from source database to a csv file.  85000 records =
transferred. No issues.
> 2.  Transfer the file to the remote location.  No issues.
> 3.  Transfer the contents of the file to the table using Copy From comman=
d. - Fails when try to transfer all the 85000 records at once.
>
> Copy from command is
>
> Copy public.table1 From 'E:\temp\file1.csv' (FORMAT CSV, DELIMITER ',', H=
EADER TRUE)
>
> The above command succeeds, when
> 1.  The trigger in Table1 is disabled with all other constraints on.
> 2.  The no. of rows is within 16000 or less, with Trigger enabled.  We ha=
ven't tried with higher no of rows.
>
> The above command goes on infinite loop, when
> 1.  We try to transfer all 85000 rows at once, with Trigger and other con=
straints in table1 enabled.  We waited for 1.5 hrs first time and 2.5 hrs s=
econd time before cancelling the operation.
>
> I read in the documentation that the fastest way to transfer data is to u=
se Copy command.  And I couldn't find any limit in transferring data using =
that command.  One could easily transfer millions of rows using this comman=
d.

Most likely, you are getting yourself into trouble with the trigger
dependencies.  Triggers are powerful, but also can be dangerous, and
this could be 'wrong tool for the job' situation.

 Here are some general tips:

* pg_trigger_depth(): can tell you if trigger A calls trigger B and
back to trigger A, etc.   you can use it with raise notify, and also
use it to guard execution on CREATE TRIGGER

* reconfiguring your logic to statement level triggers can be a good
idea. this can take some thinking, but can be much more efficient when
bulk processing since trigger execution can be deferred until the load
completes. (one trick is to use now() to check for records inserted
since it is stable though the transaction)

* reconfiguring your logic to a procedure can be a better idea; COPY
your data into some staging tables (perhaps temp, and indexed), then
write to various tables with joins, upserts, etc.

merlin