public inbox for [email protected]
help / color / mirror / Atom feedLoading data from one table to another
6+ messages / 6 participants
[nested] [flat]
* Loading data from one table to another
@ 2024-10-13 06:24 Sabyasachi Mukherjee <[email protected]>
2024-10-13 09:36 ` RE: Loading data from one table to another Jean-Paul POZZI <[email protected]>
2024-10-13 14:34 ` Re: Loading data from one table to another Scott Ribe <[email protected]>
2024-10-14 06:36 ` Re: Loading data from one table to another Laurenz Albe <[email protected]>
0 siblings, 3 replies; 6+ messages in thread
From: Sabyasachi Mukherjee @ 2024-10-13 06:24 UTC (permalink / raw)
To: [email protected] <[email protected]>
Dear All,
We have a Postgres DB with 99 GB of data. The query performances are slow as expected. I am trying to introduce monthly partitions for this table.
To copy the data - the command that I am trying to execute is (Insert into ...(Select * from )) using PGAdmin
The DB is hosted in Windows server having 8 GB of RAM.
Note - we do not have any surrogate key in this table. And the event time recorded is also duplicated.
The command fails every time.
Can you please advise some strategy to get this done?
Once the pgadmin lost connection.
Another error that I have got is,
ERROR: Failed on request of size 1048576 in memory context "AfterTriggerEvents".out of memory
ERROR: out of memory
SQL state: 53200
Detail: Failed on request of size 1048576 in memory context "AfterTriggerEvents".
Thanks & regards
Sabyasachi Mukherjee
^ permalink raw reply [nested|flat] 6+ messages in thread
* RE: Loading data from one table to another
2024-10-13 06:24 Loading data from one table to another Sabyasachi Mukherjee <[email protected]>
@ 2024-10-13 09:36 ` Jean-Paul POZZI <[email protected]>
2024-10-13 10:31 ` RE: Loading data from one table to another Holger Jakobs <[email protected]>
2 siblings, 1 reply; 6+ messages in thread
From: Jean-Paul POZZI @ 2024-10-13 09:36 UTC (permalink / raw)
To: Sabyasachi Mukherjee <[email protected]>; +Cc: pgsql-admin <[email protected]>
Hello,
It seems that the command is too lonbg to execute from pgadmin, try to execute the same Sql from a terminal ... or split your work with "from date1 to date2", you can split the work in 2,3 or more tasks and it will succed.
Regards
JP P
De: "Sabyasachi Mukherjee" <[email protected]>
À: "undefined" <[email protected]>
Envoyé: dimanche 13 octobre 2024 08:26
Objet: Loading data from one table to another
Dear All,
We have a Postgres DB with 99 GB of data. The query performances are slow as expected. I am trying to introduce monthly partitions for this table.
To copy the data - the command that I am trying to execute is (Insert into ...(Select * from )) using PGAdmin
The DB is hosted in Windows server having 8 GB of RAM.
Note - we do not have any surrogate key in this table. And the event time recorded is also duplicated.
The command fails every time.
Can you please advise some strategy to get this done?
Once the pgadmin lost connection.
Another error that I have got is,
ERROR: Failed on request of size 1048576 in memory context "AfterTriggerEvents".out of memory
ERROR: out of memory
SQL state: 53200
Detail: Failed on request of size 1048576 in memory context "AfterTriggerEvents".
Thanks & regards
Sabyasachi Mukherjee
^ permalink raw reply [nested|flat] 6+ messages in thread
* RE: Loading data from one table to another
2024-10-13 06:24 Loading data from one table to another Sabyasachi Mukherjee <[email protected]>
2024-10-13 09:36 ` RE: Loading data from one table to another Jean-Paul POZZI <[email protected]>
@ 2024-10-13 10:31 ` Holger Jakobs <[email protected]>
0 siblings, 0 replies; 6+ messages in thread
From: Holger Jakobs @ 2024-10-13 10:31 UTC (permalink / raw)
To: [email protected]
Don't forget to look up the SQL state. There are some entries regarding it on stackoverflow
--
Holger Jakobs, Bergisch Gladbach
Tel. +49 178 9759012
Am 13. Oktober 2024 11:36:48 MESZ schrieb Jean-Paul POZZI <[email protected]>:
>Hello,
>
>
>
>It seems that the command is too lonbg to execute from pgadmin, try to execute the same Sql from a terminal ... or split your work with "from date1 to date2", you can split the work in 2,3 or more tasks and it will succed.
>
>
>
>Regards
>
>
>
>JP P
>
>
>
>
>
>
>
>De: "Sabyasachi Mukherjee" <[email protected]>
>À: "undefined" <[email protected]>
>Envoyé: dimanche 13 octobre 2024 08:26
>Objet: Loading data from one table to another
>
>
>Dear All,
>
>
>We have a Postgres DB with 99 GB of data. The query performances are slow as expected. I am trying to introduce monthly partitions for this table.
>To copy the data - the command that I am trying to execute is (Insert into ...(Select * from )) using PGAdmin
>The DB is hosted in Windows server having 8 GB of RAM.
>
>
>Note - we do not have any surrogate key in this table. And the event time recorded is also duplicated.
>
>
>The command fails every time.
>Can you please advise some strategy to get this done?
>
>
>Once the pgadmin lost connection.
>Another error that I have got is,
>
>
>ERROR: Failed on request of size 1048576 in memory context "AfterTriggerEvents".out of memory
>ERROR: out of memory
>SQL state: 53200
>Detail: Failed on request of size 1048576 in memory context "AfterTriggerEvents".
>
>
>Thanks & regards
>
>
>Sabyasachi Mukherjee
>
>
>
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Loading data from one table to another
2024-10-13 06:24 Loading data from one table to another Sabyasachi Mukherjee <[email protected]>
@ 2024-10-13 14:34 ` Scott Ribe <[email protected]>
2 siblings, 0 replies; 6+ messages in thread
From: Scott Ribe @ 2024-10-13 14:34 UTC (permalink / raw)
To: Sabyasachi Mukherjee <[email protected]>; +Cc: [email protected] <[email protected]>
Failing to allocate a memory segment of only 1MB seems suspicious. What are your hardware specs?
You might want to look at your memory & parallel query settings in postgresql.conf--perhaps some of those are set too high relative to your server's memory.
I wouldn't really expect queries to be slow on 99GB, by the way. That is not a large database. Partitioning is reasonable if you want to prune old data periodically, as then you can just drop old partitions. But unless your queries are regularly have to perform table scans, I wouldn't expect them to be slow. And if they are performing table scans, you'd need to investigate how to fix that--what indexes to add. Partitioning will get you incremental performance gains in such a case, while the right index will get you exponential gains.
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Loading data from one table to another
2024-10-13 06:24 Loading data from one table to another Sabyasachi Mukherjee <[email protected]>
@ 2024-10-14 06:36 ` Laurenz Albe <[email protected]>
2024-10-15 16:01 ` Re: Loading data from one table to another vignesh kumar <[email protected]>
2 siblings, 1 reply; 6+ messages in thread
From: Laurenz Albe @ 2024-10-14 06:36 UTC (permalink / raw)
To: Sabyasachi Mukherjee <[email protected]>; [email protected] <[email protected]>
On Sun, 2024-10-13 at 06:24 +0000, Sabyasachi Mukherjee wrote:
> To copy the data - the command that I am trying to execute is (Insert into ...(Select * from )) using PGAdmin
>
> The command fails every time.
>
> ERROR: Failed on request of size 1048576 in memory context "AfterTriggerEvents".out of memory
> ERROR: out of memory
> SQL state: 53200
> Detail: Failed on request of size 1048576 in memory context "AfterTriggerEvents".
You have a row level AFTER INSERT trigger on the destination table.
Try without the trigger.
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Loading data from one table to another
2024-10-13 06:24 Loading data from one table to another Sabyasachi Mukherjee <[email protected]>
2024-10-14 06:36 ` Re: Loading data from one table to another Laurenz Albe <[email protected]>
@ 2024-10-15 16:01 ` vignesh kumar <[email protected]>
0 siblings, 0 replies; 6+ messages in thread
From: vignesh kumar @ 2024-10-15 16:01 UTC (permalink / raw)
To: Laurenz Albe <[email protected]>; Sabyasachi Mukherjee <[email protected]>; [email protected] <[email protected]>
Check if you have foreign table references and they should hold on delete cascade that should resolve data being dupliated
Sent from Outlook for Android<https://aka.ms/AAb9ysg;
________________________________
From: Laurenz Albe <[email protected]>
Sent: Monday, October 14, 2024 12:06:08 PM
To: Sabyasachi Mukherjee <[email protected]>; [email protected] <[email protected]>
Subject: Re: Loading data from one table to another
On Sun, 2024-10-13 at 06:24 +0000, Sabyasachi Mukherjee wrote:
> To copy the data - the command that I am trying to execute is (Insert into ...(Select * from )) using PGAdmin
>
> The command fails every time.
>
> ERROR: Failed on request of size 1048576 in memory context "AfterTriggerEvents".out of memory
> ERROR: out of memory
> SQL state: 53200
> Detail: Failed on request of size 1048576 in memory context "AfterTriggerEvents".
You have a row level AFTER INSERT trigger on the destination table.
Try without the trigger.
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2024-10-15 16:01 UTC | newest]
Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-10-13 06:24 Loading data from one table to another Sabyasachi Mukherjee <[email protected]>
2024-10-13 09:36 ` Jean-Paul POZZI <[email protected]>
2024-10-13 10:31 ` Holger Jakobs <[email protected]>
2024-10-13 14:34 ` Scott Ribe <[email protected]>
2024-10-14 06:36 ` Laurenz Albe <[email protected]>
2024-10-15 16:01 ` vignesh kumar <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox