public inbox for [email protected]  

help / color / mirror / Atom feed

Please implement a catch-all error handler per row, for COPY
2+ messages / 2 participants
[nested] [flat]

* Please implement a catch-all error handler per row, for COPY
@ 2025-03-01 15:18 me nefcanto <[email protected]>
  2025-03-01 16:31 ` Re: Please implement a catch-all error handler per row, for COPY Adrian Klaver <[email protected]>
  0 siblings, 1 reply; 2+ messages in thread

From: me nefcanto @ 2025-03-01 15:18 UTC (permalink / raw)
  To: [email protected]

Hello

Please consider these scenarios:

- I want to create a million fake products, sometimes even 100 million
(we're on MariaDB now and we plan to migrate to Postgres). My team uses
fake data for performance tests and other use cases.
- Another scenario is translations. Even in production, we have translation
files for more than 20 languages, and for more than 2 thousand keys. That
means we need to insert 40 thousand translation records in the production.
- Another scenario is updating nested model values for a large hierarchical
table. For example, the categories table. Anytime the user changes a record
in that table we need to recalculate the nested model for the entire
categories and bulk update the results.

All of these scenarios are such that data sanitation is difficult if not
possible before doing the bulk operation (copy).

I realized that when we specify `on_error ignore` it just handles a handful
of errors. I thought this was a bug and sent an email to the pgsql-bugs
maling list. But they said it's the intended behavior.

Can you please provide a row-level catch-all handler for the copy command?

Regards
Saeed


^ permalink  raw  reply  [nested|flat] 2+ messages in thread

* Re: Please implement a catch-all error handler per row, for COPY
  2025-03-01 15:18 Please implement a catch-all error handler per row, for COPY me nefcanto <[email protected]>
@ 2025-03-01 16:31 ` Adrian Klaver <[email protected]>
  0 siblings, 0 replies; 2+ messages in thread

From: Adrian Klaver @ 2025-03-01 16:31 UTC (permalink / raw)
  To: me nefcanto <[email protected]>; [email protected]

On 3/1/25 07:18, me nefcanto wrote:
> Hello
> 
>
> All of these scenarios are such that data sanitation is difficult if not 
> possible before doing the bulk operation (copy).
> 
> I realized that when we specify `on_error ignore` it just handles a 
> handful of errors. I thought this was a bug and sent an email to the 
> pgsql-bugs maling list. But they said it's the intended behavior.
> 

Take a look at:

http://ossc-db.github.io/pg_bulkload/index.html

and

https://pgloader.io/

They may be able to meet your needs.

> 
> Regards
> Saeed

-- 
Adrian Klaver
[email protected]







^ permalink  raw  reply  [nested|flat] 2+ messages in thread

end of thread, other threads:[~2025-03-01 16:31 UTC | newest]

Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2025-03-01 15:18 Please implement a catch-all error handler per row, for COPY me nefcanto <[email protected]>
2025-03-01 16:31 ` Adrian Klaver <[email protected]>

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox