public inbox for [email protected]  

help / color / mirror / Atom feed

From: Adrian Klaver <[email protected]>
To: yudhi s <[email protected]>
To: pgsql-general <[email protected]>
Subject: Re: Moving delta data faster
Date: Wed, 3 Apr 2024 14:11:37 -0700
Message-ID: <[email protected]> (raw)
In-Reply-To: <CAEzWdqcYGi0U5_cK1FVykx2-OZHmEUD8EZ_VE=kpoVaZKYWJeg@mail.gmail.com>
References: <CAEzWdqcYGi0U5_cK1FVykx2-OZHmEUD8EZ_VE=kpoVaZKYWJeg@mail.gmail.com>

On 4/3/24 13:38, yudhi s wrote:
>   Hi All,
>   It's postgresql database version 15.4. We have a requirement in which 
> we will be initially moving full table data for 3-4 tables, from source 
> database to target(i.e. postgres) . Maximum number of rows will be 
> ~10million rows in those tables. Then subsequently these rows will be 
> inserted/updated based on the delta number of rows that got 
> inserted/updated in the source database. In some cases these changed 
> data can flow multiple times per day to the downstream i.e. postgres 
> database and in other cases once daily.

What is the source database?

Can it be reached with a FDW?:

https://wiki.postgresql.org/wiki/Foreign_data_wrappers

Can the delta on the source be output as CSV?

> 
>   Want to understand , if we should use upsert(insert on conflict) or 
> merge statements or anything else in such a scenario so as to persist 
> those delta records faster in the target database, while making the 
> system online to the users?
> 
> Regards
> Yudhi

-- 
Adrian Klaver
[email protected]

view thread (2+ messages)  latest in thread

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected]
  Subject: Re: Moving delta data faster
  In-Reply-To: <[email protected]>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox