public inbox for [email protected]  

help / color / mirror / Atom feed

From: Adrian Klaver <[email protected]>
To: yudhi s <[email protected]>
Cc: pgsql-general <[email protected]>
Subject: Re: Moving delta data faster
Date: Thu, 4 Apr 2024 17:39:54 -0700
Message-ID: <[email protected]> (raw)
In-Reply-To: <CAEzWdqdXvqYJJ0Pbb+uLKHMAEbvLb86kKW_GJ9DDrh=5MU+_GA@mail.gmail.com>
References: <CAEzWdqcYGi0U5_cK1FVykx2-OZHmEUD8EZ_VE=kpoVaZKYWJeg@mail.gmail.com>
	<[email protected]>
	<CAEzWdqfGN5cHN4cwSJm-rruab4E0y_9tqzihR2jQGpMXHR7cqw@mail.gmail.com>
	<[email protected]>
	<CAEzWdqfts=HUoh7at7yD0M7DW97BqSe1o+4xqtXOyM_+ZX_XMA@mail.gmail.com>
	<[email protected]>
	<CAEzWdqdXvqYJJ0Pbb+uLKHMAEbvLb86kKW_GJ9DDrh=5MU+_GA@mail.gmail.com>

On 4/4/24 13:42, yudhi s wrote:
> 
> On Thu, Apr 4, 2024 at 9:04 PM Adrian Klaver <[email protected] 
> <mailto:[email protected]>> wrote:
> 
>     On 4/3/24 22:24, yudhi s wrote:
>      >
>      > On Thu, Apr 4, 2024 at 10:16 AM Adrian Klaver
>     <[email protected] <mailto:[email protected]>
>      > <mailto:[email protected]
>     <mailto:[email protected]>>> wrote:
> 
>     S3 is not a database. You will need to be more specific about '... then
>     from the S3 it will be picked and gets merged to the target postgres
>     database.'
> 
> 
> The data from S3 will be dumped into the stage table and then the 
> upsert/merge from that table to the actual table.

The S3 --> staging table would be helped by having the data as CSV and 
then using COPY. The staging --> final table step could be done as 
either ON CONFLICT or MERGE, you would need to test in your situation to 
verify which works better.

-- 
Adrian Klaver
[email protected]

view thread (2+ messages)  latest in thread

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected]
  Subject: Re: Moving delta data faster
  In-Reply-To: <[email protected]>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox