Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rsTvk-007Lb7-5H for pgsql-general@arkaria.postgresql.org; Thu, 04 Apr 2024 20:43:28 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1rsTvj-00D1WI-D9 for pgsql-general@arkaria.postgresql.org; Thu, 04 Apr 2024 20:43:27 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rsTul-00CwuH-D7 for pgsql-general@lists.postgresql.org; Thu, 04 Apr 2024 20:42:27 +0000 Received: from mail-pg1-x52b.google.com ([2607:f8b0:4864:20::52b]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1rsTui-000bUj-5X for pgsql-general@lists.postgresql.org; Thu, 04 Apr 2024 20:42:26 +0000 Received: by mail-pg1-x52b.google.com with SMTP id 41be03b00d2f7-5dbd519bde6so1222773a12.1 for ; Thu, 04 Apr 2024 13:42:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712263343; x=1712868143; darn=lists.postgresql.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=oX0Cbr6EkeU3TjWmb0iVeGWF5Vq2gCKzfGEg3tK9Akw=; b=CESXHqB2AxqIRu9TUPhB78ddUGRt5dDyxshv6B+hhDC+6YqfoRKoh9VINoOFesxGt5 fzucPOBu6Ydb+ViFshObFdDLROc2sHBh8ul265/1rKRCdiRlGHjRDVvugByauJ2khQOX 0KNmi9n0Rsv1yP32mb4go3BJVVyUPYukWbW9KGm7tsa0kDwGr/7FrtlRM8W0W5UbA666 r3+wj37Mlq6JcZHc2sxfL7WZsTHfdE8JiU3u1Gsd3sJaitw6bh0KaY2cc3rEPE1qT6PP 6r7oRhvumwd/rPyD/EsmfHseMfaFGt+mrw2kZBwADeCnMBevfqLEjfccLM2QCW646juD QzJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712263343; x=1712868143; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=oX0Cbr6EkeU3TjWmb0iVeGWF5Vq2gCKzfGEg3tK9Akw=; b=YsAFgnCRhhThptkv6OcW+st5+dwoLRAXGyDbg/nBM8db1lDszQ32VOvKVfa54ccyve ojv0cjY2ajQxQ2SiYk0OmRd/MlHlUxfoClcJtrNB/TfVtgzmCQtuEVxmtjkPXlHSLKiG yXoWW9ubqGtRGB8/dX1oPYN8JBVuox8zZAp/ziDKdECLYZr6vgDyIyP1Vj2gLKhfqLU+ MJ74XPOi9RmSRw40YgxBonBAIoGL1RDL7cvVniLoC2JnOELUOqAbVw3o+JOiVgXNZLMn KDGMtUNb8yH9EmZ7TDq+vSVAza/5WYQdF4Gz56xvJC7lSKtcpDBXZ6mCmq71cYKMwio/ 83GA== X-Gm-Message-State: AOJu0YwecoXL/H1xIVabAhY3Bp2+j5mAizDDCflzE4xoiXogqFFdtVh+ 7NCeAPqsJdUpKcAcsmLlf7anM0HTR9ZwOSwT8R97Ep4fUT5CNGsC/Knwk+g/+cXSv+V14dYyVW+ iKtt+z9NGFFcn/jsrnkU4waGaKGojAiiT X-Google-Smtp-Source: AGHT+IEoFmXspOwisuVBb0l3uM96cO/LofVswnjiXxyUmUx6KbSMpVXjuuB/4enKrb/uBr48RO+tmTtEE0aTqPFoKCA= X-Received: by 2002:a05:6a20:96d3:b0:1a3:ca1f:8baa with SMTP id hq19-20020a056a2096d300b001a3ca1f8baamr910267pzc.32.1712263343204; Thu, 04 Apr 2024 13:42:23 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: yudhi s Date: Fri, 5 Apr 2024 02:12:10 +0530 Message-ID: Subject: Re: Moving delta data faster To: Adrian Klaver Cc: pgsql-general Content-Type: multipart/alternative; boundary="000000000000556f6406154b6042" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --000000000000556f6406154b6042 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Apr 4, 2024 at 9:04=E2=80=AFPM Adrian Klaver wrote: > On 4/3/24 22:24, yudhi s wrote: > > > > On Thu, Apr 4, 2024 at 10:16=E2=80=AFAM Adrian Klaver > > wrote: > > S3 is not a database. You will need to be more specific about '... then > from the S3 it will be picked and gets merged to the target postgres > database.' > > The data from S3 will be dumped into the stage table and then the upsert/merge from that table to the actual table. --000000000000556f6406154b6042 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

On Thu, Apr 4, 2024 at 9:04=E2=80=AFPM Ad= rian Klaver <adrian.klaver@= aklaver.com> wrote:
On 4/3/24 22:24, yudhi s wrote:
>
> On Thu, Apr 4, 2024 at 10:16=E2=80=AFAM Adrian Klaver <adrian.klaver@aklaver.co= m
> <mailto:adrian.klaver@aklaver.com>> wrote:

S3 is not a database. You will need to be more specific about '... then=
from the S3 it will be picked and gets merged to the target postgres
database.'


The data from S3 wil= l be dumped into the stage table and then the upsert/merge from that table = to the actual table.
--000000000000556f6406154b6042--