Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nDDNN-0001qn-Ff for pgsql-hackers@arkaria.postgresql.org; Thu, 27 Jan 2022 22:36:21 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1nDDNL-0008Rx-GW for pgsql-hackers@arkaria.postgresql.org; Thu, 27 Jan 2022 22:36:19 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nDDNK-0008Rg-Vk for pgsql-hackers@lists.postgresql.org; Thu, 27 Jan 2022 22:36:19 +0000 Received: from mail-vs1-xe2b.google.com ([2607:f8b0:4864:20::e2b]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1nDDNI-0008Fp-7D for pgsql-hackers@postgresql.org; Thu, 27 Jan 2022 22:36:17 +0000 Received: by mail-vs1-xe2b.google.com with SMTP id a7so827498vsc.6 for ; Thu, 27 Jan 2022 14:36:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ttXLrY4FESmLOuEQKjEC7kmPSluilqh3hLTynX1QEc8=; b=YiNVJYIvvRD27HAD6R3gHYR8Vt0e+7QjcLpZvqpl9tcq6D2rLQ44x7K+6vfwudr22z bqUFAdol+i7QJxNafP04KrbTEWV+b0tAeN7jxIj17aMI3aj9+YxYsM2Enzq7dxB3/2MS E1Xq7tN9H0X0hfR4c+GwmgrC9sl7Bg8T7r5NIi4sj69Q0Jq4O/wTVwibaEvTz85MsYnw gwBVptsygFEZeP6f0np1IvliNVrJ3a15h+pLtlH9IKFOgkoCl9eyXIzh8TPuI8DTVZJ+ 2D1P46HqA0uGJqy0oqiy+uPKkXak8SHtz+JRXTetO6/WD985hjprQsNi1p6CNl7E92Vu 4D7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ttXLrY4FESmLOuEQKjEC7kmPSluilqh3hLTynX1QEc8=; b=4H6BYaQ/bWW8GdnbJmc/oNpazOnMKnBhwNZ5qPNMQYvfcv+2/MmF6fYuuORZLKOBKg pmyQs2lKLnPiGVKH1phL8A0FDVfZ1r6yI4+NCKgjJrx06/OFN581YaHxwTXkTmC9AXWw sjyV6dzCTcdPz1kTo1fPvmpySQ2S/hLGASp9Zbwy0NvTxN/+GqYMoJhy+o+REoEG9N+G Ec94i0neQQUTHD7SdVsxmMYDyQ4OwfvTa5aHgGVHRsbpCBimVBuxNJN/BNuUgM2Dvlht Wp0W83RpeNuMpXC1V3lioAb+gvaCFsRlXvLHAszYqOrBs3XmU82r3lAjoAu3EaK/tbaU D0ww== X-Gm-Message-State: AOAM531BFRKUE3ET1ILcccbHeMyfbp/rXw5nC24K6nxoMjLHBLATYOCa GS3EXe+kRBYnR+P1yL1mXBJ3iYaFxNBPOlVTFz0= X-Google-Smtp-Source: ABdhPJwUzy4VgBLJuJSwbLZgKF2YTxOcDRYg1nmB3Vjuli1oFpo1RnQtyxvRIy2mbgNx9ZTH+5oPeEVdZFjQGF2qxV4= X-Received: by 2002:a67:cc06:: with SMTP id q6mr2728301vsl.21.1643322975299; Thu, 27 Jan 2022 14:36:15 -0800 (PST) MIME-Version: 1.0 References: <20220125063131.4cmvsxbz2tdg6g65@alap3.anarazel.de> <20220127054606.7schba5oizo7jijf@alap3.anarazel.de> <20220127211518.qil7cmfvtxgfhmh3@alap3.anarazel.de> In-Reply-To: <20220127211518.qil7cmfvtxgfhmh3@alap3.anarazel.de> From: "David G. Johnston" Date: Thu, 27 Jan 2022 15:35:57 -0700 Message-ID: Subject: Re: Design of pg_stat_subscription_workers vs pgstats To: Andres Freund Cc: Amit Kapila , Masahiko Sawada , pgsql-hackers Content-Type: multipart/alternative; boundary="00000000000031acb005d697f2d3" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --00000000000031acb005d697f2d3 Content-Type: text/plain; charset="UTF-8" On Thu, Jan 27, 2022 at 2:15 PM Andres Freund wrote: > Another related thing is that using a 32bit xid for allowing skipping is a > bad > idea anyway - we shouldn't adding new interfaces with xid wraparound > dangers - > it's getting more and more common to have multiple wraparounds a day. An > easily better alternative would be the LSN at which a transaction starts. > > Interesting idea. I do not think a well-designed skipping feature need worry about wrap-around though. The XID to be skipped was just seen be a worker and because it failed it will continue to be the same XID encountered by that worker until it is resolved. There is no effective progression in time while the subscriber is stuck for wrap-around to happen. Since we want to skip the transaction as a whole adding a layer of hidden indirection to the process seems undesirable. I'm not against the idea though - to the user it is basically "copy this value from the error message in order to skip the transaction that caused the error". Then the system verifies the value and then ensures it skips one, and only one, transaction. > It's pretty easy from the POV of getting into a new transaction. > > PG_CATCH(): > > /* get us out of the failed transaction */ > AbortOutOfAnyTransaction(); > > StartTransactionCommand(); > /* do something to remember the error we just got */ > CommitTransactionCommand(); > Thank you. > It may be a bit harder to afterwards to to not just error out the whole > worker, because we'd need to know what to do instead. > > I imagine the launcher and worker startup code can be made to deal with the restart adequately. Just wait if the last thing seen was an error. Require the user to manually resume the worker - unless we really think a try-until-you-succeed with a backoff protocol is superior. Upon system restart all error information is cleared and we start from scratch and let the errors happen (or not depending) as they will. David J. --00000000000031acb005d697f2d3 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Thu, Jan 27, 2022 at 2:15 PM Andres Freund <andres@anarazel.de> wrote:
=
Another related thing is that using a 32bit xid for allowing = skipping is a bad
idea anyway - we shouldn't adding new interfaces with xid wraparound da= ngers -
it's getting more and more common to have multiple wraparounds a day.= =C2=A0 An
easily better alternative would be the LSN at which a transaction starts.

Interesting idea.=C2=A0 I do not th= ink a well-designed skipping feature need worry about wrap-around though.= =C2=A0 The XID to be skipped was just seen be a worker and because it faile= d it will continue to be the same XID encountered by that worker until it i= s resolved.=C2=A0 There is no effective progression in time while the subsc= riber is stuck for wrap-around to happen.=C2=A0 Since we want to skip the t= ransaction as a whole adding a layer of hidden indirection to the process s= eems undesirable.=C2=A0 I'm not against the idea though - to the user i= t is basically "copy this value from the error message in order to ski= p the transaction that caused the error".=C2=A0 Then the system verifi= es the value and then ensures it skips one, and only one, transaction.


It's pretty easy from the POV of getting into a new transaction.

PG_CATCH():

=C2=A0 =C2=A0 /* get us out of the failed transaction */
=C2=A0 =C2=A0 AbortOutOfAnyTransaction();

=C2=A0 =C2=A0 StartTransactionCommand();
=C2=A0 =C2=A0 /* do something to remember the error we just got */
=C2=A0 =C2=A0 CommitTransactionCommand();

Thank you.
It may be a bit harder to afterwards to to not just error out the whole
worker, because we'd need to know what to do instead.


I imagine the launcher and worker startup code can be = made to deal with the restart adequately.=C2=A0 Just wait if the last thing= seen was an error.=C2=A0 Require the user to manually resume the worker - = unless we really think a=C2=A0try-until-you-succeed with a backoff protocol= is superior.=C2=A0 Upon system restart all error information is cleared an= d we start from scratch and let the errors happen (or not depending) as the= y will.

David J.

--00000000000031acb005d697f2d3--