pg.ddx.io  pgsql-committers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Jeff Davis <pgsql@j-davis.com>
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Thomas Munro <thomas.munro@gmail.com>
Cc: Michael Paquier <michael@paquier.xyz>
Cc: Noah Misch <noah@leadboat.com>
Cc: Jeff Davis <jdavis@postgresql.org>
Cc: Postgres hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: pgsql: Add contrib/pg_walinspect.
Date: Wed, 27 Apr 2022 10:30:21 -0700
Message-ID: <8855927ae2fe471f5a9d2e371ac5fb1834357181.camel@j-davis.com> (raw)
In-Reply-To: <CALj2ACW7_d2RZzq-K8ZLv+uWqrstpW9omo2Lc-jbD2K5XoFMGQ@mail.gmail.com>
References: <E1ncj1z-000rJo-Kh@gemulon.postgresql.org>
	<Ymd/e5eeZMNAkrXo@paquier.xyz>
	<171769.1650950714@sss.pgh.pa.us>
	<YmeExX6ACB/Et9Ww@paquier.xyz>
	<CA+hUKGLtswFk9ZO3WMOqnDkGs6dK5kCdQK9gxJm0N8gip5cpiA@mail.gmail.com>
	<287795.1651019129@sss.pgh.pa.us>
	<CA+hUKG+H_VEBdtK4CVb7uRLaAKbufNOMy-djUsptcqhLxONMmA@mail.gmail.com>
	<295868.1651024073@sss.pgh.pa.us>
	<CA+hUKG+By4isKdbDkCaLkPq4V4ECWW=d+RhcLjQXTQ3KAJLobg@mail.gmail.com>
	<CA+hUKGJAULJ_NtszwEdnSgrpRiQiRjsf62DG9_1B9x0Vzoex5g@mail.gmail.com>
	<298689.1651025716@sss.pgh.pa.us>
	<304865.1651029340@sss.pgh.pa.us>
	<CALj2ACVDYTeiWF8pKv9iz1kbTajunS96pWwLtyaRq_rxwAHrKg@mail.gmail.com>
	<CALj2ACW7_d2RZzq-K8ZLv+uWqrstpW9omo2Lc-jbD2K5XoFMGQ@mail.gmail.com>

On Wed, 2022-04-27 at 13:47 +0530, Bharath Rupireddy wrote:
> I found an easy way to reproduce this consistently (I think on any
> server):
> 
> I basically generated huge WAL record (I used a fun extension that I
> wrote - https://github.com/BRupireddy/pg_synthesize_wal, but one can
> use pg_logical_emit_message as well)

Thank you Bharath for creating the extension and the simple test case.

Thomas's patch solves the issue for me as well.

Tom, the debug patch you posted[0] seems to be setting the error
message if it's not already set. Thomas's patch uses the lack of a
message as a signal that we've reached the end of WAL. That explains
why you are still seeing the problem.

Obviously, that's a sign that Thomas's patch is not the cleanest
solution. But other approaches would be more invasive. I guess the
question is whether that's a good enough solution for now, and
hopefully we could improve the API later; or whether we need to come up
with something better.

When reviewing, I considered the inability to read old WAL and the
inability to read flushed-in-the-middle-of-a-record WAL as similar
kinds of errors that the user would need to deal with. But they are
different: the former can be avoided by creating a slot; the latter
can't be easily avoided, only retried.

Depending on the intended use cases, forcing the user to retry might be
reasonable, in which case we could consider this a test problem rather
than a real problem, and we might be able to do something simpler to
just stabilize the test.

Regards,
	Jeff Davis

[0] https://postgr.es/m/295868.1651024073@sss.pgh.pa.us







view thread (22+ messages)  latest in thread

Message-ID: <8855927ae2fe471f5a9d2e371ac5fb1834357181.camel@j-davis.com>
Permalink:  ../8855927ae2fe471f5a9d2e371ac5fb1834357181.camel@j-davis.com/
Also on:    postgresql.org/message-id/8855927ae2fe471f5a9d2e371ac5fb1834357181.camel@j-davis.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-committers@postgresql.org
  Cc: pgsql@j-davis.com, bharath.rupireddyforpostgres@gmail.com, tgl@sss.pgh.pa.us, thomas.munro@gmail.com, michael@paquier.xyz, noah@leadboat.com, jdavis@postgresql.org, pgsql-hackers@lists.postgresql.org
  Subject: Re: pgsql: Add contrib/pg_walinspect.
  In-Reply-To: <8855927ae2fe471f5a9d2e371ac5fb1834357181.camel@j-davis.com>
  References: <E1ncj1z-000rJo-Kh@gemulon.postgresql.org> <Ymd/e5eeZMNAkrXo@paquier.xyz> <171769.1650950714@sss.pgh.pa.us> <YmeExX6ACB/Et9Ww@paquier.xyz> <CA+hUKGLtswFk9ZO3WMOqnDkGs6dK5kCdQK9gxJm0N8gip5cpiA@mail.gmail.com> <287795.1651019129@sss.pgh.pa.us> <CA+hUKG+H_VEBdtK4CVb7uRLaAKbufNOMy-djUsptcqhLxONMmA@mail.gmail.com> <295868.1651024073@sss.pgh.pa.us> <CA+hUKG+By4isKdbDkCaLkPq4V4ECWW=d+RhcLjQXTQ3KAJLobg@mail.gmail.com> <CA+hUKGJAULJ_NtszwEdnSgrpRiQiRjsf62DG9_1B9x0Vzoex5g@mail.gmail.com> <298689.1651025716@sss.pgh.pa.us> <304865.1651029340@sss.pgh.pa.us> <CALj2ACVDYTeiWF8pKv9iz1kbTajunS96pWwLtyaRq_rxwAHrKg@mail.gmail.com> <CALj2ACW7_d2RZzq-K8ZLv+uWqrstpW9omo2Lc-jbD2K5XoFMGQ@mail.gmail.com> <8855927ae2fe471f5a9d2e371ac5fb1834357181.camel@j-davis.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox