Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1njlUl-0008Of-Ce for pgsql-hackers@arkaria.postgresql.org; Wed, 27 Apr 2022 17:30:31 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1njlUi-00052r-J9 for pgsql-hackers@arkaria.postgresql.org; Wed, 27 Apr 2022 17:30:28 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1njlUi-00051l-6H for pgsql-hackers@lists.postgresql.org; Wed, 27 Apr 2022 17:30:28 +0000 Received: from mail-pg1-x52c.google.com ([2607:f8b0:4864:20::52c]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1njlUf-0004sd-AF for pgsql-hackers@lists.postgresql.org; Wed, 27 Apr 2022 17:30:26 +0000 Received: by mail-pg1-x52c.google.com with SMTP id x12so1975628pgj.7 for ; Wed, 27 Apr 2022 10:30:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=j-davis-com.20210112.gappssmtp.com; s=20210112; h=message-id:subject:from:to:cc:date:in-reply-to:references :mime-version:content-transfer-encoding; bh=XoxN5t/MXht3raE3PEzmDw5AMB5ZFMx/gM0+1IfXdHg=; b=JOaEm0QHCCDTM918Uf+trG9ogV89qcJxxxv9GxesULZtghvNQDfWkvQKfri3yojaP4 YSCNRdZcD5QY73ebbVlfFc5tFnA7NQTgED9jN5VHGxDaQ+PxlhrjNIILEPn6b/3DLuUm PkaPov38vIqBOhp43OkZk8PTiPpXvMQnlLQubDYUVHHD/pKxDC1sQo0zCEFrqSpflkCx xC+Ga07kVfnG5a4H1a7ZK1ySi1if/wmsiUOUSjZ0lFXm+NTDiuULFKIsyKWiJ3m9elcS BwC79o+2fsK526YqVhkg2hPz7Y4cYSVplfcSUqmEywfJyAytcMlmaP+lqJOD0v3pcudg qvaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to :references:mime-version:content-transfer-encoding; bh=XoxN5t/MXht3raE3PEzmDw5AMB5ZFMx/gM0+1IfXdHg=; b=Cz3d6+cBSrHx56MT/8PFrz4b2XFlbdi9g2rhS/DTc2PcCbvmYnCEg8hL4SugpE6tWg Fa4/AMKRUsDiRq6217eGQUl1D2y2eu218M/Y4ofYUJKPzgMlkwU80xA6MCOEWht404PQ axYeHN7eZruW/RfNFfTdFW6IpOpt5yGEn6T2oMzQDEldHQSDJeMYesyD/vzNvvtoLezK ZKANwANuxiM4R5aiLabR1AefOdoDNEav2fciCG7qWltp1fU6pmM7CqXvRa8sshJ2zQZ3 WcZar7WE/OXrQJR92YIjfERPWkIYue5PWnjHVbWRQC3my1RPiQMYztuIWc7aKBtWG9O0 wjeA== X-Gm-Message-State: AOAM531nXe4V8CygkNCt2tYndz3hfCLqG8hvs/qxrBJqwivot2082Xi2 K5H5fcKWR/hfZbRma897CW5lgg== X-Google-Smtp-Source: ABdhPJwGckIq2yY1kVozT88ph+VG61ADgcpWokp/dC4vPgZ1QPilYV63Adz7ycs0RnFpMmKyhXZeuQ== X-Received: by 2002:a63:9141:0:b0:3ab:ea6:694d with SMTP id l62-20020a639141000000b003ab0ea6694dmr18052544pge.49.1651080624237; Wed, 27 Apr 2022 10:30:24 -0700 (PDT) Received: from jdavis.lan (c-73-231-146-4.hsd1.ca.comcast.net. [73.231.146.4]) by smtp.gmail.com with ESMTPSA id 132-20020a62168a000000b004f40e8b3133sm20680300pfw.188.2022.04.27.10.30.22 (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Wed, 27 Apr 2022 10:30:22 -0700 (PDT) Message-ID: <8855927ae2fe471f5a9d2e371ac5fb1834357181.camel@j-davis.com> Subject: Re: pgsql: Add contrib/pg_walinspect. From: Jeff Davis To: Bharath Rupireddy , Tom Lane Cc: Thomas Munro , Michael Paquier , Noah Misch , Jeff Davis , Postgres hackers Date: Wed, 27 Apr 2022 10:30:21 -0700 In-Reply-To: References: <171769.1650950714@sss.pgh.pa.us> <287795.1651019129@sss.pgh.pa.us> <295868.1651024073@sss.pgh.pa.us> <298689.1651025716@sss.pgh.pa.us> <304865.1651029340@sss.pgh.pa.us> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.28.5-0ubuntu0.18.04.2 Mime-Version: 1.0 Content-Transfer-Encoding: 7bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On Wed, 2022-04-27 at 13:47 +0530, Bharath Rupireddy wrote: > I found an easy way to reproduce this consistently (I think on any > server): > > I basically generated huge WAL record (I used a fun extension that I > wrote - https://github.com/BRupireddy/pg_synthesize_wal, but one can > use pg_logical_emit_message as well) Thank you Bharath for creating the extension and the simple test case. Thomas's patch solves the issue for me as well. Tom, the debug patch you posted[0] seems to be setting the error message if it's not already set. Thomas's patch uses the lack of a message as a signal that we've reached the end of WAL. That explains why you are still seeing the problem. Obviously, that's a sign that Thomas's patch is not the cleanest solution. But other approaches would be more invasive. I guess the question is whether that's a good enough solution for now, and hopefully we could improve the API later; or whether we need to come up with something better. When reviewing, I considered the inability to read old WAL and the inability to read flushed-in-the-middle-of-a-record WAL as similar kinds of errors that the user would need to deal with. But they are different: the former can be avoided by creating a slot; the latter can't be easily avoided, only retried. Depending on the intended use cases, forcing the user to retry might be reasonable, in which case we could consider this a test problem rather than a real problem, and we might be able to do something simpler to just stabilize the test. Regards, Jeff Davis [0] https://postgr.es/m/295868.1651024073@sss.pgh.pa.us