Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rJNBf-00Cx7K-IY for pgsql-general@arkaria.postgresql.org; Sat, 30 Dec 2023 00:26:47 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1rJNBb-00DUPn-Ge for pgsql-general@arkaria.postgresql.org; Sat, 30 Dec 2023 00:26:43 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rJNBb-00DUPd-52 for pgsql-general@lists.postgresql.org; Sat, 30 Dec 2023 00:26:43 +0000 Received: from mail-ej1-x62d.google.com ([2a00:1450:4864:20::62d]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1rJNBX-00Cpjt-Dn for pgsql-general@postgresql.org; Sat, 30 Dec 2023 00:26:41 +0000 Received: by mail-ej1-x62d.google.com with SMTP id a640c23a62f3a-a26f73732c5so419838766b.3 for ; Fri, 29 Dec 2023 16:26:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=enterprisedb.com; s=google; t=1703895997; x=1704500797; darn=postgresql.org; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=odCOeXrmFhPwvuIU55JrhpRR7PnQ388nFoaZYIPVS5A=; b=mFSPId05e5RTudD/S197Lm2+ZCtnzs4+MdUNGylipjpE5iLJvHSuJQ9CVqplVhBsP/ aQqLsSVxhPjinUKFAOWet1adZ1CaW5SmBsFX6UgT93MprUPzE3Engj7gqG7tzjXRc3Xe B0gqGLePN34DZ+GyIJAGLxj3WithPcfUPkS4Pd1sxGg2+qoDpL2dBO/8RoTiYlx9InTL 2HVMJjDgSQGsXOxEoeL1Zzu71w/UHXZMOGjyG57ALSjI7tVHuDD4qHLU9xsZuPy8V5dY 9OGVKsyBDIDulbwn6WrX0dPDJG51as5/vMdhC00RsehMnPF7ZsjfQ8iSL3CYWtWfwqA4 Fjvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703895997; x=1704500797; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=odCOeXrmFhPwvuIU55JrhpRR7PnQ388nFoaZYIPVS5A=; b=GXgJ8oc5Sc4eNznjy0jH3kZtesH2QsRt7gV2PaZ3xNcJ1epv2lBSHPGmlYLwZ0NsrJ M0jGKN3UscKCeFeeLVKHKA9JDDv9Za5WmX0U6+ZWFSeOD12/1ODFlrYDZHjH3CRBurTU 4iCYsy0r5NlTZeoELYhgE9qUdnGaVZa8T6rAnuSjOBOp1+cnbOYF6H1XW+fg3IF8nMsG d4yGxr8rA8iIkLvJYvwjUNDBnQQoyhDK/dQAEJfEnt5RKGI8CkBjEimQy70ovJjWAjVe jzEqZw2DSCt2VWmUJ7siF4H9gAiL7F7jFueSd5TK4VfsyRnAiVrlBIM6aW4ZjgfzfI7I BZJA== X-Gm-Message-State: AOJu0Ywyn0m0y/1WnD6OY6cNtNtAeKZOpgbC/ehoJ20vKiVynriqUqXV PXB/g2gtbgNUTtlInQ0fRfoT2QzJNwlE X-Google-Smtp-Source: AGHT+IEDVubjIZwXA8+Aeg6w0JnKQgE5BozaG8i4ClRrjRXvsaij3jvKGOhBh9vm303RbKXMr7Q1ng== X-Received: by 2002:a17:906:2513:b0:a23:7453:fada with SMTP id i19-20020a170906251300b00a237453fadamr5970275ejb.53.1703895997047; Fri, 29 Dec 2023 16:26:37 -0800 (PST) Received: from [10.137.0.17] (ip-86-49-229-30.bb.vodafone.cz. [86.49.229.30]) by smtp.gmail.com with ESMTPSA id le19-20020a170906ae1300b00a2798d33c31sm909961ejb.77.2023.12.29.16.26.36 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 29 Dec 2023 16:26:36 -0800 (PST) Message-ID: <51b620b8-d13b-649e-4f1d-c57f1d73eb87@enterprisedb.com> Date: Sat, 30 Dec 2023 01:26:35 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.15.1 Subject: Re: Intermittent Issue with WAL Segment Removal in Logical Replication Content-Language: en-US To: Kaushik Iska Cc: pgsql-general@postgresql.org, Sai Krishna Srirampur , =?UTF-8?Q?Philip_Dub=c3=a9?= References: <3b75e08a-2d53-fbb5-731a-3a8e5c71edd8@enterprisedb.com> From: Tomas Vondra In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On 12/29/23 22:28, Kaushik Iska wrote: > I am unfortunately not really familiar with Google Cloud SQL internals > as well. But we have seen this happen on Amazon RDS as well. > Do you have a reproducer for regular Postgres? > Could it be possible that we are requesting a future WAL segment, say > WAL upto X is written and we are asking for X + 1? It could be that the > error message is misleading. > I don't think that should be possible. The LSN in the START_REPLICATION comes from the replica, where it's tracked as the last LSN received from the upstream. So that shouldn't be in the future. And it's doesn't seem to be suspiciously close to segment boundary either. In fact, the LSN in the message is 6/5AE67D79, but the "failed" segment is 000000010000000600000059, which is the *preceding* one. So it can't be in the future. > I do not have the information from pg_replication_slots as I have > terminated the test. I am fairly certain that I can reproduce this > again. I will gather both the restart_lsn and contents of pg_wal for the > failed segment. Is there any other information that would help debug > this further? > Hard to say. The best thing would be to have a reproducer script, ofc. If that's not possible, the information already requested seems like a good start. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company