Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1up9qt-0004oC-Kd for pgsql-general@arkaria.postgresql.org; Thu, 21 Aug 2025 18:17:32 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1up9qt-000tSd-2R for pgsql-general@arkaria.postgresql.org; Thu, 21 Aug 2025 18:17:31 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1up9qs-000tSV-OF for pgsql-general@lists.postgresql.org; Thu, 21 Aug 2025 18:17:31 +0000 Received: from fhigh-b6-smtp.messagingengine.com ([202.12.124.157]) by makus.postgresql.org with smtp (Exim 4.96) (envelope-from ) id 1up9qq-0013tK-2y for pgsql-general@lists.postgresql.org; Thu, 21 Aug 2025 18:17:30 +0000 Received: from phl-compute-11.internal (phl-compute-11.internal [10.202.2.51]) by mailfhigh.stl.internal (Postfix) with ESMTP id 5F4827A0148; Thu, 21 Aug 2025 14:17:29 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-11.internal (MEProxy); Thu, 21 Aug 2025 14:17:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=aklaver.com; h= cc:cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm2; t=1755800249; x=1755886649; bh=XTNAV5DLUo4ZxSMxOBrs2PFZFc4KWGz2vvdS+Q6slwg=; b= mWdAES5mM2Zwi+Ft3Y0J7RoMt0QVBxssjNHk29XYLI1oUMW1Zn/exgDfkjqpvqFf nXSYnUVHbJX+nV/0wMyssRg7d0nQsXA9tx8ehase2lZsR3oJLS0K9ibpip34AUCK LFKbMIq61FxG007xGJB9Fmwqoj+tA/dI0nCXxVCvLUUjnjZ58AVwVdEene6NgXrR GQ4C3eqrUBJ9BB/6UpQw5TPGEEc/zt+buvZlmuoFq1cJv+VCPu1GwyujCAmpRcIA hO77EX92s7TewFJ5lWnkuUmQ0bAMh4XAMV7QZt6d4Eti3ubRQ9L0W5HHo3cpM7B6 9zyGJW++zD4XoK3/MWEvxg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1755800249; x= 1755886649; bh=XTNAV5DLUo4ZxSMxOBrs2PFZFc4KWGz2vvdS+Q6slwg=; b=J Tj2OguiB4MoZ1saRP3E8ZmI1SHtI3HJYblsmT6y0Rq+oClf14T1MtMPiXEp2XMfF zR1ODuBeekAqHyNCWoqu7AkEt2UB7/u9j8TD0xKDzYqT6jO7s5SSerrsDkY8Kpb6 MzU2mXa+pzHedbabujYXRjLcmS/FhoMT5YnZMEeB7pFWXnY2MMDi9OjX9i3AI4q4 kgf8O1m67QYh5InXNSB2sdLOPwewUYm2DChaQAooDt6+t4/R54qgSLvUK3M5EZWs weI+Ve0u91bSxUJkSPAlhLtC7nXghc2FJpIyg+fi2X7IPIBC74R7Gvg30k6XoTHF BR02zgixjIXvxO8xvyxZg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtdefgdduieduleefucetufdoteggodetrf dotffvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfurfetoffkrfgpnffqhgenuceu rghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmnecujf gurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtvdejnecuhfhrohhmpeetughrihgr nhcumfhlrghvvghruceorggurhhirghnrdhklhgrvhgvrhesrghklhgrvhgvrhdrtghomh eqnecuggftrfgrthhtvghrnhepgfdugeejiedvffeiudfgvddvfedttddthfetiedtgeei veekieehheeghfeileeinecuffhomhgrihhnpehpghgsohhunhgtvghrrdhorhhgpdhgih hthhhusgdrtghomhenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhl fhhrohhmpegrughrihgrnhdrkhhlrghvvghrsegrkhhlrghvvghrrdgtohhmpdhnsggprh gtphhtthhopedvpdhmohguvgepshhmthhpohhuthdprhgtphhtthhopeguvghpvghsiies uggvphgvshiirdgtohhmpdhrtghpthhtohepphhgshhqlhdqghgvnhgvrhgrlheslhhish htshdrphhoshhtghhrvghsqhhlrdhorhhg X-ME-Proxy: Feedback-ID: i76984098:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 21 Aug 2025 14:17:28 -0400 (EDT) Message-ID: <2a3e4a8d-e8c2-46d6-ad7d-9e631ce6725e@aklaver.com> Date: Thu, 21 Aug 2025 11:17:27 -0700 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Streaming replica hangs periodically for ~ 1 second - how to diagnose/debug To: depesz@depesz.com Cc: PostgreSQL General References: <05969854-0d19-4726-ae1b-586659dd443b@aklaver.com> <25334887-f1c3-40a1-94b0-753c7d67ae2b@aklaver.com> Content-Language: en-US From: Adrian Klaver In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On 8/21/25 09:51, hubert depesz lubaczewski wrote: > On Thu, Aug 21, 2025 at 08:59:03AM -0700, Adrian Klaver wrote: >> Getting to the bottom of the bag of ideas: >> Have you looked at the OS system log for the time period involved? > > Yes. Mostly dmesg. Nothing interesting logged around the time. > >> You mentioned this seemed to involve PREPARE and DISCARD ALL. >> Is this the same set of statements or is it all over the place? > > No. From what I can tell it's random sample. > >> Also it would be helpful to know what bouncer you are actually using and >> what mode you are running in? > > pgBouncer, version 1.23.1. As for more... mostly transaction pooling. > Applications go using transaction pooling, but people (dbas, ops) have > session pooling. Have you looked at?: https://www.pgbouncer.org/changelog.html#pgbouncer-124x To see if anything stands out. Then there is: https://www.pgbouncer.org/config.html#max_prepared_statements The below may also be worth looking at: https://github.com/pgbouncer/pgbouncer/pull/1144 I can't help thinking that there is a caching issue at stake, though that is just a guess. > > Best regards, > > depesz > -- Adrian Klaver adrian.klaver@aklaver.com