Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1uomsB-00AC2y-Tw for pgsql-general@arkaria.postgresql.org; Wed, 20 Aug 2025 17:45:21 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1uomsB-00AZtd-6n for pgsql-general@arkaria.postgresql.org; Wed, 20 Aug 2025 17:45:19 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1uomsA-00AZtV-AA for pgsql-general@lists.postgresql.org; Wed, 20 Aug 2025 17:45:19 +0000 Received: from fout-a3-smtp.messagingengine.com ([103.168.172.146]) by makus.postgresql.org with smtp (Exim 4.96) (envelope-from ) id 1uoms8-000syc-0X for pgsql-general@lists.postgresql.org; Wed, 20 Aug 2025 17:45:18 +0000 Received: from phl-compute-06.internal (phl-compute-06.internal [10.202.2.46]) by mailfout.phl.internal (Postfix) with ESMTP id 79487EC0593; Wed, 20 Aug 2025 13:45:15 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-06.internal (MEProxy); Wed, 20 Aug 2025 13:45:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=aklaver.com; h= cc:cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm2; t=1755711915; x=1755798315; bh=uQ/9nOKnOWgLBY8r9w2CffaXSbEsai58U4uJOWzy7gg=; b= iUVSjluYroufgWcFefThSuJA3R31WP4GTp2XYJqykGwYw4VzAYiWJs38LRtOH9jc 6LcFQUkTowNhoRsm9auk+Ze9LpP1MGcMRIVro93pTa0u8cjkwRgJywyaevzIkITY MRLlIJqmvV5Lqyc/1C66MCn4YsEMTgVKW9al/0iDQE+uGvMDKsn8eyV1CkYrJzK0 WiCDKt+fR2UiHJLtXjJnzw1z5mu8V+0OiHZj7yB0lEZhYPycCnv+M7xVe+/GFdEr H8nLL31Lm4PQr8EBsKzfwRPH6r3zNU1ZFcKNVVQtl8ZE8fhhAg5kTt/RkdePN5fZ 8QS4v68iQ60aUoqP3JVTtA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1755711915; x= 1755798315; bh=uQ/9nOKnOWgLBY8r9w2CffaXSbEsai58U4uJOWzy7gg=; b=g uVdN1FebivNskYD9YykIe5rZFAnMXBLBLJHk8g6gri/7YK/ly1DDMOriga8S+zAx fhc9Ad+FAR3HW3zEUTtSYcO7o7g1wLevVVh8B8dY+jpzKS2MF3i/z8yPqOZ2J3gq /IDtEibMs8TR5abuQjsJhT1O7YEIsxa9seDbt4Y76YTvWt5IbSzH7Dj28au0kOvg j99sZcwvYwud+87jfHH7JfgIduZBeuhn4Q8k7nJW4wStpBYHho+Eh4Z4oqIxg6j+ NVZNcqycDC6z+GECAJoUsvEYj7tJf8Z9q8TAoXaUoEmTO1MjojgL8diuDhcgfjTk AEGwlwvfMLe224lgxjDog== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtdefgdduheeltddtucetufdoteggodetrf dotffvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfurfetoffkrfgpnffqhgenuceu rghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmnecujf gurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtvdejnecuhfhrohhmpeetughrihgr nhcumfhlrghvvghruceorggurhhirghnrdhklhgrvhgvrhesrghklhgrvhgvrhdrtghomh eqnecuggftrfgrthhtvghrnhephfeviefhveelffeftdehudekveefhfeftdegieefveet fffgfeehtdfftedutedtnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrg hilhhfrhhomheprggurhhirghnrdhklhgrvhgvrhesrghklhgrvhgvrhdrtghomhdpnhgs pghrtghpthhtohepvddpmhhouggvpehsmhhtphhouhhtpdhrtghpthhtohepuggvphgvsh iiseguvghpvghsiidrtghomhdprhgtphhtthhopehpghhsqhhlqdhgvghnvghrrghlsehl ihhsthhsrdhpohhsthhgrhgvshhqlhdrohhrgh X-ME-Proxy: Feedback-ID: i76984098:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 20 Aug 2025 13:45:14 -0400 (EDT) Message-ID: Date: Wed, 20 Aug 2025 10:45:13 -0700 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Streaming replica hangs periodically for ~ 1 second - how to diagnose/debug To: depesz@depesz.com Cc: PostgreSQL General References: <05969854-0d19-4726-ae1b-586659dd443b@aklaver.com> Content-Language: en-US From: Adrian Klaver In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On 8/20/25 09:08, hubert depesz lubaczewski wrote: > On Wed, Aug 20, 2025 at 08:14:47AM -0700, Adrian Klaver wrote: >> Hmm. >> >> From initial post: >> >> "For ~ 1 second there are no logs going to log (we usually have at 5-20 >> messages logged per second), no connection, nothing. And then we get >> bunch (30+) messages with the same milisecond time." >> >> Are the 30+ messages all coming in on one connection or multiple >> connections? > > Multiple connections. > >> Also to be clear these are statements that are being run on the replica >> locally, correct? > > What do you mean locally? I should have been clearer. Are the queries being run against the replica or the primary? > > Application servers are *not* on the same server as the db. So no, they > are not local. Where are the application servers relative to the replica server? How many applications servers are hitting the database? > >> Does the AWS monitoring indicate any issues? > > Nope. All looks "fine". > > Same for our monitoring - CPU usage, iops, load. > > Best regards, > > depesz > -- Adrian Klaver adrian.klaver@aklaver.com