Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1umcwU-0014Yu-Ul for pgsql-hackers@arkaria.postgresql.org; Thu, 14 Aug 2025 18:44:51 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1umcwT-008q2h-Ep for pgsql-hackers@arkaria.postgresql.org; Thu, 14 Aug 2025 18:44:49 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1umcwT-008q2Y-5G for pgsql-hackers@lists.postgresql.org; Thu, 14 Aug 2025 18:44:49 +0000 Received: from mail-qk1-x729.google.com ([2607:f8b0:4864:20::729]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.96) (envelope-from ) id 1umcwR-000mBR-0H for pgsql-hackers@lists.postgresql.org; Thu, 14 Aug 2025 18:44:49 +0000 Received: by mail-qk1-x729.google.com with SMTP id af79cd13be357-7e8706c880eso149674685a.3 for ; Thu, 14 Aug 2025 11:44:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bowt-ie.20230601.gappssmtp.com; s=20230601; t=1755197086; x=1755801886; darn=lists.postgresql.org; h=in-reply-to:references:subject:cc:to:from:message-id:date :content-transfer-encoding:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=NCtJs3iRmatfkr6Fscr7Dlid2tlbwedT5j60aV8x7xs=; b=fQRj3tj+HB14xXzhH498/SUQ0DwnS0vt0GcHH5pBI7gw9Vw0N+nZOqUtgZ9TKwqPnD dDGB3Yax+lj4A7MSXe5NVnZ2X8ceILCoSRy441qcvvC0NSQBqjcPrRFZ3ldZdqluoBgL 4IaAJTKbYp9PacDgH7z5rEV3NNbNm13H5uruVcyXUl79nIvNiUvMUHYt1kWyAzfLGFyv AO/j4xKxco7qwYbnSNGlMshqB8PT4q2Z8KEbhtgkqfsskScWuE0mih/vNOG9fPYWdLNa +Kufs2pwHEbmi16KwkkmFI83qH4ahIUHgcKUj9HGqTV2UnTTqT5gl02qH3yx1ia3dv0V CeRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755197086; x=1755801886; h=in-reply-to:references:subject:cc:to:from:message-id:date :content-transfer-encoding:mime-version:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=NCtJs3iRmatfkr6Fscr7Dlid2tlbwedT5j60aV8x7xs=; b=CYu1PpMVqut134jvUQpC0ojFJAmRt8wpo8J8SVJlUInmhFvSty+RD6ASLQvUU3vFx/ bnSLbLbTlt1dm17OZnr2/TLU1+ay4QkLJCt+FkJI1UaqERBLZezFegZaevQAGWln3xfy SC+NjMkkmTaDKoiRumCMKPuZtH7Bps3hUKOfKf1IqHitS/1Xja4Y9J3d+h0Oc14aMCvT AasUCBZopV/roJo6/a3+2LJOKa1miysUfenXL1WRNdl7MLFNrrb53boH8xNVXqiC+OBU XtOQvM69CMcM9Sn3i5e/dXOeI5wb71vpxutm1hs5MW87Z9JVMEpEOpY4aqGoWNcZLqUI 3J0g== X-Forwarded-Encrypted: i=1; AJvYcCX3IRakLKsC+2CinDYXvWx4Odt7YOotQiQ92p9Lx2ELzsgNL8kG/dvVXFdhtFq8E4NKTVBwAq/84oHaTKId@lists.postgresql.org X-Gm-Message-State: AOJu0Yy/ia6qPXpE6T6mcvEXTJyad9VAciBGQD1wrUC064bm+zP/cvoT ikIcFN9bLxVr2Gjn/rKR0kjDAK9PG2Y8141VBVqAJj6gvtyLs8j7vM0r+kv3bkHC5Q8= X-Gm-Gg: ASbGncuIFmCKLS6Z+HNPcwzZ+Um0B/AvTrikPTmlnDMIoJmAozLNdMPM0sNm6cgrsw9 x9atpuesYAtaGYxC5+UyeQTriyNFAo+mIGa2b1TTmNChDRLxcTxGPq8IzN7oFHIFov6T3d8u+d/ pPogmymgoWPf+kjrVqvKmcy5tDuJ0b/HdETLa3HWmNFHEgCsHff1SbxYKogYd/pvAr96xCbX/VQ u1rpkSjW0YDrZRz/wktkWsIPLKsan1PHnAu1B3sPAFeg3jFcoMNpdEguPmEmqPdgmxpvUmg/pkf lAefcLA+naqWNrI50dlW0o7YaJSdhz8Vl+VTr50i166cr5xsX4d0mU5ireCMrvxSPKlH5zERl+i 5ma1+ZIOzNZ1SJ+JHOiQzhkX9lLboBisQi7YtxKpl556rYX2TSDhL8tLY X-Google-Smtp-Source: AGHT+IEC3LAJ2pcfOUBsvK/WbnbVeuKXAvlJZvMprkEm2gMqj+rpWzvXv+2VdKDu0TkELh1PPOdUpw== X-Received: by 2002:ae9:e410:0:b0:7e6:f1ae:ec94 with SMTP id af79cd13be357-7e870481f5bmr415413985a.51.1755197085403; Thu, 14 Aug 2025 11:44:45 -0700 (PDT) Received: from localhost (syn-024-090-110-103.res.spectrum.com. [24.90.110.103]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7e87b4bd651sm12692385a.39.2025.08.14.11.44.44 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 14 Aug 2025 11:44:44 -0700 (PDT) Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 Date: Thu, 14 Aug 2025 14:44:44 -0400 Message-Id: From: "Peter Geoghegan" To: "Peter Geoghegan" , "Tomas Vondra" , "Andres Freund" Cc: "Thomas Munro" , "Nazir Bilal Yavuz" , "Robert Haas" , "Melanie Plageman" , "PostgreSQL Hackers" , "Georgios" , "Konstantin Knizhnik" , "Dilip Kumar" Subject: Re: index prefetching X-Mailer: aerc 0.20.0 References: <51b5f71b-5f19-4453-91ff-2b9f2a840c58@vondra.me> <6cb6109d-71d6-490c-8056-d8885081b008@vondra.me> <7c308720-5cd4-4508-8cc0-6e55a6352731@vondra.me> In-Reply-To: List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On Thu Aug 14, 2025 at 1:57 PM EDT, Peter Geoghegan wrote: > The only interesting thing about the flame graph is just how little > difference there seems to be (at least for this particular perf event > type). I captured method_io_uring.c DEBUG output from running each query in the server log, in the hope that it would shed some light on what's really goin= g on here. I think that it just might. I count a total of 12,401 distinct sleeps for the sequential/slow backwards scan test case: $ grep -E "wait_one with [1-9][0-9]* sleeps" sequential.txt | head 2025-08-14 14:35:03.278 EDT [2516983][client backend] [[unknown]][0/1:0] D= EBUG: 00000: wait_one with 1 sleeps 2025-08-14 14:35:03.278 EDT [2516983][client backend] [[unknown]][0/1:0] D= EBUG: 00000: wait_one with 1 sleeps 2025-08-14 14:35:03.278 EDT [2516983][client backend] [[unknown]][0/1:0] D= EBUG: 00000: wait_one with 1 sleeps 2025-08-14 14:35:03.278 EDT [2516983][client backend] [[unknown]][0/1:0] D= EBUG: 00000: wait_one with 1 sleeps 2025-08-14 14:35:03.278 EDT [2516983][client backend] [[unknown]][0/1:0] D= EBUG: 00000: wait_one with 1 sleeps 2025-08-14 14:35:03.278 EDT [2516983][client backend] [[unknown]][0/1:0] D= EBUG: 00000: wait_one with 1 sleeps 2025-08-14 14:35:03.279 EDT [2516983][client backend] [[unknown]][0/1:0] D= EBUG: 00000: wait_one with 1 sleeps 2025-08-14 14:35:03.279 EDT [2516983][client backend] [[unknown]][0/1:0] D= EBUG: 00000: wait_one with 1 sleeps 2025-08-14 14:35:03.279 EDT [2516983][client backend] [[unknown]][0/1:0] D= EBUG: 00000: wait_one with 1 sleeps 2025-08-14 14:35:03.279 EDT [2516983][client backend] [[unknown]][0/1:0] D= EBUG: 00000: wait_one with 1 sleeps $ grep -E "wait_one with [1-9][0-9]* sleeps" sequential.txt | awk '{ total = +=3D $11 } END { print total }' 12401 But there are only 3 such sleeps seen when the random backwards scan query = is run -- which might begin to explain the mystery of why it runs so much fast= er: $ grep -E "wait_one with [1-9][0-9]* sleeps" random.txt | awk '{ total +=3D= $11 } END { print total }' 104 --=20 Peter Geoghegan