Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1v7vqg-00A3iC-WD for pgsql-hackers@arkaria.postgresql.org; Sun, 12 Oct 2025 13:10:55 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1v7vqc-001Sgh-W3 for pgsql-hackers@arkaria.postgresql.org; Sun, 12 Oct 2025 13:10:51 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1v7vqc-001SgY-6i for pgsql-hackers@lists.postgresql.org; Sun, 12 Oct 2025 13:10:51 +0000 Received: from fout-b3-smtp.messagingengine.com ([202.12.124.146]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1v7vqZ-001kj1-1v for pgsql-hackers@postgresql.org; Sun, 12 Oct 2025 13:10:50 +0000 Received: from phl-compute-03.internal (phl-compute-03.internal [10.202.2.43]) by mailfout.stl.internal (Postfix) with ESMTP id 311261D00118; Sun, 12 Oct 2025 09:10:44 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-03.internal (MEProxy); Sun, 12 Oct 2025 09:10:44 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=burd.me; h=cc:cc :content-transfer-encoding:content-type:content-type:date:date :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:subject:subject:to:to; s=fm2; t= 1760274644; x=1760361044; bh=4tH4I0zCgJXqbJbcTYRchQ+090Kc8FnIXEc cwh8sAAE=; b=Ls/UGB1LWoKZhPxwMQzv0U6LyTvN5CiNlKhbD94S0ZfhyxEJz4d hw0bRZDRD6H+yAVZhTRD6TZiPGq05Dj1BTeMbX72suh8241L1KT4yifnte4bnbrp XfNftaEknIZJGde55Vfeyk09ISqzFVo+kaKz7SQ/0OTr6P4abos8Q12eYjDxik/E dunXct5nBPGdD/ivLr2p5mgV4pVqpDBUnGiCRdmTgSXn3bPWTXRPT36ekEsXBsME cs3BYDhq71P+zH3k2SOqKGK1HKWdWGWokYk2UZMO6YrsSLynB+9VenyVX/AY+IH7 eQBiWQYLzDY1GSuUxk0PqhjiiL+272GqBKQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:subject:subject:to:to:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t=1760274644; x= 1760361044; bh=4tH4I0zCgJXqbJbcTYRchQ+090Kc8FnIXEccwh8sAAE=; b=f MfjlCRgOIfyHqyhIpn5qEdWzt4+0hjWFvtLLKPYxMi7HR1930nZCJbQDqPtMk+Wy AL8wj4M0N/LWa1mUEK2NbMahVA1HHzWEr9XnwzIQCa6/6rEMT9qNoG4bFDXV3Z42 /H8M5NOOE/MFQgzbvIqu3J/9gqGCKxS4xx2Nb4vxgMIaCdULVH+I/ErDKDIjfaZ7 llGadl1m4Kp0mp8Z3/05KhWCGbF0QQ0MYZ90LQyIZsP7KvKONX5agabXM+RvTyBV OKPoPf5BRTwAFxQTrhN+29HEHNvp7Y2K1TKtaq3faDQqVOkIIYadmNNv/9shCisL 4qJfCSP6VHT/frx2aNZIg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtdeggdduudegkeeiucetufdoteggodetrf dotffvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfurfetoffkrfgpnffqhgenuceu rghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmnecujf gurhepfffhrhfvvefkjghfuffogggtgfgusehtqhertdertdejnecuhfhrohhmpefirhgv ghcuuehurhguuceoghhrvghgsegsuhhrugdrmhgvqeenucggtffrrghtthgvrhhnpeeule ekfeeljeekheevleeigffgtddtveffgeeigedufeekgfeuvdekledvfeejffenucffohhm rghinhepphhoshhtghhrvghsqhhlrdhorhhgpdhorhgrnhhgvghpihdrohhrghenucevlh hushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehgrhgvghessghu rhgurdhmvgdpnhgspghrtghpthhtohepgedpmhhouggvpehsmhhtphhouhhtpdhrtghpth htohepvgigtghluhhsihhonhesghhmrghilhdrtghomhdprhgtphhtthhopehthhhomhgr shdrmhhunhhrohesghhmrghilhdrtghomhdprhgtphhtthhopegrnhgurhgvshesrghnrg hrrgiivghlrdguvgdprhgtphhtthhopehpghhsqhhlqdhhrggtkhgvrhhssehpohhsthhg rhgvshhqlhdrohhrgh X-ME-Proxy: Feedback-ID: i675e48f3:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 12 Oct 2025 09:10:43 -0400 (EDT) Date: Sun, 12 Oct 2025 09:10:27 -0400 From: Greg Burd Reply-To: Greg Burd To: Alexander Lakhin , Thomas Munro Cc: Andres Freund , pgsql-hackers Message-ID: <3d6eb098-344e-4616-a9ce-21d08ccffd16@app.fastmail.com> In-Reply-To: <60bbebf4-0a80-4e5f-83c0-311c85f6bd5c@gmail.com> References: <60bbebf4-0a80-4e5f-83c0-311c85f6bd5c@gmail.com> Subject: Re: IO in wrong state on riscv64 X-Mailer: Mailspring MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On Sun, Oct 12, 2025, at 1:00 AM, Alexander Lakhin wrote: > Hello Thomas, > > 12.10.2025 06:35, Thomas Munro wrote: >> On Sun, Oct 12, 2025 at 2:00=E2=80=AFAM Alexander Lakhin >> wrote: >>> 2025-10-11 11:34:46.793 GMT =5B1169773:1=5D PANIC: =21=21=21pgaio=5F= io=5Fwait=7C >>> ioh->state changed from 0 to 1 at iteration 0 >>> =23 no other iteration number observed >> Can you please disassemble pgaio=5Fio=5Fupdate=5Fstate() and >> pgaio=5Fio=5Fwas=5Frecycled()=3F I wonder if the memory barriers are = not >> being generated correctly, causing the state and generation to be >> loaded out of order, or something like that... > > Please find those attached (gdb =22disass/m pgaio=5Fio=5Fupdate=5Fstate= =22 misses > the start of the function (but it's still disassembled below), so I > decided to share the whole output). > This is from clean master, without any modifications, but with the issu= e > confirmed for this build: > 2025-10-11 20:29:11.724 UTC =5B1679534:1=5D =5Bunknown=5D LOG:=C2=A0 co= nnection =20 > received: host=3D=5Blocal=5D > 2025-10-11 20:29:11.725 UTC =5B1679536:1=5D =5Bunknown=5D LOG:=C2=A0 co= nnection =20 > received: host=3D=5Blocal=5D > 2025-10-11 20:29:11.726 UTC =5B1679538:1=5D =5Bunknown=5D LOG:=C2=A0 co= nnection =20 > received: host=3D=5Blocal=5D > 2025-10-11 20:29:11.724 UTC =5B1679533:1=5D =5Bunknown=5D LOG:=C2=A0 co= nnection =20 > received: host=3D=5Blocal=5D > 2025-10-11 20:29:11.724 UTC =5B1679537:1=5D =5Bunknown=5D LOG:=C2=A0 co= nnection =20 > received: host=3D=5Blocal=5D > 2025-10-11 20:29:11.724 UTC =5B1679535:1=5D =5Bunknown=5D LOG:=C2=A0 co= nnection =20 > received: host=3D=5Blocal=5D > 2025-10-11 20:29:11.729 UTC =5B1679539:1=5D =5Bunknown=5D LOG:=C2=A0 co= nnection =20 > received: host=3D=5Blocal > ... > 2025-10-11 20:29:11.778 UTC =5B1679537:3=5D =5Bunknown=5D LOG:=C2=A0 co= nnection =20 > authorized: user=3Ddebian database=3Dregression =20 > application=5Fname=3Dpg=5Fregress/create=5Fschema > 2025-10-11 20:29:11.778 UTC =5B1679533:3=5D =5Bunknown=5D LOG:=C2=A0 co= nnection =20 > authorized: user=3Ddebian database=3Dregression =20 > application=5Fname=3Dpg=5Fregress/create=5Ftype > 2025-10-11 20:29:11.778 UTC =5B1679539:3=5D =5Bunknown=5D LOG:=C2=A0 co= nnection =20 > authorized: user=3Ddebian database=3Dregression =20 > application=5Fname=3Dpg=5Fregress/create=5Fprocedure > 2025-10-11 20:29:11.778 UTC =5B1679536:3=5D =5Bunknown=5D LOG:=C2=A0 co= nnection =20 > authorized: user=3Ddebian database=3Dregression =20 > application=5Fname=3Dpg=5Fregress/create=5Fmisc > 2025-10-11 20:29:11.778 UTC =5B1679538:3=5D =5Bunknown=5D LOG:=C2=A0 co= nnection =20 > authorized: user=3Ddebian database=3Dregression =20 > application=5Fname=3Dpg=5Fregress/create=5Ftable > 2025-10-11 20:29:11.778 UTC =5B1679535:3=5D =5Bunknown=5D LOG:=C2=A0 co= nnection =20 > authorized: user=3Ddebian database=3Dregression =20 > application=5Fname=3Dpg=5Fregress/create=5Ffunction=5Fc > 2025-10-11 20:29:11.790 UTC =5B1679534:2=5D =5Bunknown=5D =46ATAL:=C2=A0= IO in wrong =20 > state: 0 > 2025-10-11 20:29:11.790 UTC =5B1679534:3=5D =5Bunknown=5D BACKTRACE: > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 postgres: debian regression= =5Blocal=5D authentication(+0x4371ec) =20 > =5B0x555565cd61ec=5D > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 postgres: debian regression= =5Blocal=5D authentication(+0x442bb2) =20 > =5B0x555565ce1bb2=5D > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 postgres: debian regression= =5Blocal=5D =20 > authentication(StartBufferIO+0x66) =5B0x555565ce1a10=5D > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 postgres: debian regression= =5Blocal=5D authentication(+0x43df60) =20 > =5B0x555565cdcf60=5D > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 postgres: debian regression= =5Blocal=5D =20 > authentication(StartReadBuffer+0x308) =5B0x555565cdc8a4=5D > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 postgres: debian regression= =5Blocal=5D =20 > authentication(ReadBufferExtended+0x64) =5B0x555565cdaace=5D > ... > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 postgres: debian regression= =5Blocal=5D =20 > authentication(postmaster=5Fchild=5Flaunch+0x132) =5B0x555565c7787c=5D > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 postgres: debian regression= =5Blocal=5D authentication(+0x3dcd52) =20 > =5B0x555565c7bd52=5D > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 postgres: debian regression= =5Blocal=5D =20 > authentication(InitProcessGlobals+0) =5B0x555565c79c96=5D > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 postgres: debian regression= =5Blocal=5D authentication(+0x3194b2) =20 > =5B0x555565bb84b2=5D > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /lib/riscv64-linux-gnu/libc= .so.6(+0x2791c) =5B0x7fffa690891c=5D > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /lib/riscv64-linux-gnu/libc= .so.6(=5F=5Flibc=5Fstart=5Fmain+0x74) =20 > =5B0x7fffa69089c4=5D > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 postgres: debian regression= =5Blocal=5D =20 > authentication(=5Fstart+0x20) =5B0x5555659842c8=5D > >> The previous failure on greenfly was a TIMEOUT in the same test, as if= >> a query was hanging. > > Yeah, I'll try to reproduce it too... > >> On Sun, Oct 12, 2025 at 2:00=E2=80=AFAM Alexander Lakhin >> wrote: >>> I've managed to reproduce it using qemu-system-riscv64 with Debian tr= ixie >> Huh, that's interesting. What is the host architecture=3F When I saw= >> that error myself and wondered about memory order, I dismissed the >> idea of trying with qemu, figuring that my x86 host's TSO would affect= >> the coherency, but thinking again about that... I guess the compiler >> might still reorder during riscv codegen if there is something wrong >> with the barrier support, and even if it doesn't, the binary >> translation to x86 might also feel free to reorder stuff if there are >> no barrier instructions to prevent it=3F Or maybe that doesn't happen= >> but your host is ARM=3F > > I use AMD Ryzen 9 7900X, Ubuntu 24.04 and run the risc machine with: > qemu-system-riscv64 -machine virt -m 1G -smp 8 -cpu rv64 -device =20 > virtio-blk-device,drive=3Dhd =5C > -drive file=3D.../dqib=5Friscv64-virt/image.qcow2,if=3Dnone,id=3Dhd -de= vice =20 > virtio-net-device,netdev=3Dnet =5C > -netdev user,id=3Dnet,hostfwd=3Dtcp::22222-:22 -bios =20 > /usr/lib/riscv64-linux-gnu/opensbi/generic/fw=5Fjump.elf =5C > -kernel /usr/lib/u-boot/qemu-riscv64=5Fsmode/uboot.elf -object =20 > rng-random,filename=3D/dev/urandom,id=3Drng =5C > -device virtio-rng-device,rng=3Drng -nographic -append > =22root=3DLABEL=3Drootfs =20 > console=3DttyS0=22 > > I don't know what hardware greenfly uses (CC'ing Greg in case he'd > like to share some info on this), Hello Alexander, thanks for digging into this issue. The animal =22greenfly=22 is an OrangePi RV2 =5B1=5D with 8GB of RAM. It= has the optional 256GB eMMC module and uses that for the root filesystem. =20 It's running Ubuntu 24.04.3 and I'm compiling with clang 18-1.3. I think the rest of the details are in the animal registration or in the logs, but let me know if you have questions or if there are things that you'd like me to test. > but timings are similar to what > I'm seeing: > https://buildfarm.postgresql.org/cgi-bin/show=5Fstage=5Flog.pl=3Fnm=3Dg= reenfly&dt=3D2025-10-09%2000%3A06%3A03&stg=3Dcheck > ok 160=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - select=5Fparallel=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 5019 m= s > > =60make check=60 on mine, with select=5Fparallel repeated: > ok 160=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - select=5Fparallel=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 6042 m= s > ok 161=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - select=5Fparallel=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 6089 m= s > ok 162=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - select=5Fparallel=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 6116 m= s > > copperhead and boomslang, which are using real RISC hardware =5B1=5D, s= how > better timings: > https://buildfarm.postgresql.org/cgi-bin/show=5Fstage=5Flog.pl=3Fnm=3Dc= opperhead&dt=3D2025-10-11%2019%3A36%3A33&stg=3Dcheck > ok 160=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - select=5Fparallel=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2677 m= s > > https://buildfarm.postgresql.org/cgi-bin/show=5Fstage=5Flog.pl=3Fnm=3Db= oomslang&dt=3D2025-10-12%2002%3A17%3A53&stg=3Dcheck > ok 160=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - select=5Fparallel=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3651 m= s > > Thank you for looking into this=21 > > =5B1=5D =20 > https://www.postgresql.org/message-id/3db97903-884f-4b0c-b1cd-d7442e71e= a75%40app.fastmail.com > > Best regards, > Alexander > Attachments: > * pgaio=5Fio=5Fupdate=5Fstate.asm > * pgaio=5Fio=5Fwas=5Frecycled.asm best. -greg =5B1=5D http://www.orangepi.org/html/hardWare/computerAndMicrocontrollers/details= /Orange-Pi-RV2.html =20