Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nakAK-0003rW-Pd for pgsql-committers@arkaria.postgresql.org; Sat, 02 Apr 2022 20:16:09 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1nakAJ-0002cp-M3 for pgsql-committers@arkaria.postgresql.org; Sat, 02 Apr 2022 20:16:07 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nakAH-0002cc-SP for pgsql-committers@lists.postgresql.org; Sat, 02 Apr 2022 20:16:07 +0000 Received: from out4-smtp.messagingengine.com ([66.111.4.28]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nakAC-0001yL-LR for pgsql-committers@lists.postgresql.org; Sat, 02 Apr 2022 20:16:04 +0000 Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id DD29C5C002F; Sat, 2 Apr 2022 16:15:58 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Sat, 02 Apr 2022 16:15:58 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=anarazel.de; h= cc:cc:content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm1; bh=OEPIuhhXb8ncio0zpiLh/qLpXZZdP3z5vqod8v ylte0=; b=Hmkv5oRR+ORAi/WIcMIVS971KdizeIYiPBjhul52RlQYQL6pi7pbTX lfO6tuKSyn3dY9yt3ZmHhPrQ1aLCUAs4+C3mc28QuFzZk34nZco3bi9qfEeEqaNQ KK7IEKMMkB3g3/wzTZZuB4Wlkky3+G2EdcCsUlPG0nO5puCqNHTvovTU8cACSJFe x5nJmi5ABxN6Xffbggn+LMvqi7HIQwjUVFQcZEtVgkspbzVzvx+w4auX6myqoDAy UYY2sEZpVcCqfp8nRojtIUDLMy/1LLAZgP1UJ7B4GRgsxbpXf2Wi1amjBdpuaUJp 934djzb3v/b4U6WHlYxpuPOSqbGcPXag== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=OEPIuhhXb8ncio0zp iLh/qLpXZZdP3z5vqod8vylte0=; b=WlK/ROUAGBtqaXNnMQtpDMCVfxVQOqbm+ iSPf5rtag4g2L6XF653L2Ab1sF73Zg9bURUR7twMyd/g4pWXYHc9ZZSPxMr2uPL7 L10hisQV9PYQAuy3p7r9xh8Hdbwj7rIdxXo32WJbS5PlJio9X/ITnrHRw/lF73Ay M60ZH1xSHa9kwfMBTZpM2/sccpDRSI3TSyvuGcTRh5PHVIxZK2aREAteRMmvncrr COiRlq+ofSIKvFpQG0wkeVE7V69WxOeq1za+bCTaPus6jNLSMSTECCZUE5hEzOxB mBkFDOZb2SluJ8FouujSjurpCy4uBficFwM913Hffuuqo0YIF8NZw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrudeikedgudeglecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpeffhffvuffkfhggtggujgesthdtredttddtvdenucfhrhhomheptehnughr vghsucfhrhgvuhhnugcuoegrnhgurhgvshesrghnrghrrgiivghlrdguvgeqnecuggftrf grthhtvghrnhepffdvvdevleelgffhheeitdevhfefgfeivdevieeltdffkeefgffguedt keeiheejnecuffhomhgrihhnpehpohhsthhgrhdrvghsnecuvehluhhsthgvrhfuihiivg eptdenucfrrghrrghmpehmrghilhhfrhhomheprghnughrvghssegrnhgrrhgriigvlhdr uggv X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sat, 2 Apr 2022 16:15:58 -0400 (EDT) Date: Sat, 2 Apr 2022 13:15:57 -0700 From: Andres Freund To: John Naylor Cc: John Naylor , pgsql-committers@lists.postgresql.org Subject: Re: pgsql: Specialize tuplesort routines for different kinds of abbreviated Message-ID: <20220402201557.thanbsxcql5lk6pc@alap3.anarazel.de> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk Hi, On 2022-04-02 16:06:40 +0700, John Naylor wrote: > So far, kestrel and tamandua don't like this, and they both use > "-fsanitize=undefined,alignment". I'll try to reproduce locally in a > bit. Just hit this in my development environment. FWIW, I found it hard to work with ubsan without applying 0002 from https://postgr.es/m/20220323173537.ll7klrglnp4gn2um%40alap3.anarazel.de I plan to polish that to be able to commit it, but won't have cycles before the CF ends. with that applied CFLAGS=-fsanitize=alignment,undefined,address -fno-sanitize-recover=all and export ASAN_OPTIONS="detect_leaks=0:disable_coredump=0:print_stacktrace=1:abort_on_error=1" UBSAN_OPTIONS="disable_coredump=0:print_stacktrace=1:abort_on_error=1" I get a backtrace to investigate. Unfortunately abort_on_error=1 also removes the nicer error message :(. So I ran both. It triggers reliably on CLUSTER clstr_expression USING clstr_expression_minus_a; for me. /home/andres/src/postgresql/src/backend/utils/sort/tuplesort.c:723:23: runtime error: load of value 150, which is not a valid value for type '_Bool' #0 __ubsan::__ubsan_handle_load_invalid_value_abort (Data=0x7ff65e8fada0, Val=70) at ../../../../src/libsanitizer/ubsan/ubsan_handlers.cpp:548 #1 0x00007ff65a649093 in qsort_tuple_int32_compare (a=0x62b000fc0240, b=0x62b000fc0258, state=0x625000066a20) at /home/andres/src/postgresql/src/backend/utils/sort/tuplesort.c:723 #2 0x00007ff65a64bc7b in qsort_tuple_int32 (data=0x62b000fc0240, n=133, arg=0x625000066a20) at /home/andres/src/postgresql/src/include/lib/sort_template.h:313 #3 0x00007ff65a672d59 in tuplesort_sort_memtuples (state=0x625000066a20) at /home/andres/src/postgresql/src/backend/utils/sort/tuplesort.c:3613 #4 0x00007ff65a6617ec in tuplesort_performsort (state=0x625000066a20) at /home/andres/src/postgresql/src/backend/utils/sort/tuplesort.c:2154 #5 0x00007ff6586965dd in heapam_relation_copy_for_cluster (OldHeap=0x7ff64d893308, NewHeap=0x7ff64d896fc0, OldIndex=0x7ff64d89c860, use_sort=true, OldestXmin=23706, xid_cutoff=0x7ffd4650a480, multi_cutoff=0x7ffd4650a490, num_tuples=0x7ffd4650a4a0, tups_vacuumed=0x7ffd4650a4c0, tups_recently_dead=0x7ffd4650a4e0) at /home/andres/src/postgresql/src/backend/access/heap/heapam_handler.c:955 #6 0x00007ff658d6c4b7 in table_relation_copy_for_cluster (OldTable=0x7ff64d893308, NewTable=0x7ff64d896fc0, OldIndex=0x7ff64d89c860, use_sort=true, OldestXmin=23706, xid_cutoff=0x7ffd4650a480, multi_cutoff=0x7ffd4650a490, num_tuples=0x7ffd4650a4a0, tups_vacuumed=0x7ffd4650a4c0, tups_recently_dead=0x7ffd4650a4e0) at /home/andres/src/postgresql/src/include/access/tableam.h:1658 #7 0x00007ff658d728aa in copy_table_data (OIDNewHeap=73728, OIDOldHeap=47052, OIDOldIndex=47078, verbose=false, pSwapToastByContent=0x7ffd4650a670, pFreezeXid=0x7ffd4650a680, pCutoffMulti=0x7ffd4650a690) at /home/andres/src/postgresql/src/backend/commands/cluster.c:913 #8 0x00007ff658d6ff9e in rebuild_relation (OldHeap=0x7ff64d893308, indexOid=47078, verbose=false) at /home/andres/src/postgresql/src/backend/commands/cluster.c:606 #9 0x00007ff658d6e622 in cluster_rel (tableOid=47052, indexOid=47078, params=0x7ffd4650a7b0) at /home/andres/src/postgresql/src/backend/commands/cluster.c:427 #10 0x00007ff658d6d774 in cluster (pstate=0x619000001aa0, stmt=0x625000005d00, isTopLevel=true) at /home/andres/src/postgresql/src/backend/commands/cluster.c:195 #11 0x00007ff659d425b1 in standard_ProcessUtility (pstmt=0x625000006010, queryString=0x625000005220 "CLUSTER clstr_expression USING clstr_expression_minus_a;", readOnlyTree=false, context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0, dest=0x6250000060e0, qc=0x7ffd4650ae40) at /home/andres/src/postgresql/src/backend/tcop/utility.c:862 #12 0x00007ff659d40901 in ProcessUtility (pstmt=0x625000006010, queryString=0x625000005220 "CLUSTER clstr_expression USING clstr_expression_minus_a;", readOnlyTree=false, context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0, dest=0x6250000060e0, qc=0x7ffd4650ae40) (rr) p/x *b $4 = {tuple = 0x62500006ba78, datum1 = 0x7ff659cc7e9e, isnull1 = 0x46, srctape = 0x7ff6} There's definitely something borked - looks like this is ending up with bogus pointers? Using rr to set a watchpoint on isnull1, and continuing backward I see the memory written to with the following stack: (rr) watch -l b->isnull1 Hardware watchpoint 3: -location b->isnull1 (rr) reverse-continue Continuing. Hardware watchpoint 3: -location b->isnull1 Old value = 70 New value = 190 0x00007ff65a65fbe4 in puttuple_common (state=0x625000066a20, tuple=0x7ffd4650a150) at /home/andres/src/postgresql/src/backend/utils/sort/tuplesort.c:2000 2000 state->memtuples[state->memtupcount++] = *tuple; (rr) bt #0 0x00007ff65a65fbe4 in puttuple_common (state=0x625000066a20, tuple=0x7ffd4650a150) at /home/andres/src/postgresql/src/backend/utils/sort/tuplesort.c:2000 #1 0x00007ff65a65a478 in tuplesort_putheaptuple (state=0x625000066a20, tup=0x625000071d28) at /home/andres/src/postgresql/src/backend/utils/sort/tuplesort.c:1810 #2 0x00007ff658696300 in heapam_relation_copy_for_cluster (OldHeap=0x7ff64d893308, NewHeap=0x7ff64d896fc0, OldIndex=0x7ff64d89c860, use_sort=true, OldestXmin=23706, xid_cutoff=0x7ffd4650a480, multi_cutoff=0x7ffd4650a490, num_tuples=0x7ffd4650a4a0, tups_vacuumed=0x7ffd4650a4c0, (note new/old are kind of switched around due to continuing in reverse) and then Hardware watchpoint 3: -location b->isnull1 Old value = 190 New value = false __memset_evex_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:143 143 ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S: No such file or directory. (rr) bt #0 __memset_evex_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:143 #1 0x00007ff654d2c3c4 in __asan::Allocator::Allocate (this=this@entry=0x7ff654e3ce00 <__asan::instance>, size=, size@entry=24640, alignment=, alignment@entry=8, stack=stack@entry=0x7ffd465092c0, alloc_type=alloc_type@entry=__asan::FROM_MALLOC, can_fill=can_fill@entry=true) at ../../../../src/libsanitizer/asan/asan_allocator.cpp:598 #2 0x00007ff654d28a9b in __asan::asan_malloc (size=size@entry=24640, stack=stack@entry=0x7ffd465092c0) at ../../../../src/libsanitizer/asan/asan_allocator.cpp:964 #3 0x00007ff654db998f in __interceptor_malloc (size=24640) at ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:70 #4 0x00007ff65a5c4f37 in AllocSetAlloc (context=0x625000066900, size=24576) at /home/andres/src/postgresql/src/backend/utils/mmgr/aset.c:740 #5 0x00007ff65a60e385 in palloc (size=24576) at /home/andres/src/postgresql/src/backend/utils/mmgr/mcxt.c:1082 #6 0x00007ff65a6513dc in tuplesort_begin_batch (state=0x625000066a20) at /home/andres/src/postgresql/src/backend/utils/sort/tuplesort.c:963 #7 0x00007ff65a6501d9 in tuplesort_begin_common (workMem=65536, coordinate=0x0, randomAccess=false) at /home/andres/src/postgresql/src/backend/utils/sort/tuplesort.c:878 Greetings, Andres Freund