Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oOg8F-0000KK-QD for pgsql-hackers@arkaria.postgresql.org; Thu, 18 Aug 2022 14:04:23 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1oOg8E-0000wt-CS for pgsql-hackers@arkaria.postgresql.org; Thu, 18 Aug 2022 14:04:22 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oOg8E-0000wj-2w for pgsql-hackers@lists.postgresql.org; Thu, 18 Aug 2022 14:04:22 +0000 Received: from mail-io1-xd33.google.com ([2607:f8b0:4864:20::d33]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1oOg8A-0000W8-4S for pgsql-hackers@postgresql.org; Thu, 18 Aug 2022 14:04:20 +0000 Received: by mail-io1-xd33.google.com with SMTP id o2so1138019iof.8 for ; Thu, 18 Aug 2022 07:04:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telsasoft-com.20210112.gappssmtp.com; s=20210112; h=user-agent:in-reply-to:content-disposition:mime-version:references :message-id:subject:cc:to:from:date:from:to:cc; bh=yEqsc9oHTPmZWZd7sDJJEaR3Orz3m4dZE5DNgPDMgLM=; b=g4jyDPRIWgeONJoYjLpT+fOyMMxhvcSjml/BOUnfys7hqfc1qNoNN/tR72S2NA0dYd gNb9ee51KffTVSI7ldDBOYm1mPBhcYtWjK1rXrLQZ+EUsYYRDKFY+mejkhe5UM+agW7e MOiBmx4xtr2CdOKxUlCM/tAEdW9BgioGKxhjpvsDniyXVX8RewycDG86i1lWXLPJ32J+ rIbSMsQSUQj3MUErogTb6nrM6Qv59+vVYFs39pTksM2WU5gZs4lzzwWBLO18UB5xbPAj tjQyXlBJNSgWCCRsYR7NNTcP7Zh7osxlJmUGmFRp0iNNtaOKRgnIEpYocxWlemEIo9Mp acmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=user-agent:in-reply-to:content-disposition:mime-version:references :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc; bh=yEqsc9oHTPmZWZd7sDJJEaR3Orz3m4dZE5DNgPDMgLM=; b=o9lElRc8LRzhwL/zpZWZe0KgcItzSQFwJjAKmK1NsokGCJy4WCqaRlDxZkA7xSCubv EVXYkzcetnyCgkfsYdmDWguHkSfB0f0Cl0v1yEy1+6ESkKbGkTwLSNaJxmXx3CotY2EF 5kgbU+ijO58ewWH3xO7G0da+QEwlJKdASOK5+ky1kZg15qmFLfOGge2WuS8upy0DzWR+ g4cIkzuzMIBKOzxPCaFPrI/p22gfMjJ8wiguORvhBl21uuvj204SAQfAWmvUL5uOzWCQ 25OBQRO94NxYfQbkXoB1AEcWfWYe8pg0lGSfYcAznuxg56jdsN3JDMx9HdW1aEys3JVy x8BQ== X-Gm-Message-State: ACgBeo2pcLj1hV3/lNnEf4IojKLf4jwL7HYLhnZMFrX5xFlntlLuwSEO MSfMYwcAxChPBO7fGCSkkeO5HQ== X-Google-Smtp-Source: AA6agR7Tsfe5S8LjTcBhbaWwApron5gABY/GR+LFLXdgoU3FrQxRU1vnvKWgznHjbiOog8xe9T4kVA== X-Received: by 2002:a05:6638:359:b0:343:5ebe:f03c with SMTP id x25-20020a056638035900b003435ebef03cmr1462806jap.209.1660831457201; Thu, 18 Aug 2022 07:04:17 -0700 (PDT) Received: from pryzbyj.telsasoft (charmander.telsasoft.com. [50.244.222.1]) by smtp.gmail.com with ESMTPSA id f8-20020a056602070800b00678ea5747dbsm869795iox.11.2022.08.18.07.04.16 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 18 Aug 2022 07:04:16 -0700 (PDT) Received: by pryzbyj.telsasoft (Postfix, from userid 1000) id C816780071F; Thu, 18 Aug 2022 09:04:15 -0500 (CDT) Date: Thu, 18 Aug 2022 09:04:15 -0500 From: Justin Pryzby To: pgsql-hackers@postgresql.org Cc: Amit Kapila , Masahiko Sawada Subject: Re: pg15b3: crash in paralell vacuum Message-ID: <20220818140415.GN26426@telsasoft.com> References: <20220818133406.GL26426@telsasoft.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20220818133406.GL26426@telsasoft.com> User-Agent: Mutt/1.9.4 (2018-02-28) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On Thu, Aug 18, 2022 at 08:34:06AM -0500, Justin Pryzby wrote: > Unfortunately, it looks like the RPM packages are compiled with -O2, so this is > of limited use. So I'll be back shortly with more... #3 0x00000000006874f1 in parallel_vacuum_process_all_indexes (pvs=0x25bdce0, num_index_scans=0, vacuum=vacuum@entry=false) at vacuumparallel.c:611 611 Assert(indstats->status == PARALLEL_INDVAC_STATUS_INITIAL); (gdb) p *pvs $1 = {pcxt = 0x25bc1e0, indrels = 0x25bbf70, nindexes = 8, shared = 0x7fc5184393a0, indstats = 0x7fc5184393e0, dead_items = 0x7fc5144393a0, buffer_usage = 0x7fc514439280, wal_usage = 0x7fc514439240, will_parallel_vacuum = 0x266d818, nindexes_parallel_bulkdel = 5, nindexes_parallel_cleanup = 0, nindexes_parallel_condcleanup = 5, bstrategy = 0x264f120, relnamespace = 0x0, relname = 0x0, indname = 0x0, status = PARALLEL_INDVAC_STATUS_INITIAL} (gdb) p *indstats $2 = {status = 11, parallel_workers_can_process = false, istat_updated = false, istat = {num_pages = 0, estimated_count = false, num_index_tuples = 0, tuples_removed = 0, pages_newly_deleted = 0, pages_deleted = 1, pages_free = 0}} (gdb) bt f ... #3 0x00000000006874f1 in parallel_vacuum_process_all_indexes (pvs=0x25bdce0, num_index_scans=0, vacuum=vacuum@entry=false) at vacuumparallel.c:611 indstats = 0x7fc5184393e0 i = 0 nworkers = 2 new_status = PARALLEL_INDVAC_STATUS_NEED_CLEANUP __func__ = "parallel_vacuum_process_all_indexes" #4 0x0000000000687ef0 in parallel_vacuum_cleanup_all_indexes (pvs=, num_table_tuples=num_table_tuples@entry=409149, num_index_scans=, estimated_count=estimated_count@entry=true) at vacuumparallel.c:486 No locals. #5 0x00000000004f80b8 in lazy_cleanup_all_indexes (vacrel=vacrel@entry=0x25bc510) at vacuumlazy.c:2679 reltuples = 409149 estimated_count = true #6 0x00000000004f884a in lazy_scan_heap (vacrel=vacrel@entry=0x25bc510) at vacuumlazy.c:1278 rel_pages = 67334 blkno = 67334 next_unskippable_block = 67334 next_failsafe_block = 0 next_fsm_block_to_vacuum = 0 dead_items = 0x7fc5144393a0 vmbuffer = 1300 next_unskippable_allvis = true skipping_current_range = false initprog_index = {0, 1, 5} initprog_val = {1, 67334, 11184809} __func__ = "lazy_scan_heap" #7 0x00000000004f925f in heap_vacuum_rel (rel=0x7fc52df6b820, params=0x7ffd74f74620, bstrategy=0x264f120) at vacuumlazy.c:534 vacrel = 0x25bc510 verbose = true instrument = aggressive = false skipwithvm = true frozenxid_updated = false minmulti_updated = false OldestXmin = 32759288 FreezeLimit = 4277726584 OldestMxact = 157411 MultiXactCutoff = 4290124707 orig_rel_pages = 67334 new_rel_pages = new_rel_allvisible = 4 ru0 = {tv = {tv_sec = 1660830451, tv_usec = 473980}, ru = {ru_utime = {tv_sec = 0, tv_usec = 317891}, ru_stime = {tv_sec = 1, tv_usec = 212372}, {ru_maxrss = 74524, __ru_maxrss_word = 74524}, {ru_ixrss = 0, __ru_ixrss_word = 0}, {ru_idrss = 0, __ru_idrss_word = 0}, {ru_isrss = 0, __ru_isrss_word = 0}, {ru_minflt = 18870, __ru_minflt_word = 18870}, {ru_majflt = 0, __ru_majflt_word = 0}, {ru_nswap = 0, __ru_nswap_word = 0}, {ru_inblock = 1124750, __ru_inblock_word = 1124750}, {ru_oublock = 0, __ru_oublock_word = 0}, {ru_msgsnd = 0, __ru_msgsnd_word = 0}, {ru_msgrcv = 0, __ru_msgrcv_word = 0}, {ru_nsignals = 0, __ru_nsignals_word = 0}, {ru_nvcsw = 42, __ru_nvcsw_word = 42}, {ru_nivcsw = 35, __ru_nivcsw_word = 35}}} starttime = 714145651473980 startreadtime = 0 startwritetime = 0 startwalusage = {wal_records = 2, wal_fpi = 0, wal_bytes = 421} StartPageHit = 50 StartPageMiss = 0 StartPageDirty = 0 errcallback = {previous = 0x0, callback = 0x4f5f41 , arg = 0x25bc510} indnames = 0x266d838 __func__ = "heap_vacuum_rel" This is a qemu VM which (full disclosure) has crashed a few times recently due to OOM. This is probably a postgres bug, but conceivably it's being tickled by bad data (although the vm crashing shouldn't cause that, either, following recovery). This is also an instance that was pg_upgraded from v14 (and earlier versions) to v15b1 and then b2, so it's conceivably possible there's weird data pages that wouldn't be written by beta3. But that doesn't seem to be the issue here anyway. -- Justin