Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nHmsj-0005Da-KP for pgsql-hackers@arkaria.postgresql.org; Wed, 09 Feb 2022 13:19:37 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1nHmsf-0003n3-NY for pgsql-hackers@arkaria.postgresql.org; Wed, 09 Feb 2022 13:19:33 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nHbjR-0004rG-19 for pgsql-hackers@lists.postgresql.org; Wed, 09 Feb 2022 01:25:17 +0000 Received: from mail-ej1-x62c.google.com ([2a00:1450:4864:20::62c]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1nHbjN-0002aZ-Nr for pgsql-hackers@postgresql.org; Wed, 09 Feb 2022 01:25:15 +0000 Received: by mail-ej1-x62c.google.com with SMTP id p15so2719680ejc.7 for ; Tue, 08 Feb 2022 17:25:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=enterprisedb.com; s=google; h=message-id:date:mime-version:user-agent:subject:content-language:to :cc:references:from:in-reply-to:content-transfer-encoding; bh=fpjbvHpOJl1I56yg9XU1bmuNvTvdZisOKxt5fHKa1BA=; b=WzwVW9m2S/rdT9mFxg6B2b4+avHeWJKXBm+gtlVMgDIW72pTwtC+qTm5ctZZn2kTui NXZdZu5Ymwk65U8P6ZUowXpjN6cB3zOlAZiye1f3HOwG6Bakb+93It8QAA/Q9w1pgn2d QEeLg3OiruXi2YLVaYioc9OdSTO4PXC7gWTnpX4dGQgH8weHFinLwUMxavbb/0So44tR ax/GN7i216/svsXR1SS3ZCdIvVlBp1+mczLrnn9VdV+mTJFxdUf5H7Ky1mKFYQYYnmQQ KSC7alDc0cu4TUDxIDrpRbpnggcR7Kx4+05Y8O4ZwqsAq8pGEkQX4+0Ru9sf5Jn2lptf eZtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:subject :content-language:to:cc:references:from:in-reply-to :content-transfer-encoding; bh=fpjbvHpOJl1I56yg9XU1bmuNvTvdZisOKxt5fHKa1BA=; b=NqPf5HUbyDdy5S0lzBxTZAMQUO5Vz6zCMIePx35/poRnNSQDICYSGAIN/jkl1tQsyf gOJDXa3/q97S11E1NIj9yYrgyHlUs+3QhEKMhlxC2XwspKYMZhZdF3OWf1iOjnIVUfDX cKc57cQdt3Y5bXv7pHVKcqG36IwXtj+gsktuT6lm5Um8WOvtzdia3wER8PSjbQGembbt PmzyprxEVJMm37DxDmulvddG0CYz3bOcdGDubdaTVSpoxm6dxa+Fr2oq4n7FzwxqaH1Q wE4anEFwY1+EfNQxQ+OSwlbdEvgPo3oBlUOOt9vx1cCIWbWbT+IxCcpDD1WJ8wmROKLa oRSQ== X-Gm-Message-State: AOAM533U+XD0AtmgdDgxPAE1ZsS0yEnLTmRlFgWUNosc7Z7qnx4wSneK CmicwehzRQ591El1SUzaBphOGdg3aP51koAb2DMsxh3/TQ2/C7oXBTMtZl7KgJzerRN+tH2elD0 RK2DyocUlap/Q8zDVit6MPYF8LqPadh2pnoESEkHtUN7RdMFVehO9eNUdCYzCJCFLTs1mKq1+XR H3hDthbSXDWVPHTHeuWEj7OA5JiYQ/V5QlvWlx X-Google-Smtp-Source: ABdhPJxld9A76Krs2xcpZAjZ7zQarNHauoEfGwNu4b+DBpU52HgdAOi5LN+hX5qCbub8zQZLL5GM8g== X-Received: by 2002:a17:907:3e24:: with SMTP id hp36mr3937944ejc.530.1644369911637; Tue, 08 Feb 2022 17:25:11 -0800 (PST) Received: from [10.137.0.18] (ip-86-49-251-241.net.upcbroadband.cz. [86.49.251.241]) by smtp.gmail.com with ESMTPSA id n2sm5035030ejl.86.2022.02.08.17.25.10 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 08 Feb 2022 17:25:11 -0800 (PST) Message-ID: <40d0a4c8-de56-7710-5197-1304fa156aba@enterprisedb.com> Date: Wed, 9 Feb 2022 02:25:09 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.5.0 Subject: Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Content-Language: en-US To: Andres Freund , pgsql-hackers@postgresql.org, pgsql-release@postgresql.org Cc: Noah Misch , pgsql-committers@lists.postgresql.org References: <20220209004347.mnwaoiv5x7dntuvi@alap3.anarazel.de> From: Tomas Vondra In-Reply-To: <20220209004347.mnwaoiv5x7dntuvi@alap3.anarazel.de> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-CLOUD-SEC-AV-Info: enterprisedb,google_mail,monitor X-CLOUD-SEC-AV-Sent: true X-Gm-Spam: 0 X-Gm-Phishy: 0 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On 2/9/22 01:43, Andres Freund wrote: > Hi, > > On 2022-02-08 22:13:01 +0100, Tomas Vondra wrote: >> On 10/24/21 03:40, Noah Misch wrote: >>> Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURRENTLY. >>> >>> CIC and REINDEX CONCURRENTLY assume backends see their catalog changes >>> no later than each backend's next transaction start. That failed to >>> hold when a backend absorbed a relevant invalidation in the middle of >>> running RelationBuildDesc() on the CIC index. Queries that use the >>> resulting index can silently fail to find rows. Fix this for future >>> index builds by making RelationBuildDesc() loop until it finishes >>> without accepting a relevant invalidation. It may be necessary to >>> reindex to recover from past occurrences; REINDEX CONCURRENTLY suffices. >>> Back-patch to 9.6 (all supported versions). >>> >>> Noah Misch and Andrey Borodin, reviewed (in earlier versions) by Andres >>> Freund. >>> >>> Discussion: https://postgr.es/m/20210730022548.GA1940096@gust.leadboat.com >>> >> >> Unfortunately, this seems to have broken CLOBBER_CACHE_ALWAYS builds. Since >> this commit, initdb never completes due to infinite retrying over and over >> (on the first RelationBuildDesc call). > > Ugh. Do we need to do something about WRT the next set of minor releases? Is > there a a chance of this occuring in "real" workloads? > AFAICS this only affects builds with CLOBBER_CACHE_ALWAYS, and anyone running such build in production clearly likes painful things anyway. But really, for the infinite loop to happen, building a relation descriptor has to invalidate a cache. And I haven't found a way to do that without the CLOBBER_CACHE_ALWAYS thing. Also, all the November minor releases include this commit, and there were no reports about this (pretty obvious) issue. Buildfarm did not complain either (but an animal may be stuck for months and we would not know about it). regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company