Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nHXng-0001AD-Q7 for pgsql-committers@arkaria.postgresql.org; Tue, 08 Feb 2022 21:13:24 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1nHXnf-0003Om-Mw for pgsql-committers@arkaria.postgresql.org; Tue, 08 Feb 2022 21:13:23 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nHXnf-0003Od-3O for pgsql-committers@lists.postgresql.org; Tue, 08 Feb 2022 21:13:23 +0000 Received: from mail-ej1-x62c.google.com ([2a00:1450:4864:20::62c]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1nHXnb-0004yp-4X for pgsql-committers@lists.postgresql.org; Tue, 08 Feb 2022 21:13:22 +0000 Received: by mail-ej1-x62c.google.com with SMTP id j14so1357243ejy.6 for ; Tue, 08 Feb 2022 13:13:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=enterprisedb.com; s=google; h=message-id:date:mime-version:user-agent:subject:content-language:to :references:from:in-reply-to:content-transfer-encoding; bh=Z7JMneFqf5RyhFoRxtJOhIqvw1elFdLIjsI25lfcG5Q=; b=go6xbkKo0kXMLqAYuM9iSiVZPeIA+UYK5MniIRkqp+DTbKz6pzl8P9j63i4rsK3a+i SCOEPNHY7K/BSmpFi3PzzPXYCoHG5XHU1YrtJSG4SS92UKy52PnHsRskDSQ6wLLad4tC aEZHUrw+xtdNWddycyL4WyhNIvo9zvKOPMwUkyH1qj5B5ykS9Kde+AHRsrO+Y7SogFAf hJK+d6WP6K10CGvbeAm8kwEuc+T/PFjm2Vat6f5h5kv+mkzcirNX+OYh75KCZ05oj7Dm v68f4PDmrGfqf+wUTu+4qK2HXgemzgthOrxHmIdjRzNmLAveQakzKTw4vt76jz+uTuGp s5sQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:subject :content-language:to:references:from:in-reply-to :content-transfer-encoding; bh=Z7JMneFqf5RyhFoRxtJOhIqvw1elFdLIjsI25lfcG5Q=; b=3EpxMIRy5p72XFc1le1tXUxnJXQT0is5ReEXR/ziz3wofgxXI5Y7BEahlVQtAqB4ko qHjmg4Y1Vtdzc/0qld2wP9qsjYd8q+zlL5ScF1JV96zYbuppURrPTk1p02vM+iz1Zlav ar2/4irDMgB+xf5l6pD6HvyGocgQQpc8ugjVCBjy4yua2nY2yqTS/XgovnUE/CTj9gIk T5JA61CgLUwRqMd2Ns8b3COjhYs1YcRIBBcGlQPs5qxTWU+yes4uRoA7CgUegf9Il/7R PXOcZLzijc1mC6Atq/+xIrgdS93yAYJUJ6qiAhBiHmI5/gfcTS0ANsCkcWJZrMxDyYOV sxug== X-Gm-Message-State: AOAM5319THuN//SexVZDFhTXCLrw7SOgkKzd4/14iR2nxV0KbsbOGZ2p Ks8g+6YGJWnNju9XAX5+tX3uzFUJpF7XcZnhBK112CdGgaht0VS4+FfGk1nyFL98kKUIjkeaU3k /rlZgUVqO/T/+Z/uFfPkv3gfiFgo3AkfLMNqx9J4PAw7d2ByWHg/uCziDglsy84cQan3Mg9hFRf XtkPa1CN9k3hgqlpnhLfaJv29KzSjvxULT6pyONmmTl4HCvEroutyn8hxgEBimnhHErsBkX1s= X-Google-Smtp-Source: ABdhPJz9oI9CKd50LvuO0RycWc415SoFB/37MOB6vbiGeYQuE6JiO8FYinUMPtS1ek4MmFzCr2DcAg== X-Received: by 2002:a17:906:8516:: with SMTP id i22mr4936743ejx.73.1644354797992; Tue, 08 Feb 2022 13:13:17 -0800 (PST) Received: from [10.137.0.18] (ip-86-49-251-241.net.upcbroadband.cz. [86.49.251.241]) by smtp.gmail.com with ESMTPSA id v28sm7264157edw.95.2022.02.08.13.13.17 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 08 Feb 2022 13:13:17 -0800 (PST) Message-ID: Date: Tue, 8 Feb 2022 22:13:01 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.5.0 Subject: Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Content-Language: en-US To: Noah Misch , pgsql-committers@lists.postgresql.org References: From: Tomas Vondra In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-CLOUD-SEC-AV-Info: enterprisedb,google_mail,monitor X-CLOUD-SEC-AV-Sent: true X-Gm-Spam: 0 X-Gm-Phishy: 0 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On 10/24/21 03:40, Noah Misch wrote: > Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURRENTLY. > > CIC and REINDEX CONCURRENTLY assume backends see their catalog changes > no later than each backend's next transaction start. That failed to > hold when a backend absorbed a relevant invalidation in the middle of > running RelationBuildDesc() on the CIC index. Queries that use the > resulting index can silently fail to find rows. Fix this for future > index builds by making RelationBuildDesc() loop until it finishes > without accepting a relevant invalidation. It may be necessary to > reindex to recover from past occurrences; REINDEX CONCURRENTLY suffices. > Back-patch to 9.6 (all supported versions). > > Noah Misch and Andrey Borodin, reviewed (in earlier versions) by Andres > Freund. > > Discussion: https://postgr.es/m/20210730022548.GA1940096@gust.leadboat.com > Unfortunately, this seems to have broken CLOBBER_CACHE_ALWAYS builds. Since this commit, initdb never completes due to infinite retrying over and over (on the first RelationBuildDesc call). We have a CLOBBER_CACHE_ALWAYS buildfarm machine "avocet", and that currently looks like this (top): PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2626 buildfa+ 20 0 202888 21416 20084 R 98.34 0.531 151507:16 /home/buildfarm/avocet/buildroot/REL9_6_STABLE/pgsql.build/tmp_install/home/buildfarm/avocet/buildroot/REL9_6_STABLE/inst/bin/postgres --boot -x1 -F Yep, that's 151507 minutes, i.e. 104 days in initdb :-/ I haven't looked at this very closely yet, but it seems the whole problem is we do this at the very beginning: in_progress_list[in_progress_offset].invalidated = false; /* * find the tuple in pg_class corresponding to the given relation id */ pg_class_tuple = ScanPgRelation(targetRelId, true, false); which seems entirely self-defeating, because ScanPgRelation acquires a lock (on pg_class), which accepts invalidations, which invalidates system caches (in clobber_cache_always), which sets promptly sets in_progress_list[in_progress_offset].invalidated = false; guaranteeing an infinite loop. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company