Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nVtf1-0001kC-2f for pgsql-hackers@arkaria.postgresql.org; Sun, 20 Mar 2022 11:23:47 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1nVtey-00047I-Gz for pgsql-hackers@arkaria.postgresql.org; Sun, 20 Mar 2022 11:23:44 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nVtex-000478-Tk for pgsql-hackers@lists.postgresql.org; Sun, 20 Mar 2022 11:23:44 +0000 Received: from mail-lj1-x233.google.com ([2a00:1450:4864:20::233]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1nVteu-00031v-GG for pgsql-hackers@postgresql.org; Sun, 20 Mar 2022 11:23:43 +0000 Received: by mail-lj1-x233.google.com with SMTP id h11so16623531ljb.2 for ; Sun, 20 Mar 2022 04:23:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=enterprisedb.com; s=google; h=message-id:date:mime-version:user-agent:subject:content-language:to :cc:references:from:in-reply-to:content-transfer-encoding; bh=84GSbsUdCUFCooks9N1oiBUVy9+tc6RML+kjIdoQvxY=; b=hobA/Nsp7gFvrXzveeiU+dEh+3qY7dAXFtfJLvHtFvMDACH6bSX5Sb2rX17fWBQyjY T6ujq1Dm6CY40SEoeFJPvPv10xOAa2uHGttQ4R0hKLjp1P7Ll/YDuA0XqVpwvtq41IJ7 VRtqSYxeCUtneMjoZ+o2weQKP/9BgXpCn7m7rpMIoQwSy4jNeZIKQ+q4fhfF+BgWHvz5 cXjn6pWqf4zU1vqSU5DApmJDF2atWnmWLCf8NxXT/yMT6xqK8eUkeFy+FBrlRK09rEvh rGzBMGMLjFLezlBjNsw51/Oyunxy2AqExdONmYqrZT458ep6fv1/0Pgwa47+ERK60Gu4 f+1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:subject :content-language:to:cc:references:from:in-reply-to :content-transfer-encoding; bh=84GSbsUdCUFCooks9N1oiBUVy9+tc6RML+kjIdoQvxY=; b=2E9IRMP962Ynu8OxBTBeFNv6ow1gSx0zGuX4Ja5NyFKiOIiOvy91B1Gz05ROMyq+1M Ya8zi2S2+vRh3oRDwHOju2UjOVe7yROAG+7K8U7Nw6ue2y4m9pyphhcgDcTjnmA8Q/KS PskwphUbbKlqGr/+N4N43cpgKy5r0IDjkcOpR72wTHrqVjL2/0RAj0xswua5/npsc7uN C3rKIiFHUL0UqRTuuh3vXvcNcDcYOjFoIE2ZPDBZEozUJuC15nHWroV56jya/+eEOyBI OTkvpFI5O8Erz16mM6cknPp9LbruuqHp4B42E9DQ2vO1smLeKWQ7+L10wi3LISHJR2QQ ln2Q== X-Gm-Message-State: AOAM530vJ5wHPBpus060te/oyY2XGUmg8jQkMBl4HM1keF6cM0ODf2vS S1nvJBbm8r/wuMLZub45no0YBcpfZGdifrrh09UuRned+DhzB2rLrWZzETdvoX1AqmhkdiA+EMA INISyK39R3FDaG8/2YgRFM9nW2c9OgRLzMZpj2VpUCyLEPr9MFq79NWWxDCEncQY5eF1ddjm2rp cYmQwWfS26+45ExD95J5J5hUmRNTQwjAP2Jaqb X-Google-Smtp-Source: ABdhPJwt2A49oyrM+drE9TxQx0Uyv52s93wVt622+tyWV1PiS1pV852VZTHzOu+IHiLWgmA/hSCBtg== X-Received: by 2002:a2e:80c2:0:b0:249:808b:590c with SMTP id r2-20020a2e80c2000000b00249808b590cmr2278599ljg.496.1647775419221; Sun, 20 Mar 2022 04:23:39 -0700 (PDT) Received: from [10.137.0.17] (37-247-27-82.customers.ownit.se. [37.247.27.82]) by smtp.gmail.com with ESMTPSA id n13-20020a056512388d00b0044331f8f1f8sm1535397lft.128.2022.03.20.04.23.38 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 20 Mar 2022 04:23:38 -0700 (PDT) Message-ID: <840aa7d7-52bf-d3da-00d6-468426e4634c@enterprisedb.com> Date: Sun, 20 Mar 2022 12:23:39 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.6.2 Subject: Re: Column Filtering in Logical Replication Content-Language: en-US To: Amit Kapila Cc: "houzj.fnst@fujitsu.com" , Peter Eisentraut , Alvaro Herrera , Justin Pryzby , Rahila Syed , Peter Smith , pgsql-hackers , "shiy.fnst@fujitsu.com" References: <202201120041.p24wvsfcsope@alvherre.pgsql> <822a8e40-287c-59ff-0ea9-35eb759f4fe6@enterprisedb.com> <2b8471f9-3849-c91f-ef82-9181d4bca0a7@enterprisedb.com> <16e83ce3-8415-b732-37b2-c3329a2c13a1@enterprisedb.com> <3162f7f0-86ac-e3a9-dc55-0a660cb9bab1@enterprisedb.com> <369ae611-8822-f499-87cd-58ad0d60c60c@enterprisedb.com> From: Tomas Vondra In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-CLOUD-SEC-AV-Info: enterprisedb,google_mail,monitor X-CLOUD-SEC-AV-Sent: true X-Gm-Spam: 0 X-Gm-Phishy: 0 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On 3/20/22 07:23, Amit Kapila wrote: > On Sun, Mar 20, 2022 at 8:41 AM Amit Kapila wrote: >> >> On Fri, Mar 18, 2022 at 10:42 PM Tomas Vondra >> wrote: >> >>> So the question is why those two sync workers never complete - I guess >>> there's some sort of lock wait (deadlock?) or infinite loop. >>> >> >> It would be a bit tricky to reproduce this even if the above theory is >> correct but I'll try it today or tomorrow. >> > > I am able to reproduce it with the help of a debugger. Firstly, I have > added the LOG message and some While (true) loops to debug sync and > apply workers. Test setup > > Node-1: > create table t1(c1); > create table t2(c1); > insert into t1 values(1); > create publication pub1 for table t1; > create publication pu2; > > Node-2: > change max_sync_workers_per_subscription to 1 in potgresql.conf > create table t1(c1); > create table t2(c1); > create subscription sub1 connection 'dbname = postgres' publication pub1; > > Till this point, just allow debuggers in both workers just continue. > > Node-1: > alter publication pub1 add table t2; > insert into t1 values(2); > > Here, we have to debug the apply worker such that when it tries to > apply the insert, stop the debugger in function apply_handle_insert() > after doing begin_replication_step(). > > Node-2: > alter subscription sub1 set pub1, pub2; > > Now, continue the debugger of apply worker, it should first start the > sync worker and then exit because of parameter change. All of these > debugging steps are to just ensure the point that it should first > start the sync worker and then exit. After this point, table sync > worker never finishes and log is filled with messages: "reached > max_sync_workers_per_subscription limit" (a newly added message by me > in the attached debug patch). > > Now, it is not completely clear to me how exactly '013_partition.pl' > leads to this situation but there is a possibility based on the LOGs > it shows. > Thanks, I'll take a look later. From the description it seems this is an issue that existed before any of the patches, right? It might be more likely to hit due to some test changes, but the root cause is older. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company