Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1sTNlo-00Ffwf-Oh for pgsql-general@arkaria.postgresql.org; Mon, 15 Jul 2024 15:37:44 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1sTNlm-00B9MR-Pp for pgsql-general@arkaria.postgresql.org; Mon, 15 Jul 2024 15:37:42 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1sTNlm-00B9MH-F3 for pgsql-general@lists.postgresql.org; Mon, 15 Jul 2024 15:37:42 +0000 Received: from fout2-smtp.messagingengine.com ([103.168.172.145]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1sTNlf-002BdU-Q7 for pgsql-general@lists.postgresql.org; Mon, 15 Jul 2024 15:37:41 +0000 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailfout.nyi.internal (Postfix) with ESMTP id C49311388A88; Mon, 15 Jul 2024 11:37:34 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Mon, 15 Jul 2024 11:37:34 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=aklaver.com; h= cc:content-transfer-encoding:content-type:content-type:date:date :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm1; t=1721057854; x=1721144254; bh=5LMexG390y44CqqhhH7xu8QJl3FCl/pbRFM1Jnsb9DE=; b= h/x1sY04NbPj4pn/jJFEc5wTKNyp/YUkN6lLEHSk8E9swsfBRV3ECywEFc7nlf6U zNh7+iZS/H3j4I+WfYHe7H7hNH77RBJmenJKGBHfAZ5Pyb7uU/oVLcvyCG6iFUD8 ABkPXA2bZom1X4je2Dx5aTKcfrbzAk5YRJpCjNh6yLl6Cdp2jOaMgYfKIR2fQSHS LThWww9WJB8Ny+DSObDJrffoVXetI2HjulFr/Z/yu66FjUEQZZy7RcH2jDizwdxf DC53MwTVwUATnaCwGHzVIoWwBfMJ9UKCqORHNbpSpH5b3tHDePHIpxIeEWja/PZC 6JZQK0JJ+GbD/tNvLz6lEw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t=1721057854; x= 1721144254; bh=5LMexG390y44CqqhhH7xu8QJl3FCl/pbRFM1Jnsb9DE=; b=l uXVwGEhyFiMGOZSGF51di31Lvn1S8wEfflwJ9A8USu9q6aUQXuIg7Jr6UgrF2OmP WQ9g4Xw19P9IrfZ2x7BDghRQ3GlPjyhXeM0XRQ8QkbntaQb3faM+HmxhXT9WQc35 Mhpw23oCVBy9nqTzUXD1iPG+k55BFK79WhOE2mXcvqm/KhT+ZX+ck1zeK7wW3EhG AwjEyJ3OSyx46rS7+uB4J+3AZNlGYjflf9YkwBmJtUsXzUQu4s+jH35zDho1BNS8 nRQK4Q4BICiYtqDZ+OlQjAjN8dHpsMsnmnAJUSlOavrMgFDQ7NetKZfV1UcobjQd 9oveNFF2MWrdGJcXdEcTg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrgedvgdeludcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefkffggfgfuvfhfhfgjtgfgsehtkeertddtvdejnecuhfhrohhmpeetughrihgr nhcumfhlrghvvghruceorggurhhirghnrdhklhgrvhgvrhesrghklhgrvhgvrhdrtghomh eqnecuggftrfgrthhtvghrnhepffelgeeifefgveduhedthfekuedtffejveegffegjeev tdehgfduieetfeehjeehnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrg hilhhfrhhomheprggurhhirghnrdhklhgrvhgvrhesrghklhgrvhgvrhdrtghomh X-ME-Proxy: Feedback-ID: i76984098:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 15 Jul 2024 11:37:32 -0400 (EDT) Message-ID: <28c88e41-36bc-4704-9c30-f81986f6cdc8@aklaver.com> Date: Mon, 15 Jul 2024 08:37:32 -0700 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: How does this FK constraint error happen? To: Ron Johnson , "pgsql-generallists.postgresql.org" References: <20240715143518.3v5xdfuj27ryzuzh@hjp.at> <4ff0dd59-9d7b-4a13-ad75-3627080458e1@aklaver.com> Content-Language: en-US From: Adrian Klaver In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On 7/15/24 08:18, Ron Johnson wrote: > On Mon, Jul 15, 2024 at 11:06 AM Adrian Klaver > > wrote: > > On 7/15/24 07:53, Ron Johnson wrote: > > On Mon, Jul 15, 2024 at 10:35 AM Peter J. Holzer > > TAPd=# select * from rel_group_user > > where user_id between 1100 and 1300 > > order by user_id; > >   user_id | group_id | modified_by |       modified_on > > ---------+----------+-------------+------------------------- > >      1133 |        2 |        1133 | 2024-07-15 08:43:35.669 > >      1142 |        2 |        1142 | 2024-07-15 09:05:58.451 > >      1147 |        2 |        1147 | 2024-07-15 09:30:37.169 > >      1158 |        2 |        1158 | 2024-07-15 09:36:45.142 > >      1197 |        2 |        1197 | 2024-07-15 09:52:58.477 > >      1210 |        2 |        1210 | 2024-07-15 02:42:09.355 > <<<<<<<<<<<<< > > Time travel? > > > 😞 > > > 2024-07-15 02:41:15 Deleting from > FISPTAPPGS401DA/TAPd.public.access_user > DELETE FROM public.access_user; > > Or do the cron jobs take that long to execute? > > > The deletes from 26*3 tables (the same 26 tables in three children) took > from 02:40:02 to 02:41:47. > Then a bunch of COPY statements run (pg_dump from the federation master, > then COPY to the federation children).  Must be done in a specific order. I don't think it is entirely coincidental that 1210 is the only shown user_id with a modified_on value that is in proximity to the delete error. My suspicion is that actions are not happening in the exact order you think they are. I would think that combining DELETE FROM rel_group_user; and DELETE FROM public.access_user; in a single transaction would be a good start to fixing this. > > How is modified_on created? > > > It's updated by the application. At what point in the process? > > >      1229 |        2 |        1229 | 2024-07-15 08:33:48.443 > >      1242 |        2 |        1242 | 2024-07-15 10:29:51.176 > >      1260 |        2 |        1260 | 2024-07-15 07:36:21.182 > >      1283 |        2 |        1283 | 2024-07-15 09:48:25.214 > >      1288 |        2 |        1288 | 2024-07-15 08:10:33.609 > > (11 rows) > > > > TAPd=# select user_id, login_id, created_on, modified_on > > TAPd-# from public.access_user > > TAPd-# where user_id = 1210; > >   user_id |  login_id  |       created_on        |       modified_on > > > ---------+------------+-------------------------+------------------------- > >      1210 | JORIEUSER3 | 2023-10-20 11:54:24.562 | 2024-07-15 > 02:42:09.355 > > (1 row) > > -- > Adrian Klaver > adrian.klaver@aklaver.com > -- Adrian Klaver adrian.klaver@aklaver.com