Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1onTo4-0005mq-Nn for pgsql-committers@arkaria.postgresql.org; Tue, 25 Oct 2022 23:58:04 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1onTo2-00039z-Kx for pgsql-committers@arkaria.postgresql.org; Tue, 25 Oct 2022 23:58:02 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1onTo2-00039p-B3 for pgsql-committers@lists.postgresql.org; Tue, 25 Oct 2022 23:58:02 +0000 Received: from mail-il1-x12e.google.com ([2607:f8b0:4864:20::12e]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1onTnx-0006qX-RL for pgsql-committers@lists.postgresql.org; Tue, 25 Oct 2022 23:58:00 +0000 Received: by mail-il1-x12e.google.com with SMTP id s9so3901904ilu.1 for ; Tue, 25 Oct 2022 16:57:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telsasoft-com.20210112.gappssmtp.com; s=20210112; h=user-agent:in-reply-to:content-disposition:mime-version:references :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=z8AMlds/C7A8rG+E3X3BfdG+NkLytu0JeybzCwjPm7E=; b=Fg36SAAPSlPPGgWnbmGElQQ/lDSfdZnRJIOeSO1b9Ul05JhwZLdXkZRHjgCvSYDihj tdshyKVIFvG/mczq1zDkszHCN+Vlfojdb8jnJOd6lmueMT32+2EhEt47RMfIH4GFi744 Qm4GYdpBN8nJ3nTzZ3Bz3Y3cHJPRKgvumVvoFSwgYhOt6dIQfvRm2OYwzwTUkaFKPXiv Xs1u8cDvEVlfKQQPVp80lh7gZLKsMDbssa/EYMhFF5WVdYtFXbBlytp825XzcFqec+gp Wh5cPtDl+IE9E9gSu9rFIHoBmB03c0RmSjEOQ+/Gg1nUWdFfNA23U6EA6WLDU59KjSfV WLIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=user-agent:in-reply-to:content-disposition:mime-version:references :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=z8AMlds/C7A8rG+E3X3BfdG+NkLytu0JeybzCwjPm7E=; b=UooRt0hoX8wQ+j/y2iw7vpcLxKtuZnmi1gjh4imo0S/0fE2OQZoNmCFmj9w1DxTG0x KAX29CmjJioEeh9sBbf2kQmqBSenkbOwD8AXQtr0ftLU0dYn8W2cYXL+Gl1cpcJo9Th4 T2/On8sMHQsKRnoht9mAcG7eVga84caK9Ixbg+MZjhSxyqJmKQHrU9ONfgERTCxx0KbW 00zduPp6mAvzMG0zKhPxwGTAAdcvQRjj26s9uNJfOsVNcaVr3kfuDOKQfchhrGtZb54g 1kupIh6nBK1X67pKrdNk+hDeFL0cRWH+FBu0ONbMogO2449Fe3tyEsapUk8pMzpKLY8m HQCA== X-Gm-Message-State: ACrzQf2fP1BMG2TgGpJu5ueTaTBXqMWFln9lt8bBo9HZm3zgjz5HB239 3AbUHuui/4iMHUCyNuCzrBEUE6Ce96l0YA== X-Google-Smtp-Source: AMsMyM5LRx3w5g8CnOYJVtUQTIbu1aPgYX/KXNEzJMNGMwZ8scvyXSd9zj/VyOv62i4LkctsjoBwNw== X-Received: by 2002:a92:8743:0:b0:2fd:3102:e3ed with SMTP id d3-20020a928743000000b002fd3102e3edmr24626388ilm.115.1666742275712; Tue, 25 Oct 2022 16:57:55 -0700 (PDT) Received: from pryzbyj.telsasoft (charmander.telsasoft.com. [50.244.222.1]) by smtp.gmail.com with ESMTPSA id o13-20020a056e02102d00b002ff63c88824sm1483125ilj.75.2022.10.25.16.57.54 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 25 Oct 2022 16:57:55 -0700 (PDT) Received: by pryzbyj.telsasoft (Postfix, from userid 1000) id 2DCBA8007FB; Tue, 25 Oct 2022 18:57:54 -0500 (CDT) Date: Tue, 25 Oct 2022 18:57:54 -0500 From: Justin Pryzby To: Thomas Munro Cc: Thomas Munro , pgsql-committers@lists.postgresql.org Subject: Re: pgsql: Fix unlink() for STATUS_DELETE_PENDING on Windows. Message-ID: <20221025235753.GT16921@telsasoft.com> References: <20221025212102.GS16921@telsasoft.com> <20221025213055.GA8537@telsasoft.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.4 (2018-02-28) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On Wed, Oct 26, 2022 at 11:15:16AM +1300, Thomas Munro wrote: > On Wed, Oct 26, 2022 at 10:31 AM Justin Pryzby wrote: > > On Tue, Oct 25, 2022 at 04:21:02PM -0500, Justin Pryzby wrote: > > > > Restore that behavior with an explicit check, to see if it fixes the > > > > occasional 'directory not empty' failures seen in the pg_upgrade tests > > > > on CI. Further improvements are possible with proposed upgrades to > > > > modern Windows APIs that would replace this convoluted code. > > > > If I'm not wrong, this didn't fix the issue you said it fixed. > > > > s/said/hoped/sorry > > Drat. More theories needed then. Perhaps it has nothing to do with > my recent changes. I am starting to wonder if we should have an > rmdir() wrapper that waits for zombie files to go away, and spits out > the name of the file that's in the way, and also the name/pid of the > process that has it open[1] if it times out, so we have a fighting > chance of debugging this type of stuff. > > [1] https://devblogs.microsoft.com/oldnewthing/20120217-00/?p=8283 FWIW, your description of the problem sounded a bit off to me. You seemed to say that the problem is with rmdir() on a directory, which contains a file which was "recently removed", but still opened by something. But as I recall, at the point that pg_upgrade is running rmdir(), the contained files have been unlinked, and the postgres process has ended: | # Running: pg_upgrade --no-sync -d C:\cirrus\build/testrun/pg_upgrade/002_pg_upgrade\data/t_002_pg_upgrade_old_node_data/pgdata -D C:\cirrus\build/testrun/pg_upgrade/002_pg_upgrade\data/t_002_pg_upgrade_new_node_data/pgdata -b C:/cirrus/build/tmp_install/bin -B C:/cirrus/build/tmp_install/bin -s C:/Users/ContainerAdministrator/AppData/Local/Temp/f67bWmckRH -p 57611 -P 57612 --check | Performing Consistency Checks | ----------------------------- | Checking cluster versions ok | Checking database user is the install user ok | Checking database connection settings ok | Checking for prepared transactions ok | Checking for system-defined composite types in user tables ok | Checking for reg* data types in user tables ok | Checking for contrib/isn with bigint-passing mismatch ok | Checking for presence of required libraries ok | Checking database user is the install user ok | Checking for prepared transactions ok | Checking for new cluster tablespace directories ok | | *Clusters are compatible* | pg_upgrade: warning: could not remove file or directory "C:/cirrus/build/testrun/pg_upgrade/002_pg_upgrade/data/t_002_pg_upgrade_new_node_data/pgdata/pg_upgrade_output.d/20220925T193346.158/log": Directory not empty | pg_upgrade: warning: could not remove file or directory "C:/cirrus/build/testrun/pg_upgrade/002_pg_upgrade/data/t_002_pg_upgrade_new_node_data/pgdata/pg_upgrade_output.d/20220925T193346.158": Directory not empty My thinking has always been that this is essentially a bug/deficiency in the windows FS. It seems absurd that unlink/rmdir would be asynchronous. It'd be swell if we could use a separate device in CI, to be used for running tests. Bonus points if it supports COW. -- Justin