Date: Wed, 6 Jul 2022 09:51:10 -0700
From: Nathan Bossart <nathandbossart@gmail.com>
To: Andres Freund <andres@anarazel.de>
Cc: Simon Riggs <simon.riggs@enterprisedb.com>,
	Robert Haas <robertmhaas@gmail.com>,
	"Bossart, Nathan" <bossartn@amazon.com>,
	Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>,
	Maxim Orlov <orlovmg@gmail.com>, Amul Sul <sulamul@gmail.com>,
	Bruce Momjian <bruce@momjian.us>,
	"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: O(n) tasks cause lengthy startups and checkpoints
Message-ID: <20220706165110.GB2135371@nathanxps13>
References: <20220218164454.GA3403811@nathanxps13>
 <20220218205111.GA3447220@nathanxps13>
 <CANbhV-H9=DOBdpKE77XB_hiPDKVre-+1vuLvKWTke07Y3K1P5w@mail.gmail.com>
 <CA+TgmoYkDY7r8PwAGhD3tMqVAeKBfY-_7miJKcj-ZzPWN0UYzQ@mail.gmail.com>
 <20220623171552.GA96638@nathanxps13>
 <CANbhV-GjpUi43Kyts+ib+j2=X_+1_AVh+CSdj-ow=M5aZqvfuQ@mail.gmail.com>
 <20220702220554.GA1044460@nathanxps13>
 <20220702225456.zit5kjdtdfqmjujt@alap3.anarazel.de>
 <20220703170754.GA1048244@nathanxps13>
 <20220703172732.wembjsb55xl63vuw@awork3.anarazel.de>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="/04w6evG8XlLl3ft"
Content-Disposition: inline
In-Reply-To: <20220703172732.wembjsb55xl63vuw@awork3.anarazel.de>
Archived-At: 
 <https://www.postgresql.org/message-id/20220706165110.GB2135371%40nathanxps13>
Precedence: bulk


--/04w6evG8XlLl3ft
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

Here's a new revision where I've attempted to address all the feedback I've
received thus far.  Notably, the custodian now uses a queue for registering
tasks and determining which tasks to execute.  Other changes include
splitting the temporary file functions apart to avoid consecutive boolean
flags, using a timestamp instead of an integer for the staging name for
temporary directories, moving temporary directories to a dedicated
directory so that the custodian doesn't need to scan relation files,
ERROR-ing when something goes wrong when cleaning up temporary files,
executing requested tasks immediately in single-user mode, and more.

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

--/04w6evG8XlLl3ft
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
	filename="v7-0001-Introduce-custodian.patch"