Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1s7lAf-007GoC-Rn for pgsql-hackers@arkaria.postgresql.org; Fri, 17 May 2024 00:10:03 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1s7lAe-001GKk-T2 for pgsql-hackers@arkaria.postgresql.org; Fri, 17 May 2024 00:10:00 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1s7lAe-001GKc-Dc for pgsql-hackers@lists.postgresql.org; Fri, 17 May 2024 00:10:00 +0000 Received: from mail-pg1-x532.google.com ([2607:f8b0:4864:20::532]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1s7lAb-000Zr2-RU for pgsql-hackers@postgresql.org; Fri, 17 May 2024 00:09:59 +0000 Received: by mail-pg1-x532.google.com with SMTP id 41be03b00d2f7-60585faa69fso525122a12.1 for ; Thu, 16 May 2024 17:09:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=j-davis-com.20230601.gappssmtp.com; s=20230601; t=1715904597; x=1716509397; darn=postgresql.org; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:to:from:subject:message-id:from:to:cc:subject:date :message-id:reply-to; bh=XfUi0QbIZdoHI6ABDtCwqZBlK4Pu5i45glE5hy4s918=; b=N+fWipRjSAMoabm/2GQy/21yKcrtJkXm9cbi1ulsucZCOKxd2GTySwi0hqjrjQ632b 8x2soBf97BvLhlZlTeX/iAa2dXRmhvtKHxSHTLGAYIJJsbOkeqL+6MxWozIkc/299tDP BVKTgIG9uwKdtxFkWRUV9cKHDV6+wzheddo8z7+VsaJ3lGYnvqhSiqGxMrFapX7r0IST elId8CnwjiasBeBoXbmxxTlEtNPNWDJtQ1zHatJav/7y32sRSMiVKLEl7iDRdcbo4LNp aRjJNowrs5PM7szofl1sMzfroQjvDLF8U6JWdwY43h/SfdfWDbqQq35PqiLmLEbUW27Q mIUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715904597; x=1716509397; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:to:from:subject:message-id:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=XfUi0QbIZdoHI6ABDtCwqZBlK4Pu5i45glE5hy4s918=; b=N79OPPjF/TaZZbPx6KOFgipFSOOZgvxFukDyN6ZFH+3dbpUE3P/ey7q6uiaewOTGCO U0sU1ekcd5GcbNoWJVLYrXrae/FKJu2r75K+nhXJ6wS7O0c2vEycdY7TqgaxdcpHD0WE ZYABGPxXlDS9uDAjf3lj1/ZdY339KGVOkEqioFgSo3XgozervwUMKj3bS8+hQiHGww5H SBJC8br7O4EBjhiBDY3u2CbxNtJ5O+PYOw6Kba7ZiybmpC2PtMMwiweIrPuWdI01fSsV yNLvUWSsKE2PjOnMWuGR5htm2bvRU7IuQQRVwSzuIyKt+UVSLmKce3wy+vH9QeHWaT/c dzYg== X-Forwarded-Encrypted: i=1; AJvYcCVKKsaUZxn3hfVRlPSxKTCoxlpWNC6xP7AXHbYBLMS3Ksh1noNM94RlpnNcaQqhYY5uDtORbMomQA++g4Dn8b/Dz7TwgF3QGCS3oXF4 X-Gm-Message-State: AOJu0YwTdNLLWQ7qRnB+0HKdUAm9aGgkPTep/AZJKQZKCvOuwCdJ36OJ fCHxQ7pDRa7COeroQV4OPZWCgE4mviioHFqa/v5uW57OgBgyCB3HxO3UdrKRpQ== X-Google-Smtp-Source: AGHT+IH+AyMBXykqYvyHgHNLRrKWoD+oA0BZZCVD2L1jO7IiOWTL9x84fCEZSYnyNOKl2HDN8tzn+w== X-Received: by 2002:a17:90a:6982:b0:2ab:8324:1b47 with SMTP id 98e67ed59e1d1-2b6c75dc6ddmr29870430a91.15.1715904596704; Thu, 16 May 2024 17:09:56 -0700 (PDT) Received: from [172.18.10.46] ([12.126.244.130]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2b973234901sm6259034a91.19.2024.05.16.17.09.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 May 2024 17:09:56 -0700 (PDT) Message-ID: <3b97ab8e295ec3db22baa2dfdd49073449ae1696.camel@j-davis.com> Subject: Re: optimizing pg_upgrade's once-in-each-database steps From: Jeff Davis To: Nathan Bossart , pgsql-hackers@postgresql.org Date: Thu, 16 May 2024 17:09:55 -0700 In-Reply-To: <20240516211638.GA1688936@nathanxps13> References: <20240516211638.GA1688936@nathanxps13> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.44.4-0ubuntu2 MIME-Version: 1.0 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On Thu, 2024-05-16 at 16:16 -0500, Nathan Bossart wrote: > I am posting this early to get thoughts on the general approach.=C2=A0 If > we > proceeded with this strategy, I'd probably create some generic > tooling that > each relevant step would provide a set of callback functions. The documentation states: "pg_dump -j uses multiple database connections; it connects to the database once with the leader process and once again for each worker job." That might need to be adjusted. How much complexity do you avoid by using async instead of multiple processes? Also, did you consider connecting once to each database and running many queries? Most of those seem like just checks. Regards, Jeff Davis