Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mTmMe-0001X7-CW for pgsql-performance@arkaria.postgresql.org; Fri, 24 Sep 2021 14:39:48 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1mTmMd-00086A-0J for pgsql-performance@arkaria.postgresql.org; Fri, 24 Sep 2021 14:39:47 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mTmMc-000862-OC for pgsql-performance@lists.postgresql.org; Fri, 24 Sep 2021 14:39:46 +0000 Received: from mail-io1-xd34.google.com ([2607:f8b0:4864:20::d34]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1mTmMV-0001k1-D1 for pgsql-performance@lists.postgresql.org; Fri, 24 Sep 2021 14:39:46 +0000 Received: by mail-io1-xd34.google.com with SMTP id r75so12920116iod.7 for ; Fri, 24 Sep 2021 07:39:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telsasoft-com.20210112.gappssmtp.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=HYor7mmHJTDbuicnPC6JRMXI9CWqAe1oBF/QjXPuTsY=; b=01CzXCBw1xE5QRoFr6/50gHZ3WguF0nWragXOHEFgLSDjQ8X6+vaDylgQ8HR+ijwce njQDJmlthMJvA439ffTGJzD+v7w+Cm9o0HgKJt+RqPx2H2vXTAPwdLRy98MQYgs5cqES Yu7ux3mxo01bnxvAClBKE67uc0/3ZXiuOf8uCYHI4pWMx5TZGbjguiY0fgXPlJyY41Cw Atiy49iPyRFzUO6SVkyGyHpEO7FVb1xIvRJ7GIFgNIAS+WrDf6rK/Sn68NC15Uq+8ZFe XkyQdyyArMrv/GFWnuIRuO5m+XVbn+V2fIW2ePcXjLxextU3ShUXoRVRAvBF91+x+ajL hIMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=HYor7mmHJTDbuicnPC6JRMXI9CWqAe1oBF/QjXPuTsY=; b=Y9nR9BEDNcwCgJftbGS2FzBKTzL5yZmQknk3sOCVWV8eHeIm6zhCVOUsiIgnr8pUeU deoncC2wDeaKWDK8CbJD2xVcCsoei5zzGoTsnAtPBbgZy8A5zqU8qU0DzonC5sidQVll bZMRL+OVzcechTP/r99di9b7BKWEVYtJ+8OLPOzwUnP53MyXeE5I8SF1uqv9CvP32m6X zEocdIZHHj/fblK9mgLULFfK4nxv7X2eRP3gO9UKyUwxoVnh9ECZLqK21QlN/e2SGpbA zHvOj7dEY07xNoXLVV31zkZ3027LzBsHkfmytu7Xb26foTG7IpU4PbOz88P4FLKUrXKu gUww== X-Gm-Message-State: AOAM5334Lpx8VogAavuzGM+LeeCPNC1o3YY57kjN5iEKXmLwLL+XcCbT w7PSRs1DAWO849073Pkjq/WEyPekLnaw9g== X-Google-Smtp-Source: ABdhPJwDdzoCuPuNATJkbPKFzXK91qllw9hJPw1wTAzZHLyqPyqnXG4eiAHJq+4j6MmfZX23tLOlKg== X-Received: by 2002:a05:6638:1929:: with SMTP id p41mr9614750jal.21.1632494377437; Fri, 24 Sep 2021 07:39:37 -0700 (PDT) Received: from pryzbyj.telsasoft (charmander.telsasoft.com. [50.244.222.1]) by smtp.gmail.com with ESMTPSA id p73sm4011127iod.43.2021.09.24.07.39.36 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 24 Sep 2021 07:39:36 -0700 (PDT) Received: by pryzbyj.telsasoft (Postfix, from userid 1000) id 22D798009E1; Fri, 24 Sep 2021 09:39:35 -0500 (CDT) Date: Fri, 24 Sep 2021 09:39:35 -0500 From: Justin Pryzby To: "Westwood, Giles" Cc: pgsql-performance@lists.postgresql.org Subject: Re: Performance for initial copy when using pg_logical to upgrade Postgres Message-ID: <20210924143934.GN831@telsasoft.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.4 (2018-02-28) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On Fri, Sep 24, 2021 at 03:28:50PM +0100, Westwood, Giles wrote: > At Orcid we're trying to upgrade our Postgres database (10 to 13) using > pg_logical for no downtime. The problem we have is how long the initial > copy is taking for the ~500GB database. If it takes say 20days to complete, > will we need to have 20days of WAL files to start catching up when it's > complete? Did you see this thread and its suggestions to 1) set bulk load parameters; and, 2) drop indexes and FKs ? https://www.postgresql.org/message-id/flat/4A8EFC4E-A264-457D-A8E7-AE324ED9ABD4@thebuild.com -- Justin