Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1qyu5V-00Bvw7-Kq for pgsql-general@arkaria.postgresql.org; Fri, 03 Nov 2023 13:19:49 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1qyu5T-000at9-Gh for pgsql-general@arkaria.postgresql.org; Fri, 03 Nov 2023 13:19:47 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1qyu5T-000at1-1z for pgsql-general@lists.postgresql.org; Fri, 03 Nov 2023 13:19:47 +0000 Received: from mail-yb1-xb2f.google.com ([2607:f8b0:4864:20::b2f]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1qyu5Q-003lgZ-EX for pgsql-general@lists.postgresql.org; Fri, 03 Nov 2023 13:19:45 +0000 Received: by mail-yb1-xb2f.google.com with SMTP id 3f1490d57ef6-d9a58f5f33dso2069989276.1 for ; Fri, 03 Nov 2023 06:19:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699017583; x=1699622383; darn=lists.postgresql.org; h=in-reply-to:from:references:to:content-language:subject:user-agent :mime-version:date:message-id:from:to:cc:subject:date:message-id :reply-to; bh=d40cAHM3N6rs0cII+6EAYUxbHhrXNZaVCu8xj9108HY=; b=BAib+grwKUTKoSne7rc5fCCZhAJ+77VM3jGET5h31Ubd5PV7YCqLyO8duKnbyNcoM2 0UQ8IizyJxD6KW34wNEnE+8m/YRW0uaDXBufjatmzUolDErzUy4iDwXnInEi7ZvDNADD bWCClhm2kxCyZOKR3+6DjrRxfITJUdXOaHaYAjcX9NaPKkO8R2EP9pMwwECbF2RopSWa fOwYa3EgEiR30z2fTczNpkJEfAcI+i+t/2Mm1EbIPl4xJwoNpBLojAnk/NwRQ4TP5Zo9 IGiNay4o2qlUkS4OZzUCT9XbnO5HlLewnfz4MTzc/sRh5mosSpeg/8pq3nVBJjZH4bSx AsvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699017583; x=1699622383; h=in-reply-to:from:references:to:content-language:subject:user-agent :mime-version:date:message-id:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=d40cAHM3N6rs0cII+6EAYUxbHhrXNZaVCu8xj9108HY=; b=Hmn53NvyM7e1szPpAooDhRyKrPv8fYzhAq5tU2qJuWXZNavw63vLgYNqi8IFhEDjEj ppcTyb/fzv9c0o+A29guAdHIZAx8yIAOZZxX4JH4okL+pkipEI8szNitrQjzIOAU7S2r b76Fn42k5V1wzl8akOffq8gD/j8/NnZno1NXrMgY1bau4d2XqIvlAAPyk2izfkyZV35E tSzzfnz0yM6uuJ2pYgCq9+WkDh2+iOwht6EUl4rEsa0i3aOiHq5toLy24xrkOFOFj9yn Q6gYV3HPEqZ8nu5hgBVEdyLoATHCG3b0N7gfPevy/3qz+N/DAju6PVfukYucUQW7Vgv7 yBTQ== X-Gm-Message-State: AOJu0YzajNB0GjMrj9DAax8QvKvdwxRKzeXHRPbk40qyJgh2MdXgBSDC Yznwtrix0zsmqhNgckfYv2dcBUqMiLM= X-Google-Smtp-Source: AGHT+IFLi2F6bF2qDVrCqCtc8EVExgxUkbWg1Z28VmjxK7YsGsYvkFDF67xmIJKqAYxXlFyruGur2A== X-Received: by 2002:a25:5f4c:0:b0:da0:7c0c:edb7 with SMTP id h12-20020a255f4c000000b00da07c0cedb7mr19528456ybm.1.1699017583064; Fri, 03 Nov 2023 06:19:43 -0700 (PDT) Received: from ?IPV6:2600:1700:bd2c:b810::44? ([2600:1700:bd2c:b810::44]) by smtp.googlemail.com with ESMTPSA id 203-20020a2516d4000000b00d7745e2bb19sm828309ybw.29.2023.11.03.06.19.42 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 03 Nov 2023 06:19:42 -0700 (PDT) Content-Type: multipart/alternative; boundary="------------NsGGZTYm3rf7R55kBaAPvMAm" Message-ID: <413c6c5e-3c1d-47c6-98cd-a7f046493c76@gmail.com> Date: Fri, 3 Nov 2023 08:19:42 -0500 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: pg_dump/pg_restore --jobs practical limit? Content-Language: en-US To: "pgsql-general@lists.postgresql.org" References: <0c82a8d32181fca3b979347f34ffbb2f7fdf0e90.camel@cybertec.at> <93e2708b-2dae-4a04-8660-ba5e1f5b857a@gmail.com> <1828c313-5a85-4e31-93b4-ff2a37fad862@gmail.com> From: Ron In-Reply-To: List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk This is a multi-part message in MIME format. --------------NsGGZTYm3rf7R55kBaAPvMAm Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 11/3/23 05:09, Marc Millas wrote: > > Marc MILLAS > Senior Architect > +33607850334 > www.mokadb.com > > >> Testing pg_restore with different --jobs= values will be easier.   >> pg_dump is what's going to be reading from a constantly varying system. > Hello, > > each time I do a replatforming of this kind, with DB up to 2 TB, I did > create the target DB, eventually needed users then the appropriate > databases, and finally,  a simple script to pipe pg_dump into psql, > databases one by one. > So.. one thread. Each time, it was limited by the network bandwidth. My > last replatforming with a 10 Gb net and a 1.5 TB DB did show a transfer of > 500 Mbytes per second (5Gbs) so.. less than an hour. I'm shocked that old-school "SQL" single-threading works that fast. Thanks for the data point. > which is just fine. Launch it, have lunch, a coffee, and ...done for > test.  For Prod, I am used to do it at the quietest night of the week end. > and have a nap ( a short one !)...:-) -- Born in Arizona, moved to Babylonia. --------------NsGGZTYm3rf7R55kBaAPvMAm Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit On 11/3/23 05:09, Marc Millas wrote:

Marc MILLAS
Senior Architect
+33607850334


Testing pg_restore with different --jobs= values will be easier.   pg_dump is what's going to be reading from a constantly varying system.
Hello,
each time I do a replatforming of this kind, with DB up to 2 TB, I did create the target DB, eventually needed users then the appropriate databases, and finally,  a simple script to pipe pg_dump into psql, databases one by one.
So.. one thread. Each time, it was limited by the network bandwidth. My last replatforming with a 10 Gb net and a 1.5 TB DB did show a transfer of 500 Mbytes per second (5Gbs) so.. less than an hour.

I'm shocked that old-school "SQL" single-threading works that fast.

Thanks for the data point.

which is just fine. Launch it, have lunch, a coffee, and ...done for test.  For Prod, I am used to do it at the quietest night of the week end. and have a nap ( a short one !)...:-)

--
Born in Arizona, moved to Babylonia.
--------------NsGGZTYm3rf7R55kBaAPvMAm--