Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1knQF3-0004wK-1s for pgsql-admin@arkaria.postgresql.org; Thu, 10 Dec 2020 18:00:37 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1knQF0-0004XN-FT for pgsql-admin@arkaria.postgresql.org; Thu, 10 Dec 2020 18:00:34 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1knQF0-0004XF-5R for pgsql-admin@lists.postgresql.org; Thu, 10 Dec 2020 18:00:34 +0000 Received: from mail-ot1-x331.google.com ([2607:f8b0:4864:20::331]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1knQEw-0007Tw-3e for pgsql-admin@lists.postgresql.org; Thu, 10 Dec 2020 18:00:32 +0000 Received: by mail-ot1-x331.google.com with SMTP id w3so5705186otp.13 for ; Thu, 10 Dec 2020 10:00:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language; bh=vfEird9GOUNvPgGLQwt+CSRXCzrc89PSWCNB08ptlz8=; b=YmYIyVjU+z/cZoSsLtzmwxU4yuxMTz6HG1gNVCkYn8FgewDIVyyI5xrOi2k1TPAtpg Qdq6GcczwRpVSbNmb1iwmk6kdeuq2YbDZaXErb1maYgxcze0VdecAtKVVFgFxhpPs+1d AIhrK3avSNharQj3AB++LN+oO3BnhjLYlomMuwsCC3aFdMsqS+Cv750rS6Y4qGDlJQlr gMSKrl93hSjJBe1UrOjUOvH91B+w+GXmtYgS4KNN/oL8NHY+cY9nbV4Sbnxxqj81D5ec BXYow4pf2RugV8cKDsS17bEICAt+CouCGi9fgngdq2ZdxZMBEfR5bk1Uqk7goIuJkE2a ijmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=vfEird9GOUNvPgGLQwt+CSRXCzrc89PSWCNB08ptlz8=; b=ohQu7YK2PLdGhYJlttx/pYfRkJ1q1UJcE5OekAv2nJsJ1CX4GU3HBr5twzBwmbpzqb tHJVfQ56x/rfzaDYAOG85//Wi8cZcIpqd6Rp9sMl5F0gVxFjpI6CdPzYCUH8xIgk81Ut Y2CCfeAuG4w4efHCVtB3PJooAxlEZM8oogRviKzZM6zABZfHGlZCHYi9Ck8bLOeFbGdK nAekuI4lH4AowfnuCfrfiGrNY0onKjMPN/QGLF00MQKFdEn8y50arZMJmh8yptd7UYta yEd5I1kXHHjEt3Yh9NpNucylPPwhw39uo8Bg/liRBH+uoZ5jP8gXkL0izkxN/EKMwRRc RBQQ== X-Gm-Message-State: AOAM530cAZXYkFc+Pe4ozpwrWU7KvPGuiQ1R9i6c/mU0H1cAg4wtQhgS PCwn6yiB7bXpPu+146nMTp/q6Uf2704SfQ== X-Google-Smtp-Source: ABdhPJxd3UCFc/TrMcgelne+zKBW/oWO/hfGehCFMgUyP33G2vc7sl89/I+NJmh40xkmkasPRK/AWQ== X-Received: by 2002:a05:6830:4b9:: with SMTP id l25mr6543479otd.218.1607623228840; Thu, 10 Dec 2020 10:00:28 -0800 (PST) Received: from [192.168.1.10] (ip174-67-141-204.no.no.cox.net. [174.67.141.204]) by smtp.googlemail.com with ESMTPSA id 66sm1232180ooj.0.2020.12.10.10.00.28 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 10 Dec 2020 10:00:28 -0800 (PST) Subject: Re: Optimize streaming replication because of network latency To: pgsql-admin@lists.postgresql.org References: <99AA0D0E-E266-48AF-AB29-4E2B73CBE3EA@webedia-group.com> From: Ron Message-ID: Date: Thu, 10 Dec 2020 12:00:27 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <99AA0D0E-E266-48AF-AB29-4E2B73CBE3EA@webedia-group.com> Content-Type: multipart/alternative; boundary="------------22FB7762CC5FF04ABF6A3755" Content-Language: en-US List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk This is a multi-part message in MIME format. --------------22FB7762CC5FF04ABF6A3755 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit On 12/10/20 11:41 AM, Mai Peng wrote: > Hello, > > We’ve added a streaming standby on a gcp container. There’s a lot of > network latency between our primary in EU and this standby in South America. > I did a big update on a table ( 4 millions rows) and it generates a lot of > lag : more than 5 hours. > The standby has great hardware ressources: cpu, ssd disk and had no load > during this update. > How could we handle this lag and have less impact, less lag ? How effectively fast is the pipe between the two servers?  Where are the bottlenecks?  Is the pipe shared by other users? > Basic PG Config: >    shared_buffers = 8GB >    work_mem = 128MB >    max_connections = 600 >    wal_keep_segments = 1000 >    wal_sender_timeout = 0 >    replication_timeout=(not set) >    wal_receiver_status_interval=10s >    max_wal_senders=20 > Checkpoint_timeout = 10min >  Max_wal_size= 2GB >  min_wal_size= 1GB > > > Thank you > Mai -- Angular momentum makes the world go 'round. --------------22FB7762CC5FF04ABF6A3755 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit On 12/10/20 11:41 AM, Mai Peng wrote:
Hello,

We’ve added a streaming standby on a gcp container. There’s a lot of network latency between our primary in EU and this standby in South America.
I did a big update on a table ( 4 millions rows) and it generates a lot of lag : more than 5 hours. 
The standby has great hardware ressources: cpu, ssd disk and had no load during this update.
How could we handle this lag and have less impact, less lag ?

How effectively fast is the pipe between the two servers?  Where are the bottlenecks?  Is the pipe shared by other users?

Basic PG Config:
   shared_buffers = 8GB
   work_mem = 128MB
   max_connections = 600
   wal_keep_segments = 1000
   wal_sender_timeout = 0
   replication_timeout=(not set)
   wal_receiver_status_interval=10s
   max_wal_senders=20
  Checkpoint_timeout = 10min
 Max_wal_size= 2GB
 min_wal_size= 1GB


Thank you
Mai

--
Angular momentum makes the world go 'round.
--------------22FB7762CC5FF04ABF6A3755--