Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.84_2) (envelope-from ) id 1c21GK-00013N-KL for pgsql-admin@arkaria.postgresql.org; Wed, 02 Nov 2016 19:31:52 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.84_2) (envelope-from ) id 1c21GJ-0003wd-TO for pgsql-admin@arkaria.postgresql.org; Wed, 02 Nov 2016 19:31:51 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1c21GI-0003w1-Oo for pgsql-admin@postgresql.org; Wed, 02 Nov 2016 19:31:50 +0000 Received: from mail-yw0-x234.google.com ([2607:f8b0:4002:c05::234]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.84_2) (envelope-from ) id 1c21GF-000657-7E for pgsql-admin@postgresql.org; Wed, 02 Nov 2016 19:31:49 +0000 Received: by mail-yw0-x234.google.com with SMTP id l124so21170475ywb.3 for ; Wed, 02 Nov 2016 12:31:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=postmates.com; s=google; h=to:from:subject:message-id:date:user-agent:mime-version :content-transfer-encoding; bh=Q6WUqtx+Yvwv1ZUxxl81eYMnfqgC4I06ie+WIhRPPy8=; b=HRfZzZc41JCQHBGETTrZG9PA89//RGpg9LJAIqLAHlD1ZuYuiE7/Nf5TuCxLEbSYqE DGmnp6ZCrYfA2/6DQAJeVy5TmW9gmY8R3s0hHqzCj+b1jg5D8v/aCyCSivUVFrP7pPSV oTnstO6xlaBrXx6fQRSSWAsumXDxVWQvqhQSE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:to:from:subject:message-id:date:user-agent :mime-version:content-transfer-encoding; bh=Q6WUqtx+Yvwv1ZUxxl81eYMnfqgC4I06ie+WIhRPPy8=; b=SCSEjtEQHiqZ1+sKrunBakx8bbgDuBKacstVvslZIy7+pZvcBglQpW5P3YaAwSva9l k36YTqzwnHb8ADRXseKwMV4fddXCVWjV1N1Vt+eh503lfArau8SF7DMKa3JTp9nazYl/ HzFqYNH2W0JmBHe8Fubb37bl14QRe0bpeZ4s/KYCCJNFpvzycpcfwi+WchvJsajY5GJb fdCvHRSwelOI1gosPkVVwouatW24QXQ9CU6nsQmqUgd0ow2WFi8DY0gCDfW+ccYRw0nR OkBYVdXONFJmmxgPuPI9BWjKlE1ei0bVMDkEWSn1S0W3yCTgeLhR572dbPlHjz8HFnzh OSBg== X-Gm-Message-State: ABUngve+doW4b6AyS6SLIW3pHrJX5jcIaZj77QHxLT8xStLT8mOjS/sBRPc3EwBUBm54W/9t X-Received: by 10.36.133.136 with SMTP id r130mr3828904itd.25.1478115105098; Wed, 02 Nov 2016 12:31:45 -0700 (PDT) Received: from [192.168.1.102] (c-73-161-191-57.hsd1.mi.comcast.net. [73.161.191.57]) by smtp.gmail.com with ESMTPSA id b133sm11436565ita.1.2016.11.02.12.31.44 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 02 Nov 2016 12:31:44 -0700 (PDT) To: pgsql-admin@postgresql.org From: Phil Frost Subject: Replica lag, high read IO, vacuum index scanning bug? Message-ID: <3fe69397-9feb-707a-032f-968a8d48ad0f@postmates.com> Date: Wed, 2 Nov 2016 15:31:44 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Pg-Spam-Score: -2.7 (--) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-admin Precedence: bulk Sender: pgsql-admin-owner@postgresql.org We are seeing an issue on RDS 9.4.5 where replica lag gets really high. This lag seems to be correlated to a saturation of disk read throughput, and vacuums of one of our larger and more frequently updated tables. I've heard that there's a bug where the replica will scan indexes unnecessarily when doing the WAL replay when the vacuums happen. Can anyone confirm this and maybe point me towards the commit where it was fixed? Has there been a backport to the 9.4 series? Any other ideas for investigation or mitigation? -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin