Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1hAHGT-0002Y0-Gq for pgsql-hackers@arkaria.postgresql.org; Sat, 30 Mar 2019 16:55:29 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1hAHGS-0001zi-7Q for pgsql-hackers@arkaria.postgresql.org; Sat, 30 Mar 2019 16:55:28 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1hAHGR-0001zU-OD for pgsql-hackers@lists.postgresql.org; Sat, 30 Mar 2019 16:55:28 +0000 Received: from mail-wr1-x443.google.com ([2a00:1450:4864:20::443]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1hAHGO-0003zm-OZ for pgsql-hackers@lists.postgresql.org; Sat, 30 Mar 2019 16:55:26 +0000 Received: by mail-wr1-x443.google.com with SMTP id y7so6348041wrn.11 for ; Sat, 30 Mar 2019 09:55:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=2ndquadrant-com.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:openpgp:autocrypt:organization :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=R/a9uiGnTwhHVI9eV+mjbwhObc7Q3uKX1A9UbrVo8Rs=; b=LZ9eTGB8LEMAczBjLHIJ0bppjyVdy6xsdazCTK6/1rJxhHoc+GswemNxC0ElZnADLv sj9TLbj9SZ98s7gUyKIsMcNogcCHmp/yRQJYmWAwBlvwo9HW2qobSa6ntYIE98weJFyj Ppd0pyjVG1jFUDWU94gYNwzNNIOpH/StbxeQgZWdofrfMctBExR/SOVasreswh56RBAD E3pFGyXDVEwWE9mYLHfuvc+wz1YJ5W4I5rA/fCLap3yabh+vyZgvwNd3wG5jLkhWhDoE 5N7tWWuD/UsTYgoW0Z35bS+DyYbWrJFfdCphaev+/79MFvIeJGzPPQJkeLzWpKdcnBw5 9HUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:openpgp:autocrypt :organization:message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=R/a9uiGnTwhHVI9eV+mjbwhObc7Q3uKX1A9UbrVo8Rs=; b=pwBwjSiaKDXuCs9fscWdy/E5/MDk6VsrhPhTu8Dx6IxdDnrnG7cBOzEMDb4Oq4iM28 9I9XLpmjYEfuvaT4SRGn86YwX9za30qsdt2e6VvWKZojtPeEUmOtjv5Mp4m17PtdijBT i3no4wclo65OAV+RACJdojpmf8x3h1q7yQuEPG9AplJ3RMYKZ2gd0QVy742yG0H1796I 6bQhpSJOlvLHRzg9krR/IZTO+3zMf7Ev3/3FCkbLJ4f3YhEbfd7jPVdZsxrDWU2e7a00 ma4aMDOhCuzwPpMORsax9ilMURkR9bUD6Y5mtlcbwdpI6/6/2p5T4fVgy8fXaZpkCc/t wR/g== X-Gm-Message-State: APjAAAVyJgTnsQ0lMZeN3ruydQSAETw1CV1YcQXcBYdZiDNPEfe7CG5I uQUNlTph7ERpp9aH2vXqQ8lCEmYhtQ2dXKTLSgF2pkMlqmZ1U54cC9m5CnObKjevoiL8zv0w13+ +MhgAfu6+eqY27l3GHrA/iq88eYyKQrc2swloSmB8XbmWHA7cIbDl/6Oubo5tw5fpbkwUg2+sis um3Sf+gqG+S9DYZqopGTs3 X-Google-Smtp-Source: APXvYqz+qhdpiez4iUf1mwWEgirH/2wv8HtHsVAReSZ9t7gBRMVjUWr81rjN3NQUSUD65pYWUfUpQg== X-Received: by 2002:a05:6000:5:: with SMTP id h5mr4174233wrx.271.1553964922785; Sat, 30 Mar 2019 09:55:22 -0700 (PDT) Received: from [192.168.0.3] (ath91-3-88-167-161-184.fbx.proxad.net. [88.167.161.184]) by smtp.gmail.com with ESMTPSA id c20sm9515696wre.28.2019.03.30.09.55.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 30 Mar 2019 09:55:22 -0700 (PDT) Subject: Re: Berserk Autovacuum (let's save next Mandrill) To: =?UTF-8?Q?Darafei_=22Kom=d1=8fpa=22_Praliaskouski?= , PostgreSQL Developers References: From: Vik Fearing Openpgp: preference=signencrypt Autocrypt: addr=vik.fearing@2ndquadrant.com; prefer-encrypt=mutual; keydata= mQINBFg9q8kBEADfBQMKbYmvL7k4D4ghRYon0lMRf9c+dADesMGsBh5SYTK1/t4S4iCfbc8d QUCyJWSiJYkT1MZNSywbr3hNahZ7qAN0y4kvGNUIxXJGcza5qH5gHQYFrpB/IDoZcZ5/l260 Y5roqQos7kimpALNrl2cXWkR98Z9FWZOocImzdBnKV+qHzD9ZCPISPqR92N67gPqLD4YNx5Y T4NkT2N2nYZlyvv6X3t5/haLWpSLTJ/7ESD6jwTs7Ok4meLidSqr8U25p/L4mb+pscR6l5Hq aJXagW9oQ0uI50Wlri1GeE/JTqaURxfOCWBE3RzrRdhH0HYMOz49gLqba0s4eBlMw18R0OWt TD4mRnpD41X4aaSrMEqm2xXZAa8T+UFlQXWliMvFJ4rQhAogPLN9yI93FDy5pqJdtu3T/mJu HycJfiJHtgGVP8bsMQdg5X4kNWO2LnerJQiO+cYambgb5RWBS3byjO8f05sIXQxIR1nLdGEY 7Cy5/eIN6+qRA3kpRjfQ2QOlEKxVWNLu9t+xGphRi6TidZWnhHRwfCkeelR9R4niubLQPirl /XvToUtm0opXBHbVxTVblM0i7z+x9tnQUCa1f710pnJ/LRqE2Se/JPbEeg5JH4MLLGdr8uxf fv9E8dJVvmTRIWOJFNFnd9e3b9K6I/ek+ysVFsNrKnP9dS4R0QARAQABtCBWaWsgRmVhcmlu ZyA8dmlrQDJuZHF1YWRyYW50LmZyPokCPQQTAQgAJwUCWD2ryQIbIwUJCWYBgAULCQgHAgYV CAkKCwIEFgIDAQIeAQIXgAAKCRBzOWT4unEhz0W+D/0ZqG/7SmWxSPSm3IcwQaGvxKIE/XFv jZ5USS6W1U/Ttq8qXnJbWOTzEKl6NVQcAuAIAYdvOjwrDDR9SMSMPvHmd7LniOZ4/+V2rdLj fOXzExI+1yJ1TYv1pGTgt5ZmjARIvge942iEYHQey5pavoNI1/C9V0RLuh2yS6dLax21295V OFOhQD11mDJLJ8c/ILVpadqt90V9bQzBWycPOjnwjhxD4kVXMjihnl54EuMp08UW6OTCP4ux /SxJFfTM1WDr+GDk7RgsG3bJfwftdUP2Qc8drwu0LAT41edmaigFGM1RMKeBWeo5Tdcm/Wmo uDeLJwL+CGvEZEjNrLxoXBhuRcv+9m5eYOafqewKgZXTmeZImbk75Jp502LCQHXhdatr9O8a KKjmkKqz+uPOV+4qsvqXdMCCFWb15/Pd/a8ffPMRwq6ebDOPYXJUkGqc65tSlf0gRxfIKof3 gO3TevZ0U1nc7kBa/QNe2QN6RC24+GbyJNEvQ6g2ZbBddffiuYZElsGM3+yiM6jz2RWG4uLQ /Qjw6ZwhNUCOepVa3SsLkSE0HN2M2emN8exQIdJw37IiKIsZPPFa+J3netLfXB4F+/BnwVqK GW8eaLYQWQYRBPTMKgLCqGDCgNIB8cVG7zmx7zYDPXNEwKPfcGSkixfQJxgq1InWDyVg2qwU TP0iVLkCDQRYPavJARAAsdzHCSxLnqy1UW0qNhJoZaIkSbQQ6Bjqgfo4KZKvt2eBe363ubgw 5j6VCfpBpOSJYQrOadY7GOKw0m8iwtr1bvrNg/7SoqfwoKLBTvuxo5WsUD9e8f/Hy1CoMz/3 o1qAvBiQK+MnZvmgtwvwqWpx/g+H2uH8aPttNC1x9lO866o4JtoVWumi01si6KKjO+ZNazPi n9a0JW4OVeLxhIy0OnrE4KMOrnHZlJPIdGP36dtFMK9tyqQ3z3zOp/HN+3mrQZAphH/2idLK wRZURPpisvy3HES9yN/JAWtPEYB0OAhKGJhgLDue08zM8FRNa5ldc/oJCqMEC8D5bh6g3LCz uIM11hOMD8OJNX1lGnzdicBLMghUfhtAJ+KfBGqIG3hLxzYp8RiIfG/HJyA6dhqpxG3y6yjO 0r24THuXceW7ygqQUB+J2KK8B1iWTwiMBvvIav9tOQTcYkU+EFtN7zwkC3aN2HV8I7QLDUWw OyWZ1+J1ff3xyCjMhZ0SaBV8kUy981RE8lnmy/T6lE9iFfj2roSkCLcUkODb9mz1BSeg7l1J yRGqQ2ypxS3l77h+JC1SMRrDUz1vFIIAuvQEjJgnV5JUjmNi/0s+/HQU8YRlQWERSH9pHwFL ZgSwH5hqCLFjiGLxxHqJ4JZLoP7J1I5tpuUkbVPhreRNNZ9ro0Wtd9cAEQEAAYkCJQQYAQgA DwUCWD2ryQIbDAUJCWYBgAAKCRBzOWT4unEhz2qgD/95NfvOedTfZ2P0hXTHlE0oGL9pIP21 dOng/fJT1yc4nA4MfNV3Hf72Gpw/XLpnMgovanqj+15ZyFPirCaB9lCjEiorzSsLN9k0I1OC +gIbouFHgYo3nzeRJf/hMlAzef7sqjuD5WyI7miMeaKXkjTGyjC39xX2nq5EHCI8V/uBtT94 3cyZmCrEl64AKjlnFfkNKhlYWpfDWSVSxz79QAIw0wW4ILNFfAp3z4DPjqGut98XBlmtQlDj RMlqeUHDIGLQjKNmxH95vbTS9eAFom6jA99Bl4nWCWUYHoGA/y7Ov6z9vQwqkJ4WqPMg+mas 7nZNQiexBj3tG5pRU2M7NURnaLiINFXid+kjKmNEUgNyd0haIqiIRtkRWZ2zvAwfudS7HDq1 RB6GWGHtJrVFRIkgs3gBoyPz3kSwlYieid6Ywquxd3dQqkN6cdlnvt2uossgRnqXx5HugP81 Faadir4t4t6HqoLHynzvvT9fMijT//O6Fb0MFqZP2rf3i7zh7/rwtpdxI2m/SbTJgvMqlvGn riBgDbjEz8XNMGqMk/WPNcKkk8prCPta9ltMl7yd3LqzCG6KRz7TtQae25BcBF1IKobZYbgf qc5fRxkCHCy6IDu70pve2tt1hzvZquE2Z5uTMFp2k213HJQKklY/jAOmVhM2O2T2oSaKAr1u MUmXcw== Organization: 2ndQuadrant France Message-ID: <271bc1a7-2b59-e3f3-56b7-c8bc5456424a@2ndquadrant.com> Date: Sat, 30 Mar 2019 17:55:20 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk On 27/03/2019 21:54, Darafei "Komяpa" Praliaskouski wrote: > Hi hackers, > > Attached is sketch of small patch that fixes several edge cases with > autovacuum. Long story short autovacuum never comes to append only > tables, killing large productions. > > First case, mine. >  https://www.postgresql.org/message-id/CAC8Q8tLBeAxR%2BBXWuKK%2BHP5m8tEVYn270CVrDvKXt%3D0PkJTY9g%40mail.gmail.com > > We had a table we were appending and wanted Index Only Scan to work. For > it to work, you need to call VACUUM manually, since VACUUM is the only > way to mark pages all visible, and autovacuum never comes to append only > tables. We were clever to invent a workflow without dead tuples and it > painfully bit us. > > Second case, just read in the news. > https://mailchimp.com/what-we-learned-from-the-recent-mandrill-outage/ > > Mandrill has 6TB append only table that autovacuum probably never > vacuumed. Then anti-wraparound came and production went down. If > autovacuum did its job before that last moment, it would probably be okay. > > Idea: look not on dead tuples, but on changes, just like ANALYZE does. > It's my first patch on Postgres, it's probably all wrong but I hope it > helps you get the idea. This was suggested and rejected years ago: https://www.postgresql.org/message-id/b970f20f-f096-2d3a-6c6d-ee887bd30cfb@2ndquadrant.fr -- Vik Fearing +33 6 46 75 15 36 http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support