Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iXB8i-000389-TF for pgsql-hackers@arkaria.postgresql.org; Tue, 19 Nov 2019 21:34:25 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1iXB8h-0002KB-93 for pgsql-hackers@arkaria.postgresql.org; Tue, 19 Nov 2019 21:34:23 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iXB8h-0002K4-0K for pgsql-hackers@lists.postgresql.org; Tue, 19 Nov 2019 21:34:23 +0000 Received: from mail.thelabyrinth.net ([45.56.70.56]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iXB8e-0007Dh-9F for pgsql-hackers@postgresql.org; Tue, 19 Nov 2019 21:34:22 +0000 Received: from Davids-iMac.local (c-73-39-102-148.hsd1.md.comcast.net [73.39.102.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: dsteele) by mail.thelabyrinth.net (Postfix) with ESMTPSA id DC86F54625; Tue, 19 Nov 2019 21:34:17 +0000 (UTC) Subject: Re: backup manifests To: Rushabh Lathia , Robert Haas , PostgreSQL Hackers Cc: Jeevan Chalke , vignesh C References: <4b6244ea-4094-9055-080b-7afb274bf270@pgmasters.net> <146910fc-3df7-eeb4-ac5b-2f57a01f646f@pgmasters.net> From: David Steele Openpgp: preference=signencrypt Autocrypt: addr=david@pgmasters.net; prefer-encrypt=mutual; keydata= mQINBFSheh0BEADGNb3c1kOaX5OGUDhqHTwJ3JW7XAllfWLdknpRg+/fihgLPPRnQHZhjq28 nmMLqM+C8TjezT2BgVACeu8qqASnuzN/E5+bAl9hVcgCP4eV/lEIWwVxmhcHqSdeWqWZSBRB tf07fLBnz22+3LwVsESN+fanWxnoqqvE0+hyGnCEDhOBkihgbYHuq4vw1+xgoGN1uDk4NShV El0ZWO90P4e5mOWA2Zsex+8YhGJlfRwMpU69okFCvjpWJKE+ufeJdlrxEpSSKW7SQ0kG1VF7 CvDKpF70TCWkb4Bagb6c7lwZrMjxuBnlIvOlV3RRY67PAaVIFRRvQJeJABOPHp8HnH8cwici 7gwwfWBrZZeAdPqs6VNjANqI9uaDasRnD3yHP6CcHIAeCl5hYnzNTkIqBRlnmD5y8s0VRDLr 6JR3FxscOsvwXPJ+He1RpRlxQDlBboaRFUz8endeQHhGUeHyekQcSYUaz4PX3OJBG0Xi7nr8 EOP6zOeP61z3dL8NuO3QjgPiygvS0qwMEPvOYo52nEGYn08nvFVajI8yxOXFnIwnmpgr5gdq nL9ga72tvSyJQAUx+YB7r1bKjRn/Mlt/fPIS5PXONofiO1K8uX+pzvIX6dnm7ihpFf1XC+uG 5+DKBnU7yS7jCRL52GxWHbuC3mT4tvzqH2b13TLng1Y7Izi+hwARAQABtCJEYXZpZCBTdGVl bGUgPGRhdmlkQHBnbWFzdGVycy5uZXQ+iQI5BBMBCAAjAhsDAh4BAheABQsJCAcDBRUKCQgL BBYCAwEFAlShuEECGQEACgkQn830JKJznPPsLhAAtKkIu0mFfxcVqApg4PNjZKYY4TWeN+I+ go/VKX8QWMadr33t8Sxcgyn48bjBYzgWQXhF4rMcFxDxI3ly5bdE8QKZkhFJJT7N2fBoo9Zm ZOsOvPd+5hz07VZpgNSiTXMAphj8+GSTLbYufm/OUnfpS1eovPtPdYPQpNC3RVHA0gtisz26 NgMLg6QN0iQuITs9N+wiaGtyVIornl4XAY2dM/Hvfr1aDFkukNNr52sc/mGqocG/dKC7dONA 3860y2MLszQcbAE45iEkmwpGkvxyc3i3tvdD3VNlVI1MKJUBnxFPUEI0lVq5WpApFSIU31+j aNunKIcfVxRnM8FTJBsw3RnFRpSsEtiLaZRm5sOZHa9+dy/DWDb+8O1I+momCo88+Jksd6pZ 2Tz9msOiJiTnFsCZgfbbmayH6w8ZyfHuO0DuxScM1UK+6CLC2VVjQFuH6WHqJpaENg5INswS qpUc05iC4i+6dZOe44cviX6LpHuP8vWM0E63CMun0l42yHWy+k6nLvx/fj7t++tF2JIJ92OG n/2MkM5lIKknRg27UjYAspD6f3NSZUMIe6tGyWPTtLOu8zOL5CeRyCHMWwUVDKJSBzlK2mV8 e0EbmtUJFsoNWXWKPKUIxX7JXPN3v523HmOhtyz5SnkGtwEqCZiv/JccYmh+S2IokAuscLNP EJq5Ag0EVKF6HQEQALPpUsP67xfcj6NUMyySXSytUzMIVi5XNTyazuNcVfFZzT9QL8wFQkgy i/HOD3tmvdGxz6H8V+v+cszp88gCd8TwClss7u+iPsLlAQQWC4q0WiX4OhwFv2H004iuft8i odjiWDkYEgIFxlPNxQfHFmchkyODIJXQmnzbgsXwGZtJzAxEVh6AgYJqE/Ev/+kTJhJ7703n oIb+PXvafoB5axnU4nf7DuFa9X+q2/ZfzyAXUCKvzDvxb24iUgWFzlWBx0CnI21Kh6aS6hR6 o00D2OBiU+yCpW9eRTi3SuDuslmtCUaZz5V4SHo37ochEYfZ2LYOPsBO3PUMx8T7MDYrmv/x 4W5rdjugz4NXWROStNNLpS74LW20rdrEKMDltph7U2zfjjS663K+CUxoXn/ha2WnzBocoY5o KwdViLln4s0ivOTa4clHJJUsYiAxLShAe2AP+5kJim24tAabbEOsrRTn9RpT7J8jVT/G0YDE glujEt0U04Akkj5NO4G7vJP/AyX5zKZetkp/HFiFBZMkUuXD/KgDc5EM2fpQtaxGGY2fcQqY VHUbbxstxyl5d349M9rok7O90hw9BkP0Q8UEpgmlZ6ftwx+ZfZmBEdQ7HcMHnSPSNJkXt/Vi ozlcrSnkVXzCXvH9DHrIpgenp8btsJ2XyjSdrEu4jAqYeirW+3qRABEBAAGJAh8EGAEIAAkF AlSheh0CGwwACgkQn830JKJznPNmERAApYaQK6i9F6hM2RUD/PqQC45WmzOXK9bDSS9hdk4p Nc5jSW2+FkcMFe1XFG2NyGf1v5KBPxuDTQp35PIwRsL59xNmPICu0ae6zmCPUpxbMIEzb/VU wXqeuNp3cRVll06goDt5w+2LlAtBrxuDjiWxby9LH+vsIUMLWYu+M50pdtOcXOsO1+zyfs3w U6xRZ3PaFDKO1cB68okDFG/cFHnYfY5aETV7LgVBHEOQLwL01U01QC/iqY/aNO4iUH/+PBc0 R6mYlP7IqDQOQVTJ0dXP365thl0Q7c9e2JQSS9wuSv2EEiR//nnZREtLyrQNLwsXE8UFBDe7 uA8Lm2ldmQYpHHKAcorGVTHYJaaus2LjH8tZ1i4RGg4hKMJSHil/5C+Ghl+PmcACVeGaxB0c mJ8+0jyWDkLCDMAXJJyvXRWIONTAxVt5kKg1/iMAq1IQYNd66agnmgxdSvKLPUo0JJ5fhQJp 2bqDDTrINTZg+Rub2HeoOMxYJsMypK+6aY4iVlm2Z7A9SgyQDjGAXzD9+XaezNFeXxfPbjo6 46qA9SycdYhiwaTcTQakiLkZWRIkSjGQwNCK9NktE1fYnXfR0+WaWFduQ4BfM5uta1bt8esB /mLEjVYS9RRAWZ/untJrR9JYGTC4Mu6tcLGZD7385yX3TNM1iaIUGxlKkHHCZCnILyq5Ag0E VKGipwEQALe8Af9Kgql5rC6NPrOeuOuuqwXMNIwKsTLG8PmmoHKEHKukYuX/YfnGWFU1Ir6t y0qXe2XZs8AnLNI753b/7Z7ftMuMWzWqzfU8Pb+8/NrOQqfO05++gtsczY/tflVW2YWJWDZ4 lhytY4frlOSRuMOI4TjM3NwgkVri4sM4U+OdM8tdkW9IdajLCP0J+wFO+U2dU9b3yCss0GNY c0dlbmI6sOmpV70ldAb8i0BqfkGpq3xDcnzf4EQDdF/mBLZwBe0v6dzlxHh2HqL9yztjGXnk NtRFYGzKw+ZBuOaSj+OPG7iaBSgeNxDYjww2KTzdSE3DbfGqwrxn/ejD+ZueNcqxsoaT3oNr ADeluXmN63C1CcdFmJjAiCCjt3diLgmVL3MyBV+LcSwEz+2B/tlRHV1FWvy8VX1Nj2CwCO9u RuJ9HrF05TXjjNC2o6fFME/TkEzsklzvLxP/Bs/wKDvZQKGlzshk8EsmRhnJjzekOg4gOyVM LvjbGiBPMh309i0nVgExz0LIvI5Wp8G1YXHiuf6i52pTgeiGfiV6D12b69ABqInOfvROSvfe F36OLmoRicJcd8wEwkNPoSHHn/Zxf0fMHe7ZV7EtNSeqB+179SxDqeIEAOhpye6/q57LqvpZ qm0ADa5O3J6IMpId+ov0el553e7HxfDmaWJpm+NemLWvABEBAAGJBD4EGAEIAAkFAlShoqcC GwICKQkQn830JKJznPPBXSAEGQEIAAYFAlShoqcACgkQgDa4glDlIUCXfQ//aozIciE9Je5m kUlKECjXut8vvpO9fbAbxY5VIEWANV2PpIw1LO/8WVstCl38vBZtOxCcZqccLge12Ow72WPF 5G+v+8OJRwHNjrdI/WvFeMUfqlWB2ok+OOlm9t4U0lTK7YX4K6PvjXiMh1Kpvx/JsxrnGJJx X+Ky4icNWdiyaVBs+NDROasZXmy+7S8yeR9ECPxu/DPziC+aPW5lJ3tGqFEPADpmeD8G5K7U QyDJ4GRgD+Agez8r60qyhiwFyUXfcn0bOsIx/og4gae8GnadPl1+93+PI0L8Xb+oYcJnvVGE zJZJIMsp1Zjhi3PnJEulvDNOxkY7Og2dEwkNSHrru3Z5odl8s629VaYyT1b0Q/GnDAnKtZc3 tKZVKCvvAf9UmbuBfbuP084dBFeMeFWl3ET9UWpGhKm22KXg8TrMKoM62bUSjW5gIxC8ywLT mu7ivcKbxj0rXFB5AdYUff9QsCZFdDZG3fUymOQmYgNhwXAAIwKDKD8CINc2CIVc0qMhh3Hy WyyA8JZKw/iK+652WHoKEMmxVunSvvlDIMDf/bYN/HxoUEBcCSWsISKW6xGu1hPKfYK2t6/9 CVjluTOPeFiBt1ELXsOlpRPe4f0M2Cc/AEFukE1ZoJWlUhtU4zuOpvDtGST9D9imYsgMaMa8 TtEGFWGDmOfws4yU4tTTUNqFRRAArysPyXEhNNbLncfR5lPGPfZORPqLmIdkNMcDAwmzvC80 g1WH46wPzIcUUTc7DnP0pZNWGmI2NAnAW/c4Ii3aiXQXZDNVRfdZKL5rEY6dtgxdHl4hc6KN KH17wsX4T3dglMqP7VikYT23sNQJVE1oIGjfVR1sOe+DleI5Awa50DgGyLZS9vLfZz/9F28J 2YM1DExVPxbj75bwfOl65hh8B5BCqqrYp5n/JqV2x5+OSPgCHmd7rJtTbTl89MVKc0uR79ZZ hxUBU6m5v2UGHakSlgdbh+qmr6rdFkqRxRFhdepD7u9qvPKHdVIPp23dqkx9r7gIe+RjfkT6 aq12wQhqgGYaae5bqlg0ls1CpfZxsxJmwmO/DJ2W1hdrPhs/7ml9rv552FBQuQ7i9rddeSju lkFsXVWOsbLf45pKAWTwqZihb3nSIoBv53EH0V+1SPzDbgQk2CaQ+DR6k7eujgsI/srPpMtl ehqEQ7kzaKFACBgXBOxkheDR/kJzVJxx0a1rIFnvJGL/h5lgo8onxP5scCpotNx/bJjLkBVk XQvKDUIu9YPy0yENT58C91WWX/Nac3B5W5ZkmWHh6Lh+5eOiRSLV9MDau6ziULNNpuIA90iY 1onr0xRNO2DMyn0NzZzL+Y59WlMyATUcQdTgtmlmTcaxWeRiBRlUBLvHvgHLV94= Message-ID: <36be23f1-0bf2-b68b-efee-76d1eb729e14@pgmasters.net> Date: Tue, 19 Nov 2019 16:34:16 -0500 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:60.0) Gecko/20100101 Thunderbird/60.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk On 11/19/19 5:00 AM, Rushabh Lathia wrote: > > My colleague Suraj did testing and noticed the performance impact > with the checksums.   On further testing, he found that specifically with > sha its more of performance impact.   We have found that SHA1 adds about 3% overhead when the backup is also compressed (gzip -6), which is what most people want to do. This percentage goes down even more if the backup is being transferred over a network or to an object store such as S3. We judged that the lower collision rate of SHA1 justified the additional expense. That said, making SHA256 optional seems reasonable. We decided not to make our SHA1 checksums optional to reduce the test matrix and because parallelism largely addressed performance concerns. Regards, -- -David david@pgmasters.net