pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: David Steele <david@pgmasters.net>
To: Rushabh Lathia <rushabh.lathia@gmail.com>
To: Robert Haas <robertmhaas@gmail.com>
To: PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Cc: Jeevan Chalke <jeevan.chalke@enterprisedb.com>
Cc: vignesh C <vignesh21@gmail.com>
Subject: Re: backup manifests
Date: Tue, 19 Nov 2019 16:34:16 -0500
Message-ID: <36be23f1-0bf2-b68b-efee-76d1eb729e14@pgmasters.net> (raw)
In-Reply-To: <CAGPqQf3DcmB6OtTpwqayH5bJgMnx2j-r4v2dFR6_=iFiqPq1Sw@mail.gmail.com>
References: <CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com>
	<4b6244ea-4094-9055-080b-7afb274bf270@pgmasters.net>
	<CA+TgmobANXUCZ1XrLJQbX95e3fJa0dRqV09Scjb2kmqt+gdkvg@mail.gmail.com>
	<b2b696b4-b11f-e954-c86a-8252a62a2e40@pgmasters.net>
	<CA+TgmobZrNYR-ATtfZiZ_k-W7tSPgvmYZmyiqumQig4R4fkzHw@mail.gmail.com>
	<146910fc-3df7-eeb4-ac5b-2f57a01f646f@pgmasters.net>
	<CA+TgmoYZDTHbSpwZtW=JDgAhwVAYvmdSrRUjOd+AYdfNNXVBDg@mail.gmail.com>
	<CALDaNm0pReK3ENbadRM=CJXNLsw4tm-MfaK_ckE7RMYEp9pgQg@mail.gmail.com>
	<CAM2+6=VHWifa=+8mtnFy3dqFvQtsEQPy9NBQEv-FbR26L22+uQ@mail.gmail.com>
	<CA+Tgmob1c3Lg4yQmMUwWh5vw67ZvB2HUJXfHGP+Opi+LDzq_bQ@mail.gmail.com>
	<CAGPqQf3DcmB6OtTpwqayH5bJgMnx2j-r4v2dFR6_=iFiqPq1Sw@mail.gmail.com>

On 11/19/19 5:00 AM, Rushabh Lathia wrote:
> 
> My colleague Suraj did testing and noticed the performance impact
> with the checksums.   On further testing, he found that specifically with
> sha its more of performance impact.  

We have found that SHA1 adds about 3% overhead when the backup is also
compressed (gzip -6), which is what most people want to do.  This
percentage goes down even more if the backup is being transferred over a
network or to an object store such as S3.

We judged that the lower collision rate of SHA1 justified the additional
expense.

That said, making SHA256 optional seems reasonable.  We decided not to
make our SHA1 checksums optional to reduce the test matrix and because
parallelism largely addressed performance concerns.

Regards,
-- 
-David
david@pgmasters.net





view thread (243+ messages)  latest in thread

Message-ID: <36be23f1-0bf2-b68b-efee-76d1eb729e14@pgmasters.net>
Permalink:  ../36be23f1-0bf2-b68b-efee-76d1eb729e14@pgmasters.net/
Also on:    postgresql.org/message-id/36be23f1-0bf2-b68b-efee-76d1eb729e14@pgmasters.net

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-hackers@postgresql.org
  Cc: david@pgmasters.net, rushabh.lathia@gmail.com, robertmhaas@gmail.com, jeevan.chalke@enterprisedb.com, vignesh21@gmail.com
  Subject: Re: backup manifests
  In-Reply-To: <36be23f1-0bf2-b68b-efee-76d1eb729e14@pgmasters.net>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox