Date: Thu, 8 Aug 2024 14:28:31 -0500
From: Nathan Bossart <nathandbossart@gmail.com>
To: Andres Freund <andres@anarazel.de>
Cc: "Amonson, Paul D" <paul.d.amonson@intel.com>,
	Daniel Gustafsson <daniel@yesql.se>,
	"pgsql-hackers@lists.postgresql.org" <pgsql-hackers@lists.postgresql.org>,
	"Shankaran, Akash" <akash.shankaran@intel.com>
Subject: Re: Proposal for Updating CRC32C with AVX-512 Algorithm.
Message-ID: <ZrUcX2kq-0doNBea@nathan>
References: <BL1PR11MB530401FA7E9B1CA432CF9DC3DC192@BL1PR11MB5304.namprd11.prod.outlook.com>
 <BN0SPR01MB00084DB3E6F61E09F59533FFDCEE2@BN0SPR01MB0008.namprd11.prod.outlook.com>
 <E76B4CB6-E6C6-4F1F-ABAB-BF521B5A4F28@yesql.se>
 <BL1PR11MB530492830A59F48DB90F4E4ADCC02@BL1PR11MB5304.namprd11.prod.outlook.com>
 <20240612193746.rjeiip4hcamjedgo@awork3.anarazel.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240612193746.rjeiip4hcamjedgo@awork3.anarazel.de>
Archived-At: <https://www.postgresql.org/message-id/ZrUcX2kq-0doNBea%40nathan>
Precedence: bulk

On Wed, Jun 12, 2024 at 12:37:46PM -0700, Andres Freund wrote:
> I'm wonder if this isn't going in the wrong direction. We're using CRCs for
> something they're not well suited for in my understanding - and are paying a
> reasonably high price for it, given that even hardware accelerated CRCs aren't
> blazingly fast.

I tend to agree, especially that we should be more concerned about all
bytes after a certain point being garbage than bit flips.  (I think we
should also care about bit flips, but I hope those are much less common
than half-written WAL records.)

> With that I perhaps have established that CRC guarantees aren't useful for us.
> But not yet why we should use something else: Given that we already aren't
> relying on hard guarantees, we could instead just use a fast hash like xxh3.
> https://github.com/Cyan4973/xxHash which is fast both for large and small
> amounts of data.

Would it be out of the question to reuse the page checksum code (i.e., an
FNV-1a derivative)?  The chart in your link claims that xxh3 is
substantially faster than "FNV64", but I wonder if the latter was
vectorized.  I don't know how our CRC-32C implementations (and proposed
implementations) compare, either.

-- 
nathan