Content-Type: text/plain;
	charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.80.82.1.1\))
Subject: Re: optimize lookups in snapshot [sub]xip arrays
From: Zhang Mingli <zmlpostgres@gmail.com>
In-Reply-To: <20220713170950.GA3116318@nathanxps13>
Date: Sun, 24 Jul 2022 12:48:25 +0800
Cc: pgsql-hackers@postgresql.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <76ACEAF0-8050-41D5-B2F5-01164533BE99@gmail.com>
References: <20220713170950.GA3116318@nathanxps13>
To: Nathan Bossart <nathandbossart@gmail.com>
Archived-At: 
 <https://www.postgresql.org/message-id/76ACEAF0-8050-41D5-B2F5-01164533BE99%40gmail.com>
Precedence: bulk

Hi, all


>=20
>  		if (!snapshot->suboverflowed)
>  		{
>  			/* we have full data, so search subxip */
> -			int32		j;
> -
> -			for (j =3D 0; j < snapshot->subxcnt; j++)
> -			{
> -				if (TransactionIdEquals(xid, =
snapshot->subxip[j]))
> -					return true;
> -			}
> +			if (XidInXip(xid, snapshot->subxip, =
snapshot->subxcnt,
> +						 &snapshot->subxiph))
> +				return true;
> =20
>  			/* not there, fall through to search xip[] */
>  		}


If snaphost->suboverflowed is  false then the subxcnt must be less than =
PGPROC_MAX_CACHED_SUBXIDS which is 64 now.

And we won=E2=80=99t use hash if the xcnt is less than =
XIP_HASH_MIN_ELEMENTS which is 128 currently during discussion.

So that, subxid=E2=80=99s hash table will never be used, right?

Regards,

Zhang Mingli


> On Jul 14, 2022, at 01:09, Nathan Bossart <nathandbossart@gmail.com> =
wrote:
>=20
> Hi hackers,
>=20
> A few years ago, there was a proposal to create hash tables for long
> [sub]xip arrays in snapshots [0], but the thread seems to have fizzled =
out.
> I was curious whether this idea still showed measurable benefits, so I
> revamped the patch and ran the same test as before [1].  Here are the
> results for 60=E2=82=8Bsecond runs on an r5d.24xlarge with the data =
directory on
> the local NVMe storage:
>=20
>     writers  HEAD  patch  diff
>    ----------------------------
>     16       659   664    +1%
>     32       645   663    +3%
>     64       659   692    +5%
>     128      641   716    +12%
>     256      619   610    -1%
>     512      530   702    +32%
>     768      469   582    +24%
>     1000     367   577    +57%
>=20
> As before, the hash table approach seems to provide a decent benefit =
at
> higher client counts, so I felt it was worth reviving the idea.
>=20
> The attached patch has some key differences from the previous =
proposal.
> For example, the new patch uses simplehash instead of open-coding a =
new
> hash table.  Also, I've bumped up the threshold for creating hash =
tables to
> 128 based on the results of my testing.  The attached patch waits =
until a
> lookup of [sub]xip before generating the hash table, so we only need =
to
> allocate enough space for the current elements in the [sub]xip array, =
and
> we avoid allocating extra memory for workloads that do not need the =
hash
> tables.  I'm slightly worried about increasing the number of memory
> allocations in this code path, but the results above seemed =
encouraging on
> that front.
>=20
> Thoughts?
>=20
> [0] =
https://postgr.es/m/35960b8af917e9268881cd8df3f88320%40postgrespro.ru
> [1] =
https://postgr.es/m/057a9a95-19d2-05f0-17e2-f46ff20e9b3e%402ndquadrant.com=

>=20
> --=20
> Nathan Bossart
> Amazon Web Services: https://aws.amazon.com
> <v1-0001-Optimize-lookups-in-snapshot-transactions-in-prog.patch>