From: ilmari@ilmari.org (Dagfinn Ilmari =?utf-8?Q?Manns=C3=A5ker?=)
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: "Daniel Verite" <daniel@manitou-mail.org>,
  "movead.li\@highgo.ca" <movead.li@highgo.ca>,
  "ashutosh.bapat" <ashutosh.bapat@2ndquadrant.com>,
  "pgsql-hackers" <pgsql-hackers@lists.postgresql.org>
Subject: Re: A bug when use get_bit() function for a long bytea string
References: <84525289-3891-4368-9157-5452f53a6908@manitou-mail.org>
	<563.1585857846@sss.pgh.pa.us>
Date: Thu, 02 Apr 2020 22:56:39 +0100
In-Reply-To: <563.1585857846@sss.pgh.pa.us> (Tom Lane's message of "Thu, 02
	Apr 2020 16:04:06 -0400")
Message-ID: <87y2rdtuug.fsf@wibble.ilmari.org>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
Precedence: bulk

Tom Lane <tgl@sss.pgh.pa.us> writes:

> "Daniel Verite" <daniel@manitou-mail.org> writes:
>> These 2 tests need to allocate big chunks of contiguous memory, so they
>> might fail for lack of memory on tiny machines, and even when not failing,
>> they're pretty slow to run. Are they worth the trouble?
>
> Yeah, I'd noticed those on previous readings of the patch.  They'd almost
> certainly fail on some of our older/smaller buildfarm members, so they're
> not getting committed, even if they didn't require multiple seconds apiece
> to run (even on a machine with plenty of memory).  It's useful to have
> them for initial testing though.

Perl's test suite has a similar issue with tests for handling of huge
strings, hashes, arrays, regexes etc.  We've taken the approach of
checking the environment variable PERL_TEST_MEMORY and skipping tests
that need more than that many gigabytes.  We currently have tests that
check for values from 1 all the way up to 96 GiB.

This would be trivial to do in the Postgres TAP tests, but something
similar might feasible in the pg_regress too?

> It'd be great if there was a way to test get_bit/set_bit on large
> indexes without materializing a couple of multi-hundred-MB objects.
> Can't think of one offhand though.

For this usecase it might make sense to express the limit in megabytes,
and have a policy for how much memory tests can assume without explicit
opt-in from the developer or buildfarm animal.

- ilmari
-- 
"The surreality of the universe tends towards a maximum" -- Skud's Law
"Never formulate a law or axiom that you're not prepared to live with
 the consequences of."                              -- Skud's Meta-Law