Subject: cutting down the TODO list thread

From: John Naylor <[email-masked]>
Date: Tue, 27 Oct 2020 15:24:35 -0400

As I mentioned in [1], I've volunteered to clear out the TODO list of items
that appear to be too difficult, controversial, or otherwise not worth
doing to warrant being listed there. I'll be working a few sections at a
time, and every so often I'll have a list of proposed items for removal. If
I don't hear objections, I'll remove the items after a few days while going
through the next set.

Where there's an email thread, I've skimmed a few messages to get a sense
of the community's thoughts on it. Where easily determined, I've taken age
into account, insofar as something from 2017 is going to get much more
benefit of doubt than something from 2008. I've added after each item a
phrase that sums up the reason I believe it doesn't belong anymore.
Feedback welcome, of course, although I suspect there won't be much.

**Administration

- Have custom variables be transaction-safe
  Old and found to be difficult after attempting

- Allow custom variables to appear in pg_settings()
  Old and controversial

- Implement the SQL-standard mechanism whereby REVOKE ROLE revokes only the
privilege granted by the invoking role, and not those granted by other roles
  Old and difficult

- Prevent query cancel packets from being replayed by an attacker,
especially when using SSL
  Old and difficult

*Configuration files

- Consider normalizing fractions in postgresql.conf, perhaps using '%'
  At the time (2007), some gucs used an actual percentage.

- Add external tool to auto-tune some postgresql.conf parameters
  There are already out-of-core tools that try to do this.

- Create utility to compute accurate random_page_cost value
  Seems outdated: In the current age of SSDs and cloud environments, it's
often just set to 1.1, and there hasn't been a demand to be more accurate
than that.

- Allow synchronous_standby_names to be disabled after communication
failure with all synchronous standby servers exceeds some timeout
  Controversial

- Adjust rounding behavior for numeric GUC values
  Controversial

*Tablespaces

- Allow WAL replay of CREATE TABLESPACE to work when the directory
structure on the recovery computer is different from the original
  Thread quote: "part of the difficult, perhaps-not-worth doing impossible
problems"

- Allow per-tablespace quotas
  This seems to point to the larger problem space of disk space monitoring,
and should probably be phrased thusly, and is a much bigger project or set
of projects.

- Allow tablespaces on RAM-based partitions for temporary objects
  In the thread, what's desired is the ability to have some amount of
durability on a RAM-disk without WAL logging.

- Close race in DROP TABLESPACE on Windows
  This refers to buildfarm failures from 2014.

*Statistics Collector

- Track number of WAL files ready to be archived in pg_stat_archiver
  Thread quote: "pg_stat_archiver already has a column for
last_archived_wal and last_failed_wal, so you can already work out how many
files there must be between then and now"

*Point-In-Time Recovery

- Allow archive_mode to be changed without server restart
  Controversial and old

*Standby server mode

- Allow pg_xlogfile_name() to be used in recovery mode
  Controversial and old

- Change walsender so that it applies per-role settings
  Old and possibly obsolete

--
[1]
https://www.postgresql.org/message-id/CAFBsxsHbqMzDoGB3eAGmpcpB%2B7uae%2BLLi_G%2Bo8HMEECM9CbQcQ%40mail.gmail.com

-- 
John Naylor
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

---

From: Andres Freund <[email-masked]>
Date: Tue, 27 Oct 2020 12:36:09 -0700

Hi,

> especially when using SSL

FWIW, I don't think we should remove this. Our current solution has some
serious issues that we should address at some point.

>   Thread quote: "part of the difficult, perhaps-not-worth doing impossible

I think we ought to do something here. Mostly because the current
situation makes it impossible to test many things on a single
system. And we have a partial solution with the tablespace mapping
files.

Greetings,

Andres Freund

---

From: Bruce Momjian <[email-masked]>
Date: Tue, 27 Oct 2020 15:46:14 -0400

On Tue, Oct 27, 2020 at 03:24:35PM -0400, John Naylor wrote:
> that appear to be too difficult, controversial, or otherwise not worth doing to
> warrant being listed there. I'll be working a few sections at a time, and every
> so often I'll have a list of proposed items for removal. If I don't hear
> objections, I'll remove the items after a few days while going through the next
> set.
> 
> Where there's an email thread, I've skimmed a few messages to get a sense of
> the community's thoughts on it. Where easily determined, I've taken age into
> account, insofar as something from 2017 is going to get much more benefit of
> doubt than something from 2008. I've added after each item a phrase that sums
> up the reason I believe it doesn't belong anymore. Feedback welcome, of course,
> although I suspect there won't be much.

Thanks for working on this.  It certainly needs new eyes (not mine).  ;-)

I am fine reomving all the items below.  I am kind of disappointed we
have these _stuck_ items, but I don't see a clear way forward, so let's
just remove them and see what requests we get for them.

---------------------------------------------------------------------------

> - Prevent query cancel packets from being replayed by an attacker, especially
> when using SSL
>   Old and difficult
> 
> *Configuration files
> 
>   Seems outdated: In the current age of SSDs and cloud environments, it's often
> just set to 1.1, and there hasn't been a demand to be more accurate than that.
> 
> - Allow synchronous_standby_names to be disabled after communication failure
> with all synchronous standby servers exceeds some timeout
>   Controversial
> 
> - Allow WAL replay of CREATE TABLESPACE to work when the directory structure on
> the recovery computer is different from the original
>   This seems to point to the larger problem space of disk space monitoring, and
> should probably be phrased thusly, and is a much bigger project or set of
> projects.
> 
>   Thread quote: "pg_stat_archiver already has a column for last_archived_wal
> and last_failed_wal, so you can already work out how many files there must be
> between then and now"
> 
> *Point-In-Time Recovery
> 

-- 
  Bruce Momjian  <[email-masked]>        https://momjian.us
  EnterpriseDB                             https://enterprisedb.com

  The usefulness of a cup is in its emptiness, Bruce Lee

---

From: Bruce Momjian <[email-masked]>
Date: Tue, 27 Oct 2020 15:52:48 -0400

> just remove them and see what requests we get for them.

Do any of these limitations need to be documented before removing them
from the TODO list?

---

From: Thomas Munro <[email-masked]>
Date: Wed, 28 Oct 2020 09:00:17 +1300

> system. And we have a partial solution with the tablespace mapping

+1, we need to get something like this working so that we can write
decent replication tests.  FWIW there was another little thread on the
topic, not listed there:

https://www.postgresql.org/message-id/flat/CALfoeisEF92F5nJ-aAcuWTvF_Aogxq_1bHLem_kVfM_tHc2mfg%40mail.gmail.com

---

From: John Naylor <[email-masked]>
Date: Tue, 27 Oct 2020 16:19:47 -0400

> https://www.postgresql.org/message-id/flat/CALfoeisEF92F5nJ-aAcuWTvF_Aogxq_1bHLem_kVfM_tHc2mfg%40mail.gmail.com

Thanks, I've added this thread to the entry.

---

From: John Naylor <[email-masked]>
Date: Tue, 27 Oct 2020 16:54:24 -0400

> Do any of these limitations need to be documented before removing them

I see two areas that might use a mention:

- pg_settings not displaying custom variables
- SQL standard difference with REVOKE ROLE (I haven't looked further into
this)

---

From: Bruce Momjian <[email-masked]>
Date: Tue, 27 Oct 2020 18:05:55 -0400

On Tue, Oct 27, 2020 at 04:54:24PM -0400, John Naylor wrote:
> - SQL standard difference with REVOKE ROLE (I haven't looked further into this)

OK, thanks.  Do you want to work on a doc patch or should I?  Having it
the docs at least warns our users.

---

From: Oleksandr Shulgin <[email-masked]>
Date: Wed, 28 Oct 2020 10:55:18 +0100

On Tue, Oct 27, 2020 at 8:25 PM John Naylor <[email-masked]>
wrote:

> As I mentioned in [1], I've volunteered to clear out the TODO list of
> items that appear to be too difficult, controversial, or otherwise not
> worth doing to warrant being listed there. I'll be working a few sections
> at a time, and every so often I'll have a list of proposed items for
> removal. If I don't hear objections, I'll remove the items after a few days
> while going through the next set.
>

I'm totally on board with cleaning the list up, but how about marking as
"won't fix" (or similar) instead of actually removing the items?  That
should help to prevent the same exact items from appearing on the list
again, which they eventually would, I believe.

--
Alex

---

From: Julien Rouhaud <[email-masked]>
Date: Wed, 28 Oct 2020 18:15:15 +0800

> again, which they eventually would, I believe.

+1

---

From: Magnus Hagander <[email-masked]>
Date: Wed, 28 Oct 2020 11:52:30 +0100

On Wed, Oct 28, 2020 at 11:15 AM Julien Rouhaud <[email-masked]> wrote:
>
> On Wed, 28 Oct 2020, 17:55 Oleksandr Shulgin <[email-masked] wrote:
>>
>> On Tue, Oct 27, 2020 at 8:25 PM John Naylor <[email-masked]> wrote:
>>>
>>> As I mentioned in [1], I've volunteered to clear out the TODO list of items that appear to be too difficult, controversial, or otherwise not worth doing to warrant being listed there. I'll be working a few sections at a time, and every so often I'll have a list of proposed items for removal. If I don't hear objections, I'll remove the items after a few days while going through the next set.
>>
>>
>> I'm totally on board with cleaning the list up, but how about marking as "won't fix" (or similar) instead of actually removing the items?  That should help to prevent the same exact items from appearing on the list again, which they eventually would, I believe.
>
>
> +1

A small technical detail on the topic but if doing that, let's not
mark them as that inline -- create a separate page with those items on
it.

-- 
 Magnus Hagander
 Me: https://www.hagander.net/
 Work: https://www.redpill-linpro.com/

---

From: John Naylor <[email-masked]>
Date: Wed, 28 Oct 2020 09:26:09 -0400

> the docs at least warns our users.

I'll work on that.

---

From: John Naylor <[email-masked]>
Date: Wed, 28 Oct 2020 09:27:43 -0400

On Wed, Oct 28, 2020 at 6:52 AM Magnus Hagander <[email-masked]> wrote:

> >> I'm totally on board with cleaning the list up, but how about marking
> as "won't fix" (or similar) instead of actually removing the items?  That

How about a section on the same page at the bottom, near "features we don't
want"?

---

From: Julien Rouhaud <[email-masked]>
Date: Wed, 28 Oct 2020 22:35:15 +0800

On Wed, Oct 28, 2020 at 9:27 PM John Naylor
<[email-masked]> wrote:
> How about a section on the same page at the bottom, near "features we don't want"?

Yes, that's what I was thinking, otherwise it'll still be too easy to miss.

---

From: Magnus Hagander <[email-masked]>
Date: Wed, 28 Oct 2020 16:20:03 +0100

> Yes, that's what I was thinking, otherwise it'll still be too easy to miss.

I would personally prefer a completely seprate page, but I don't feel
strongly enough about it to push for it :) As long as it's very
clearly marked as such (and maybe the title of the page also changed
to that it's a combined todo and todon't list? :)

---

From: Andres Freund <[email-masked]>
Date: Wed, 28 Oct 2020 10:57:44 -0700

On 2020-10-28 16:20:03 +0100, Magnus Hagander wrote:
> I would personally prefer a completely seprate page

Same.

---

From: John Naylor <[email-masked]>
Date: Fri, 30 Oct 2020 11:37:59 -0400

> > I would personally prefer a completely seprate page

Ok, that's two votes for a separate page, and one for a new section on the
same page, so it looks like it's a new page. That being the case, I would
think it logical to move "features we don't want" there. As for the name,
we should probably encompass both "won't fix" bugs and features not wanted.
Maybe "past development ideas" or "not worth doing", but I'm open to better
ideas. Once that's agreed upon, I'll make a new page and migrate the items
over, minus the two that were mentioned upthread.

---

From: John Naylor <[email-masked]>
Date: Tue, 3 Nov 2020 14:06:13 -0400

> over, minus the two that were mentioned upthread.

Hearing no preference, I've created

https://wiki.postgresql.org/wiki/Not_Worth_Doing

...with links between the two. I've moved over the items I suggested
upthread, minus the two where I heard feedback otherwise (prevent replay of
query cancel packets and improve WAL replay of CREATE TABLESPACE)

I have patches for documenting some behavior we won't fix in [1][2].

I was thinking of not having the next updates during commitfest, but it
could also be argued this is a type of review, and the things here will be
returned with feedback or rejected, in a way. Ultimately, it comes down to
"when time permits".

[1]
https://www.postgresql.org/message-id/flat/CAFBsxsGsBZsG%3DcLM0Op5HFb2Ks6SzJrOc_eRO_jcKSNuqFRKnQ%40mail.gmail.com
[2]
https://www.postgresql.org/message-id/CAFBsxsEmg=[email-masked]

---

From: Bruce Momjian <[email-masked]>
Date: Tue, 10 Nov 2020 18:08:48 -0500

On Tue, Nov  3, 2020 at 02:06:13PM -0400, John Naylor wrote:
> I was thinking of not having the next updates during commitfest, but it could
> also be argued this is a type of review, and the things here will be returned
> with feedback or rejected, in a way. Ultimately, it comes down to "when time
> permits".

I don't understand what this is referring to.  Thanks for the rest of
the work.

---

From: John Naylor <[email-masked]>
Date: Wed, 11 Nov 2020 09:29:00 -0400

On Tue, Nov 10, 2020 at 7:08 PM Bruce Momjian <[email-masked]> wrote:

> could

This was awkwardly phrased, but I was concerned future proposals for
removal would be easy to miss during commitfest. At this point, I'm
thinking it isn't an issue.

---

From: John Naylor <[email-masked]>
Date: Wed, 11 Nov 2020 16:45:06 -0400

Here is the next section on data types, proposed to be moved to the "not
worth doing" page. As before, if there are any objections, do speak up.
I'll make the move in a few days.

**Datatypes

- Fix data types where equality comparison is not intuitive, e.g. box
  There is likely no way to do this without breaking applications. We
already have a big warning about this in the docs.

- Add IMMUTABLE column attribute
  Controversial

*Domains (this entire section would go)

- Allow functions defined as casts to domains to be called during casting
- Allow values to be cast to domain types
- Make domains work better with polymorphic functions
  Old and difficult

*Date/Time

- Allow TIMESTAMP WITH TIME ZONE to store the original timezone information
- Have timestamp subtraction not call justify_hours()
  Very old

- Allow a comma to denote fractional seconds in ISO-8601-compliant times
(and timestamps)
  Apparent lack of interest

*Arrays

- Add function to detect if an array is empty
- Improve handling of NULLs in arrays
  Lack of interest

*Money (this entire section would go)

- Add locale-aware MONEY type, and support multiple currencies
- MONEY dumps in a locale-specific format making it difficult to restore to
a system with a different locale
  The money datatype seems kind of obsolete anyway, and there doesn't seem
to be demand to improve it.

*Text Search

- Allow dictionaries to change the token that is passed on to later
dictionaries
- Consider a function-based API for '@@' searches
  Very old

- Improve text search error messages
  One of the gripes has been fixed already, in any case it's very old

- tsearch and tsdicts regression tests fail in Turkish locale on glibc
  Bug report that refers to locale behavior from 2009

- Improve handling of dash and plus signs in email address user names, and
perhaps improve URL parsing
  Difficult

*XML (difficult section -- plenty of bugs which should be fixed, but also
old and low interest)

- Allow XML arrays to be cast to other data types
  Very old

- Allow reliable XML operation non-UTF8 server encodings
  Difficult

- Move XSLT from contrib/xml2 to a more reasonable location
  Lack of consensus

- Improve the XSLT parameter passing API
  Lack of consensus

- xpath_table needs to be implemented/implementable to get rid of
contrib/xml2
- xpath_table is pretty broken anyway
  Unclear path forward

- better handling of XPath data types
- Improve handling of PIs and DTDs in xmlconcat()
  Zero interest

- Restructure XML and /contrib/xml2 functionality
  As discussed in the thread, it's an unrealistically large project

---

From: John Naylor <[email-masked]>
Date: Mon, 16 Nov 2020 10:15:03 -0400

> I'll make the move in a few days.

Hearing no objection, these have been moved over.

---

From: John Naylor <[email-masked]>
Date: Wed, 18 Nov 2020 14:26:46 -0400

Here are the next couple of sections with items proposed to be moved to the
"not worth doing" page. As before, if there are any objections, let me
know. I'll make the move in a few days.

Also, since 13 has been released, I'll change the explanation of Done items
to "will appear in the PostgreSQL 14 release." Also, since that wasn't
updated, it's not immediately clear to me which release the [D] marking for
"improve setting of visibility map bits for read-only and insert-only
workloads" refers to. Does anyone know which commit that is?

*Functions

- Enforce typmod for function inputs, function results and parameters for
spi_prepare'd statements called from PLs
  Lack of consensus

- Reduce memory usage of aggregates in set returning functions
  The issue and proposed patch is likely no longer an important thing to
improve in this area nowadays.

- Fix /contrib/ltree operator
  Bug from 2007 with zero followup

- Fix /contrib/btree_gist's implementation of inet indexing
  Bug from 2010 and apparent lack of interest

*Character Formatting (this entire section would be removed)

- Allow to_date() and to_timestamp() to accept localized month names
  The following attempts to pick this from the TODO list in 2008 didn't go
anywhere:

https://www.postgresql.org/message-id/flat/010401c86788%246f1ddb60%240a01a8c0%40gevmus

https://www.postgresql.org/message-id/flat/CA%2BheTbrDQ6b0Am_mk0dJEcwNxwQz%2Br%3Daz_%3DzHTva%2B5BDnfOKjA%40mail.gmail.com

- Add missing parameter handling in to_char()
  Very old

- Throw an error from to_char() instead of printing a string of "#" when a
number doesn't fit in the desired output format.
  Lack of consensus

- Fix to_number() handling for values not matching the format string
  Large amount of work for questionable benefit

*Multi-Language Support

- Add a cares-about-collation column to pg_proc, so that
unresolved-collation errors can be thrown at parse time
  Proposed while listing open items during 9.1. Doesn't seem to justify the
amount of work it would take.

- Add octet_length_server() and octet_length_client()
- Make octet_length_client() the same as octet_length()?
  Very old

- Improve UTF8 combined character handling?
  Too vague

- Fix problems with wrong runtime encoding conversion for NLS message files
  What problems? There is no discussion thread.

- More sensible support for Unicode combining characters, normal forms
  We have normalization as of PG13, so I propose to mark this Done rather
than move it.

---

From: Bruce Momjian <[email-masked]>
Date: Wed, 18 Nov 2020 13:42:47 -0500

On Wed, Nov 18, 2020 at 02:26:46PM -0400, John Naylor wrote:
> "not worth doing" page. As before, if there are any objections, let me know.
> Also, since 13 has been released, I'll change the explanation of Done items to
> "will appear in the PostgreSQL 14 release." Also, since that wasn't updated,

Yes, please do that.  I didn't go through this for PG 13 since you were
already working in this area.

> it's not immediately clear to me which release the [D] marking for "improve
> setting of visibility map bits for read-only and insert-only workloads" refers
> to. Does anyone know which commit that is?

Yes, this was fixed in PG 13 with this commit:

	commit b07642dbcd
	Author: David Rowley <[email-masked]>
	Date:   Sat Mar 28 19:20:12 2020 +1300
	
	    Trigger autovacuum based on number of INSERTs
>   What problems? There is no discussion thread.

I have no idea.  I don't think I added this one since there is no URL,
or it is very old.

>   We have normalization as of PG13, so I propose to mark this Done rather than
> move it.

Agreed.  Also done items should be remove anyway since they were done in
PG 13.

---

From: Tom Lane <[email-masked]>
Date: Wed, 18 Nov 2020 14:05:12 -0500

>   Bug from 2007 with zero followup

Actually, I believe this was fixed by 70dc4c509; at least, the
case shown in the bug report now yields "false" as expected.

>   Bug from 2010 and apparent lack of interest

This one's pretty clearly a bug.  Lack of interest or no, we
should keep it around.

> - Allow to_date() and to_timestamp() to accept localized month names

This is done too, see d67755049.

			regards, tom lane

---

From: John Naylor <[email-masked]>
Date: Thu, 19 Nov 2020 10:53:23 -0400

> already working in this area.

OK, I'll do that and remove items done during or before the PG13 cycle.

---

From: John Naylor <[email-masked]>
Date: Thu, 19 Nov 2020 10:55:11 -0400

> case shown in the bug report now yields "false" as expected.

I'll remove this item.

> This one's pretty clearly a bug.  Lack of interest or no, we

Okay.

> This is done too, see d67755049.

I'll remove this too.

---

From: John Naylor <[email-masked]>
Date: Mon, 23 Nov 2020 10:41:25 -0400

With the exception of "Fix /contrib/btree_gist's implementation of inet
indexing", all items above have been either moved over, or removed if they
were done already by PG13.

---

From: Bruce Momjian <[email-masked]>
Date: Mon, 23 Nov 2020 10:39:36 -0500

On Mon, Nov 23, 2020 at 10:41:25AM -0400, John Naylor wrote:
> indexing", all items above have been either moved over, or removed if they were
> done already by PG13.

Thanks.

---

From: John Naylor <[email-masked]>
Date: Thu, 10 Dec 2020 15:29:07 -0400

Hi,

Continuing with TODO list maintenance, first a couple things to clean up:

- Allow ALTER INDEX ... RENAME concurrently

This was in the wrong section, but it's irrelevant: The lock level was
lowered in commit 1b5d797cd4f, so I went ahead and removed this already.

- Add CORRESPONDING BY to UNION/INTERSECT/EXCEPT

The link titled "how not to write this patch" points to a web archive of
the author's description of how he implemented the rejected patch. That
doesn't seem useful, since it was...rejected. I propose to replace that
with the -hackers thread, where there is discussion of the design problem:

https://www.postgresql.org/message-id/flat/CAJZSWkWN3YwQ01C3%2Bcq0%2BeyZ1DmK%3D69_6vryrsVGMC%3D%2BfWrSZA%40mail.gmail.com

Now, for the proposed items to move to "Not Worth Doing". As before, let me
know of any objections. I plan to move these early next week:

*Views and Rules

- Allow VIEW/RULE recompilation when the underlying tables change

The entry itself says "This is both difficult and controversial." and the
linked threads confirm that.

- Make it possible to use RETURNING together with conditional DO INSTEAD
rules, such as for partitioning setups

This was from before we got native partitioning, so the stated rationale is
outdated.

*SQL Commands (this is a huge section, for now just doing the miscellany at
the top before the various subsections)

- Add a GUC variable to warn about non-standard SQL usage in queries

I don't see the reason for this, and sounds difficult anyway.

- Add NOVICE output level for helpful messages

This would only be useful if turned on, so is going to be least used where
it might help the most. It also sounds like a lot of slow menial work to
implement.

- Allow DISTINCT to work in multiple-argument aggregate calls

Tom suggested this in 2006 for the sake of orthogonality. Given the amount
of time passed, it seems not very important.

- Allow DELETE and UPDATE to be used with LIMIT and ORDER BY

Some use cases mentioned, but nearly all have some kind of workaround
already.

---

From: John Naylor <[email-masked]>
Date: Mon, 14 Dec 2020 14:12:34 -0400

> *SQL Commands

Hearing no objections, the items mentioned have been moved over.

--
John Naylor
EDB: http://www.enterprisedb.com

---

From: Bruce Momjian <[email-masked]>
Date: Thu, 17 Dec 2020 09:58:58 -0500

> Hi,

I agree with all of your analysis, but have some feedback;

> This was in the wrong section, but it's irrelevant: The lock level was lowered
> in commit 1b5d797cd4f, so I went ahead and removed this already.

Good.
> The link titled "how not to write this patch" points to a web archive of the
> author's description of how he implemented the rejected patch. That doesn't
> seem useful, since it was...rejected. I propose to replace that with the
> -hackers thread, where there is discussion of the design problem:
>   https://www.postgresql.org/message-id/flat/
> CAJZSWkWN3YwQ01C3%2Bcq0%2BeyZ1DmK%3D69_6vryrsVGMC%3D%2BfWrSZA%40mail.gmail.com
> 

Agreed.

> linked threads confirm that.

Yes, probably shouldn't be an item.
> 
> - Make it possible to use RETURNING together with conditional DO INSTEAD rules,
> such as for partitioning setups
> 

I don't think we need that anymore.

> *SQL Commands (this is a huge section, for now just doing the miscellany at the
> top before the various subsections)
> 

It is hard.

> This would only be useful if turned on, so is going to be least used where it
> might help the most. It also sounds like a lot of slow menial work to
> implement.

It is menial work, but I thought it might inspire someone to do it. 
Removal at this point seems fine.

> Tom suggested this in 2006 for the sake of orthogonality. Given the amount of
> time passed, it seems not very important.

Yes.

> Some use cases mentioned, but nearly all have some kind of workaround already.

Agreed.

---

From: John Naylor <[email-masked]>
Date: Wed, 19 May 2021 13:52:03 -0400

Hi,

I let this drop off my radar a few months ago, but I'm going to try to get
back into the habit of looking at a few items a week. As before, let me
know in the next few days if anyone has thoughts or objections.

(Optimizer / Executor)

- Consider increasing the default values of from_collapse_limit,
join_collapse_limit, and/or geqo_threshold

http://archives.postgresql.org/message-id/[email-masked]

This seems to have been rejected.

- Improve use of expression indexes for ORDER BY

http://archives.postgresql.org/pgsql-hackers/2009-08/msg01553.php

Skimming the thread, I'm not quite sure if index-only scans (not available
at the time) solves the problem, or is orthogonal to it?

- Modify the planner to better estimate caching effects

http://archives.postgresql.org/pgsql-performance/2010-11/msg00117.php

Huge discussion. This sounds like a research project, and maybe a risky one.

- Allow shared buffer cache contents to affect index cost computations

http://archives.postgresql.org/pgsql-hackers/2011-06/msg01140.php

Related to the above, but has a more specific approach in mind. The
discussion thread is not useful for getting one's head around how to think
about the problem, much less to decide if it's worth working on -- it's
mostly complaining about the review process. Independent of that, the idea
of inspecting the buffer cache seems impractical.

---

From: Bruce Momjian <[email-masked]>
Date: Thu, 20 May 2021 14:38:55 -0400

On Wed, May 19, 2021 at 01:52:03PM -0400, John Naylor wrote:
> Related to the above, but has a more specific approach in mind. The discussion
> thread is not useful for getting one's head around how to think about the
> problem, much less to decide if it's worth working on -- it's mostly
> complaining about the review process. Independent of that, the idea of
> inspecting the buffer cache seems impractical.

Yes, I think you are right about all of these.

-- 
  Bruce Momjian  <[email-masked]>        https://momjian.us
  EDB                                      https://enterprisedb.com

  If only the physical world exists, free will is an illusion.

---

From: John Naylor <[email-masked]>
Date: Mon, 28 Jun 2021 17:41:50 -0400

Here I'm just reviewing a couple items in the Sorting section:

https://wiki.postgresql.org/wiki/Todo#Sorting

- Consider whether duplicate keys should be sorted by block/offset

https://www.postgresql.org/message-id/flat/23321.1205726381%40sss.pgh.pa.us

It's moot since we started requiring tid as a tiebreaker for btree indexes,
so I'll just remove this.

- Consider being smarter about memory and external files used during sorts

https://www.postgresql.org/message-id/[email-masked]

We've had a large number of improvements in this area since 2007, and it
seems the ideas in this thread have mostly been done or superseded already.

---

From: Bruce Momjian <[email-masked]>
Date: Thu, 1 Jul 2021 21:23:32 -0400

On Mon, Jun 28, 2021 at 05:41:50PM -0400, John Naylor wrote:
> It's moot since we started requiring tid as a tiebreaker for btree indexes, so
> I'll just remove this.
> 
> We've had a large number of improvements in this area since 2007, and it seems
> the ideas in this thread have mostly been done or superseded already. 

Agreed.  Please remove them or I can do it.

---

From: Thomas Munro <[email-masked]>
Date: Fri, 2 Jul 2021 14:18:55 +1200

Spotted in the "Hashing" section:

"Use "lazy" hash tables to look up only the tuples that are actually requested"

David Rowley has knocked that one off.  He called it Result Cache.

---

From: John Naylor <[email-masked]>
Date: Thu, 1 Jul 2021 22:33:46 -0400

> David Rowley has knocked that one off.  He called it Result Cache.

Thanks, I'll take care of that one also.

---

From: John Naylor <[email-masked]>
Date: Sat, 3 Jul 2021 13:06:59 -0400

> Agreed.  Please remove them or I can do it.

Done, and also changed next release to "15".

---

From: John Naylor <[email-masked]>
Date: Wed, 8 Dec 2021 13:40:31 -0400

It's been a while, but here are a few more suggested
removals/edits/additions to the TODO list. Any objections or new
information, let me know:

- Auto-fill the free space map by scanning the buffer cache or by
checking pages written by the background writer
http://archives.postgresql.org/pgsql-hackers/2006-02/msg01125.php
https://www.postgresql.org/message-id/[email-masked]

Both these threads are from 2006, so have nothing to do with the current FSM.

- Allow concurrent inserts to use recently created pages rather than
creating new ones
http://archives.postgresql.org/pgsql-hackers/2010-05/msg00853.php

Skimming the first few messages, I believe this has been covered by
commit 719c84c1b? (Extend relations multiple blocks at a time to
improve scalability.)

- Allow VACUUM FULL and CLUSTER to update the visibility map

This topic has a current CF entry which seems to have stalled, so that
newer thread would be better to list here than the one from 2013.

- Bias FSM towards returning free space near the beginning of the heap
file, in hopes that empty pages at the end can be truncated by VACUUM
http://archives.postgresql.org/pgsql-hackers/2009-09/msg01124.php
https://www.postgresql.org/message-id/[email-masked]

I'm not sure what to think of this, but independently of that, the
second thread is actually talking about bringing back something like
the pre-9.0 vacuum full, so maybe it should be its own entry?

- Consider a more compact data representation for dead tuple locations
within VACUUM
http://archives.postgresql.org/pgsql-patches/2007-05/msg00143.php

Great, but let's link to this more recent thread instead:
https://www.postgresql.org/message-id/flat/CAD21AoAfOZvmfR0j8VmZorZjL7RhTiQdVttNuC4W-Shdc2a-AA%40mail.gmail.com

- Improve autovacuum tuning
http://www.postgresql.org/message-id/[email-masked]
http://www.postgresql.org/message-id/[email-masked]

I'm kind of on the fence about these. The title is way too broad, and
I doubt we are going to forget to keep improving this area.

It seems the first thread is really about auto-analyze thresholds, so
maybe it should be in a separate entry if we want to do anything
mentioned in the thread?

The second thread is really about autovacuum launcher scheduling.
Probably still relevant, but the thread is very long and doesn't seem
terribly helpful to someone trying to get up to speed on the issues
that are still relevant. I don't see any more recent discussion,
either. Thoughts?

---

From: Masahiko Sawada <[email-masked]>
Date: Thu, 9 Dec 2021 09:11:57 +0900

> that are still relevant. I don't see any more recent discussion,

There is another discussion on autovacuum scheduling in 2018 here:

https://www.postgresql.org/message-id/0A3221C70F24FB45833433255569204D1F8A4DC6%40G01JPEXMBYT05

Some algorithms were proposed there and I implemented a PoC patch:

https://www.postgresql.org/message-id/CAD21AoBUaSRBypA6pd9ZD%3DU-2TJCHtbyZRmrS91Nq0eVQ0B3BA%40mail.gmail.com

But not completed.

Regards,

--
Masahiko Sawada
EDB:  https://www.enterprisedb.com/

---

From: John Naylor <[email-masked]>
Date: Tue, 14 Dec 2021 10:58:56 -0400

> Both these threads are from 2006, so have nothing to do with the current FSM.

Moved to the Not Worth Doing list.

> commit 719c84c1b? (Extend relations multiple blocks at a time to

Removed.

> newer thread would be better to list here than the one from 2013.

Added.

> the pre-9.0 vacuum full, so maybe it should be its own entry?

Done.

> https://www.postgresql.org/message-id/flat/CAD21AoAfOZvmfR0j8VmZorZjL7RhTiQdVttNuC4W-Shdc2a-AA%40mail.gmail.com

Done.

> > that are still relevant. I don't see any more recent discussion,

Split into two entries.

> https://www.postgresql.org/message-id/CAD21AoBUaSRBypA6pd9ZD%3DU-2TJCHtbyZRmrS91Nq0eVQ0B3BA%40mail.gmail.com

Added, thanks!