Subject: logical decoding bug when mapped relation with toast contents is rewritten repeatedly

From: Andres Freund <[email-masked]>
Date: Thu, 13 Sep 2018 19:10:46 -0700

Hi,

(Tomas, CCing you because you IIRC mentioned encountered an issue like
this)

I just spent quite a while debugging an issue where running logical
decoding yielded a:
ERROR:  could not map filenode "base/$X/$Y" to relation OID
error.

After discarding like 30 different theories, I have found the cause:

During rewrites (i.e. VACUUM FULL / CLUSTER) of a mapped relation with a
toast table with actual live toasted tuples (pg_proc in my case and
henceforth) heap inserts with the toast data happen into the new toast
relation, triggered by:

static void
raw_heap_insert(RewriteState state, HeapTuple tup)
...
	/*
	 * If the new tuple is too big for storage or contains already toasted
	 * out-of-line attributes from some other relation, invoke the toaster.
	 *
	 * Note: below this point, heaptup is the data we actually intend to store
	 * into the relation; tup is the caller's original untoasted data.
	 */
	if (state->rs_new_rel->rd_rel->relkind == RELKIND_TOASTVALUE)
	{
		/* toast table entries should never be recursively toasted */
		Assert(!HeapTupleHasExternal(tup));
		heaptup = tup;
	}
	else if (HeapTupleHasExternal(tup) || tup->t_len > TOAST_TUPLE_THRESHOLD)
		heaptup = toast_insert_or_update(state->rs_new_rel, tup, NULL,
										 HEAP_INSERT_SKIP_FSM |
										 (state->rs_use_wal ?
										  0 : HEAP_INSERT_SKIP_WAL));
	else
		heaptup = tup;

At that point the new toast relation does *NOT* appear to be a system
catalog, it's just appears as an "independent" table.  Therefore we do
not trigger, in heap_insert():

/*
 * RelationIsLogicallyLogged
 *		True if we need to log enough information to extract the data from the
 *		WAL stream.
 *
 * We don't log information for unlogged tables (since they don't WAL log
 * anyway) and for system tables (their content is hard to make sense of, and
 * it would complicate decoding slightly for little gain). Note that we *do*
 * log information for user defined catalog tables since they presumably are
 * interesting to the user...
 */
#define RelationIsLogicallyLogged(relation) \
	(XLogLogicalInfoActive() && \
	 RelationNeedsWAL(relation) && \
	 !IsCatalogRelation(relation))

		/*
		 * For logical decoding, we need the tuple even if we're doing a full
		 * page write, so make sure it's included even if we take a full-page
		 * image. (XXX We could alternatively store a pointer into the FPW).
		 */
		if (RelationIsLogicallyLogged(relation))
		{
			xlrec.flags |= XLH_INSERT_CONTAINS_NEW_TUPLE;
			bufflags |= REGBUF_KEEP_DATA;
		}

i.e. the inserted toast tuple will be marked as
XLH_INSERT_CONTAINS_NEW_TUPLE - which it shouldn't, because it's a
system table. Which we currently do not allow do be logically decoded.

That normally ends up being harmless, because ReorderBufferCommit() has the
following check:
					if (!RelationIsLogicallyLogged(relation))
						goto change_done;

but to reach that check, we first have to map the relfilenode from the
WAL to the corresponding OID:
					reloid = RelidByRelfilenode(change->data.tp.relnode.spcNode,
												change->data.tp.relnode.relNode);

That works correctly if there's only one rewrite - the relmapper
contains the data for the new toast table.  But if there's been *two*
consecutive rewrites, the relmapper *does not* contain the intermediary
relfilenode of pg_proc.  There's no such problem for non-mapped tables,
because historic snapshots allow us to access the relevant data, but the
relmapper isn't mvcc.

Therefore the catalog-rewrite escape hatch of:
					/*
					 * Catalog tuple without data, emitted while catalog was
					 * in the process of being rewritten.
					 */
					if (reloid == InvalidOid &&
						change->data.tp.newtuple == NULL &&
						change->data.tp.oldtuple == NULL)
						goto change_done;
does not trigger and we run into:
					else if (reloid == InvalidOid)
						elog(ERROR, "could not map filenode \"%s\" to relation OID",
							 relpathperm(change->data.tp.relnode,
										 MAIN_FORKNUM));

commenting out this error / converting it into a warning makes this case
harmless, but could obviously be problematic in other scenarios.

I suspect the proper fix would be to have a new HEAP_INSERT_NO_LOGICAL
option, and specify that in raw_heap_insert() iff
RelationIsLogicallyLogged(state->rs_old_rel) or something like that.

Attached is a *prototype* patch of that approach.  Without the code
level changes the addition to test_decoding's rewrite.sql trigger the
bug, after it they're fixed.

The only reason the scenario I was debugging hit this was that there was
a cluster wide VACUUM FULL a couple times a day, and replication was
several hours behind due to slow network / receiving side.

Now I'm having a beer outside.

Greetings,

Andres Freund
diff --git a/contrib/test_decoding/expected/rewrite.out b/contrib/test_decoding/expected/rewrite.out
index 4dcd4895438..12eaab99b83 100644
--- a/contrib/test_decoding/expected/rewrite.out
+++ b/contrib/test_decoding/expected/rewrite.out
@@ -34,6 +34,407 @@ VACUUM FULL pg_opclass;
 VACUUM FULL pg_type;
 VACUUM FULL pg_index;
 VACUUM FULL pg_database;
+-- ensure there's a catalog table that is toasted.
+CREATE FUNCTION speaketh() RETURNS TEXT IMMUTABLE LANGUAGE SQL AS $f$
+SELECT text $t$
+ACT I
+SCENE I. Elsinore. A platform before the castle.
+
+    FRANCISCO at his post. Enter to him BERNARDO
+
+BERNARDO
+
+    Who's there?
+
+FRANCISCO
+
+    Nay, answer me: stand, and unfold yourself.
+
+BERNARDO
+
+    Long live the king!
+
+FRANCISCO
+
+    Bernardo?
+
+BERNARDO
+
+    He.
+
+FRANCISCO
+
+    You come most carefully upon your hour.
+
+BERNARDO
+
+    'Tis now struck twelve; get thee to bed, Francisco.
+
+FRANCISCO
+
+    For this relief much thanks: 'tis bitter cold,
+    And I am sick at heart.
+
+BERNARDO
+
+    Have you had quiet guard?
+
+FRANCISCO
+
+    Not a mouse stirring.
+
+BERNARDO
+
+    Well, good night.
+    If you do meet Horatio and Marcellus,
+    The rivals of my watch, bid them make haste.
+
+FRANCISCO
+
+    I think I hear them. Stand, ho! Who's there?
+
+    Enter HORATIO and MARCELLUS
+
+HORATIO
+
+    Friends to this ground.
+
+MARCELLUS
+
+    And liegemen to the Dane.
+
+FRANCISCO
+
+    Give you good night.
+
+MARCELLUS
+
+    O, farewell, honest soldier:
+    Who hath relieved you?
+
+FRANCISCO
+
+    Bernardo has my place.
+    Give you good night.
+
+    Exit
+
+MARCELLUS
+
+    Holla! Bernardo!
+
+BERNARDO
+
+    Say,
+    What, is Horatio there?
+
+HORATIO
+
+    A piece of him.
+
+BERNARDO
+
+    Welcome, Horatio: welcome, good Marcellus.
+
+MARCELLUS
+
+    What, has this thing appear'd again to-night?
+
+BERNARDO
+
+    I have seen nothing.
+
+MARCELLUS
+
+    Horatio says 'tis but our fantasy,
+    And will not let belief take hold of him
+    Touching this dreaded sight, twice seen of us:
+    Therefore I have entreated him along
+    With us to watch the minutes of this night;
+    That if again this apparition come,
+    He may approve our eyes and speak to it.
+
+HORATIO
+
+    Tush, tush, 'twill not appear.
+
+BERNARDO
+
+    Sit down awhile;
+    And let us once again assail your ears,
+    That are so fortified against our story
+    What we have two nights seen.
+
+HORATIO
+
+    Well, sit we down,
+    And let us hear Bernardo speak of this.
+
+BERNARDO
+
+    Last night of all,
+    When yond same star that's westward from the pole
+    Had made his course to illume that part of heaven
+    Where now it burns, Marcellus and myself,
+    The bell then beating one,--
+
+    Enter Ghost
+
+MARCELLUS
+
+    Peace, break thee off; look, where it comes again!
+
+BERNARDO
+
+    In the same figure, like the king that's dead.
+
+MARCELLUS
+
+    Thou art a scholar; speak to it, Horatio.
+
+BERNARDO
+
+    Looks it not like the king? mark it, Horatio.
+
+HORATIO
+
+    Most like: it harrows me with fear and wonder.
+
+BERNARDO
+
+    It would be spoke to.
+
+MARCELLUS
+
+    Question it, Horatio.
+
+HORATIO
+
+    What art thou that usurp'st this time of night,
+    Together with that fair and warlike form
+    In which the majesty of buried Denmark
+    Did sometimes march? by heaven I charge thee, speak!
+
+MARCELLUS
+
+    It is offended.
+
+BERNARDO
+
+    See, it stalks away!
+
+HORATIO
+
+    Stay! speak, speak! I charge thee, speak!
+
+    Exit Ghost
+
+MARCELLUS
+
+    'Tis gone, and will not answer.
+
+BERNARDO
+
+    How now, Horatio! you tremble and look pale:
+    Is not this something more than fantasy?
+    What think you on't?
+
+HORATIO
+
+    Before my God, I might not this believe
+    Without the sensible and true avouch
+    Of mine own eyes.
+
+MARCELLUS
+
+    Is it not like the king?
+
+HORATIO
+
+    As thou art to thyself:
+    Such was the very armour he had on
+    When he the ambitious Norway combated;
+    So frown'd he once, when, in an angry parle,
+    He smote the sledded Polacks on the ice.
+    'Tis strange.
+
+MARCELLUS
+
+    Thus twice before, and jump at this dead hour,
+    With martial stalk hath he gone by our watch.
+
+HORATIO
+
+    In what particular thought to work I know not;
+    But in the gross and scope of my opinion,
+    This bodes some strange eruption to our state.
+
+MARCELLUS
+
+    Good now, sit down, and tell me, he that knows,
+    Why this same strict and most observant watch
+    So nightly toils the subject of the land,
+    And why such daily cast of brazen cannon,
+    And foreign mart for implements of war;
+    Why such impress of shipwrights, whose sore task
+    Does not divide the Sunday from the week;
+    What might be toward, that this sweaty haste
+    Doth make the night joint-labourer with the day:
+    Who is't that can inform me?
+
+HORATIO
+
+    That can I;
+    At least, the whisper goes so. Our last king,
+    Whose image even but now appear'd to us,
+    Was, as you know, by Fortinbras of Norway,
+    Thereto prick'd on by a most emulate pride,
+    Dared to the combat; in which our valiant Hamlet--
+    For so this side of our known world esteem'd him--
+    Did slay this Fortinbras; who by a seal'd compact,
+    Well ratified by law and heraldry,
+    Did forfeit, with his life, all those his lands
+    Which he stood seized of, to the conqueror:
+    Against the which, a moiety competent
+    Was gaged by our king; which had return'd
+    To the inheritance of Fortinbras,
+    Had he been vanquisher; as, by the same covenant,
+    And carriage of the article design'd,
+    His fell to Hamlet. Now, sir, young Fortinbras,
+    Of unimproved mettle hot and full,
+    Hath in the skirts of Norway here and there
+    Shark'd up a list of lawless resolutes,
+    For food and diet, to some enterprise
+    That hath a stomach in't; which is no other--
+    As it doth well appear unto our state--
+    But to recover of us, by strong hand
+    And terms compulsatory, those foresaid lands
+    So by his father lost: and this, I take it,
+    Is the main motive of our preparations,
+    The source of this our watch and the chief head
+    Of this post-haste and romage in the land.
+
+BERNARDO
+
+    I think it be no other but e'en so:
+    Well may it sort that this portentous figure
+    Comes armed through our watch; so like the king
+    That was and is the question of these wars.
+
+HORATIO
+
+    A mote it is to trouble the mind's eye.
+    In the most high and palmy state of Rome,
+    A little ere the mightiest Julius fell,
+    The graves stood tenantless and the sheeted dead
+    Did squeak and gibber in the Roman streets:
+    As stars with trains of fire and dews of blood,
+    Disasters in the sun; and the moist star
+    Upon whose influence Neptune's empire stands
+    Was sick almost to doomsday with eclipse:
+    And even the like precurse of fierce events,
+    As harbingers preceding still the fates
+    And prologue to the omen coming on,
+    Have heaven and earth together demonstrated
+    Unto our climatures and countrymen.--
+    But soft, behold! lo, where it comes again!
+
+    Re-enter Ghost
+    I'll cross it, though it blast me. Stay, illusion!
+    If thou hast any sound, or use of voice,
+    Speak to me:
+    If there be any good thing to be done,
+    That may to thee do ease and grace to me,
+    Speak to me:
+
+    Cock crows
+    If thou art privy to thy country's fate,
+    Which, happily, foreknowing may avoid, O, speak!
+    Or if thou hast uphoarded in thy life
+    Extorted treasure in the womb of earth,
+    For which, they say, you spirits oft walk in death,
+    Speak of it: stay, and speak! Stop it, Marcellus.
+
+MARCELLUS
+
+    Shall I strike at it with my partisan?
+
+HORATIO
+
+    Do, if it will not stand.
+
+BERNARDO
+
+    'Tis here!
+
+HORATIO
+
+    'Tis here!
+
+MARCELLUS
+
+    'Tis gone!
+
+    Exit Ghost
+    We do it wrong, being so majestical,
+    To offer it the show of violence;
+    For it is, as the air, invulnerable,
+    And our vain blows malicious mockery.
+
+BERNARDO
+
+    It was about to speak, when the cock crew.
+
+HORATIO
+
+    And then it started like a guilty thing
+    Upon a fearful summons. I have heard,
+    The cock, that is the trumpet to the morn,
+    Doth with his lofty and shrill-sounding throat
+    Awake the god of day; and, at his warning,
+    Whether in sea or fire, in earth or air,
+    The extravagant and erring spirit hies
+    To his confine: and of the truth herein
+    This present object made probation.
+
+MARCELLUS
+
+    It faded on the crowing of the cock.
+    Some say that ever 'gainst that season comes
+    Wherein our Saviour's birth is celebrated,
+    The bird of dawning singeth all night long:
+    And then, they say, no spirit dares stir abroad;
+    The nights are wholesome; then no planets strike,
+    No fairy takes, nor witch hath power to charm,
+    So hallow'd and so gracious is the time.
+
+HORATIO
+
+    So have I heard and do in part believe it.
+    But, look, the morn, in russet mantle clad,
+    Walks o'er the dew of yon high eastward hill:
+    Break we our watch up; and by my advice,
+    Let us impart what we have seen to-night
+    Unto young Hamlet; for, upon my life,
+    This spirit, dumb to us, will speak to him.
+    Do you consent we shall acquaint him with it,
+    As needful in our loves, fitting our duty?
+
+MARCELLUS
+
+    Let's do't, I pray; and I this morning know
+    Where we shall find him most conveniently.
+
+    Exeunt
+$t$
+$f$;
+SELECT pg_relation_size((SELECT reltoastrelid FROM pg_class WHERE oid = 'pg_proc'::regclass)) > 0;
+ ?column? 
+----------
+ t
+(1 row)
+
+VACUUM FULL pg_proc;
 -- repeated rewrites that fail
 BEGIN;
 CLUSTER pg_class USING pg_class_oid_index;
@@ -76,6 +477,13 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
  COMMIT
 (15 rows)
 
+VACUUM FULL pg_proc;
+VACUUM FULL pg_proc;
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ data 
+------
+(0 rows)
+
 SELECT pg_drop_replication_slot('regression_slot');
  pg_drop_replication_slot 
 --------------------------
@@ -83,3 +491,4 @@ SELECT pg_drop_replication_slot('regression_slot');
 (1 row)
 
 DROP TABLE IF EXISTS replication_example;
+DROP FUNCTION speaketh();
diff --git a/contrib/test_decoding/sql/rewrite.sql b/contrib/test_decoding/sql/rewrite.sql
index 8a7329423de..b3ea848cad0 100644
--- a/contrib/test_decoding/sql/rewrite.sql
+++ b/contrib/test_decoding/sql/rewrite.sql
@@ -28,6 +28,404 @@ VACUUM FULL pg_type;
 VACUUM FULL pg_index;
 VACUUM FULL pg_database;
 
+-- ensure there's a catalog table that is toasted.
+CREATE FUNCTION speaketh() RETURNS TEXT IMMUTABLE LANGUAGE SQL AS $f$
+SELECT text $t$
+ACT I
+SCENE I. Elsinore. A platform before the castle.
+
+    FRANCISCO at his post. Enter to him BERNARDO
+
+BERNARDO
+
+    Who's there?
+
+FRANCISCO
+
+    Nay, answer me: stand, and unfold yourself.
+
+BERNARDO
+
+    Long live the king!
+
+FRANCISCO
+
+    Bernardo?
+
+BERNARDO
+
+    He.
+
+FRANCISCO
+
+    You come most carefully upon your hour.
+
+BERNARDO
+
+    'Tis now struck twelve; get thee to bed, Francisco.
+
+FRANCISCO
+
+    For this relief much thanks: 'tis bitter cold,
+    And I am sick at heart.
+
+BERNARDO
+
+    Have you had quiet guard?
+
+FRANCISCO
+
+    Not a mouse stirring.
+
+BERNARDO
+
+    Well, good night.
+    If you do meet Horatio and Marcellus,
+    The rivals of my watch, bid them make haste.
+
+FRANCISCO
+
+    I think I hear them. Stand, ho! Who's there?
+
+    Enter HORATIO and MARCELLUS
+
+HORATIO
+
+    Friends to this ground.
+
+MARCELLUS
+
+    And liegemen to the Dane.
+
+FRANCISCO
+
+    Give you good night.
+
+MARCELLUS
+
+    O, farewell, honest soldier:
+    Who hath relieved you?
+
+FRANCISCO
+
+    Bernardo has my place.
+    Give you good night.
+
+    Exit
+
+MARCELLUS
+
+    Holla! Bernardo!
+
+BERNARDO
+
+    Say,
+    What, is Horatio there?
+
+HORATIO
+
+    A piece of him.
+
+BERNARDO
+
+    Welcome, Horatio: welcome, good Marcellus.
+
+MARCELLUS
+
+    What, has this thing appear'd again to-night?
+
+BERNARDO
+
+    I have seen nothing.
+
+MARCELLUS
+
+    Horatio says 'tis but our fantasy,
+    And will not let belief take hold of him
+    Touching this dreaded sight, twice seen of us:
+    Therefore I have entreated him along
+    With us to watch the minutes of this night;
+    That if again this apparition come,
+    He may approve our eyes and speak to it.
+
+HORATIO
+
+    Tush, tush, 'twill not appear.
+
+BERNARDO
+
+    Sit down awhile;
+    And let us once again assail your ears,
+    That are so fortified against our story
+    What we have two nights seen.
+
+HORATIO
+
+    Well, sit we down,
+    And let us hear Bernardo speak of this.
+
+BERNARDO
+
+    Last night of all,
+    When yond same star that's westward from the pole
+    Had made his course to illume that part of heaven
+    Where now it burns, Marcellus and myself,
+    The bell then beating one,--
+
+    Enter Ghost
+
+MARCELLUS
+
+    Peace, break thee off; look, where it comes again!
+
+BERNARDO
+
+    In the same figure, like the king that's dead.
+
+MARCELLUS
+
+    Thou art a scholar; speak to it, Horatio.
+
+BERNARDO
+
+    Looks it not like the king? mark it, Horatio.
+
+HORATIO
+
+    Most like: it harrows me with fear and wonder.
+
+BERNARDO
+
+    It would be spoke to.
+
+MARCELLUS
+
+    Question it, Horatio.
+
+HORATIO
+
+    What art thou that usurp'st this time of night,
+    Together with that fair and warlike form
+    In which the majesty of buried Denmark
+    Did sometimes march? by heaven I charge thee, speak!
+
+MARCELLUS
+
+    It is offended.
+
+BERNARDO
+
+    See, it stalks away!
+
+HORATIO
+
+    Stay! speak, speak! I charge thee, speak!
+
+    Exit Ghost
+
+MARCELLUS
+
+    'Tis gone, and will not answer.
+
+BERNARDO
+
+    How now, Horatio! you tremble and look pale:
+    Is not this something more than fantasy?
+    What think you on't?
+
+HORATIO
+
+    Before my God, I might not this believe
+    Without the sensible and true avouch
+    Of mine own eyes.
+
+MARCELLUS
+
+    Is it not like the king?
+
+HORATIO
+
+    As thou art to thyself:
+    Such was the very armour he had on
+    When he the ambitious Norway combated;
+    So frown'd he once, when, in an angry parle,
+    He smote the sledded Polacks on the ice.
+    'Tis strange.
+
+MARCELLUS
+
+    Thus twice before, and jump at this dead hour,
+    With martial stalk hath he gone by our watch.
+
+HORATIO
+
+    In what particular thought to work I know not;
+    But in the gross and scope of my opinion,
+    This bodes some strange eruption to our state.
+
+MARCELLUS
+
+    Good now, sit down, and tell me, he that knows,
+    Why this same strict and most observant watch
+    So nightly toils the subject of the land,
+    And why such daily cast of brazen cannon,
+    And foreign mart for implements of war;
+    Why such impress of shipwrights, whose sore task
+    Does not divide the Sunday from the week;
+    What might be toward, that this sweaty haste
+    Doth make the night joint-labourer with the day:
+    Who is't that can inform me?
+
+HORATIO
+
+    That can I;
+    At least, the whisper goes so. Our last king,
+    Whose image even but now appear'd to us,
+    Was, as you know, by Fortinbras of Norway,
+    Thereto prick'd on by a most emulate pride,
+    Dared to the combat; in which our valiant Hamlet--
+    For so this side of our known world esteem'd him--
+    Did slay this Fortinbras; who by a seal'd compact,
+    Well ratified by law and heraldry,
+    Did forfeit, with his life, all those his lands
+    Which he stood seized of, to the conqueror:
+    Against the which, a moiety competent
+    Was gaged by our king; which had return'd
+    To the inheritance of Fortinbras,
+    Had he been vanquisher; as, by the same covenant,
+    And carriage of the article design'd,
+    His fell to Hamlet. Now, sir, young Fortinbras,
+    Of unimproved mettle hot and full,
+    Hath in the skirts of Norway here and there
+    Shark'd up a list of lawless resolutes,
+    For food and diet, to some enterprise
+    That hath a stomach in't; which is no other--
+    As it doth well appear unto our state--
+    But to recover of us, by strong hand
+    And terms compulsatory, those foresaid lands
+    So by his father lost: and this, I take it,
+    Is the main motive of our preparations,
+    The source of this our watch and the chief head
+    Of this post-haste and romage in the land.
+
+BERNARDO
+
+    I think it be no other but e'en so:
+    Well may it sort that this portentous figure
+    Comes armed through our watch; so like the king
+    That was and is the question of these wars.
+
+HORATIO
+
+    A mote it is to trouble the mind's eye.
+    In the most high and palmy state of Rome,
+    A little ere the mightiest Julius fell,
+    The graves stood tenantless and the sheeted dead
+    Did squeak and gibber in the Roman streets:
+    As stars with trains of fire and dews of blood,
+    Disasters in the sun; and the moist star
+    Upon whose influence Neptune's empire stands
+    Was sick almost to doomsday with eclipse:
+    And even the like precurse of fierce events,
+    As harbingers preceding still the fates
+    And prologue to the omen coming on,
+    Have heaven and earth together demonstrated
+    Unto our climatures and countrymen.--
+    But soft, behold! lo, where it comes again!
+
+    Re-enter Ghost
+    I'll cross it, though it blast me. Stay, illusion!
+    If thou hast any sound, or use of voice,
+    Speak to me:
+    If there be any good thing to be done,
+    That may to thee do ease and grace to me,
+    Speak to me:
+
+    Cock crows
+    If thou art privy to thy country's fate,
+    Which, happily, foreknowing may avoid, O, speak!
+    Or if thou hast uphoarded in thy life
+    Extorted treasure in the womb of earth,
+    For which, they say, you spirits oft walk in death,
+    Speak of it: stay, and speak! Stop it, Marcellus.
+
+MARCELLUS
+
+    Shall I strike at it with my partisan?
+
+HORATIO
+
+    Do, if it will not stand.
+
+BERNARDO
+
+    'Tis here!
+
+HORATIO
+
+    'Tis here!
+
+MARCELLUS
+
+    'Tis gone!
+
+    Exit Ghost
+    We do it wrong, being so majestical,
+    To offer it the show of violence;
+    For it is, as the air, invulnerable,
+    And our vain blows malicious mockery.
+
+BERNARDO
+
+    It was about to speak, when the cock crew.
+
+HORATIO
+
+    And then it started like a guilty thing
+    Upon a fearful summons. I have heard,
+    The cock, that is the trumpet to the morn,
+    Doth with his lofty and shrill-sounding throat
+    Awake the god of day; and, at his warning,
+    Whether in sea or fire, in earth or air,
+    The extravagant and erring spirit hies
+    To his confine: and of the truth herein
+    This present object made probation.
+
+MARCELLUS
+
+    It faded on the crowing of the cock.
+    Some say that ever 'gainst that season comes
+    Wherein our Saviour's birth is celebrated,
+    The bird of dawning singeth all night long:
+    And then, they say, no spirit dares stir abroad;
+    The nights are wholesome; then no planets strike,
+    No fairy takes, nor witch hath power to charm,
+    So hallow'd and so gracious is the time.
+
+HORATIO
+
+    So have I heard and do in part believe it.
+    But, look, the morn, in russet mantle clad,
+    Walks o'er the dew of yon high eastward hill:
+    Break we our watch up; and by my advice,
+    Let us impart what we have seen to-night
+    Unto young Hamlet; for, upon my life,
+    This spirit, dumb to us, will speak to him.
+    Do you consent we shall acquaint him with it,
+    As needful in our loves, fitting our duty?
+
+MARCELLUS
+
+    Let's do't, I pray; and I this morning know
+    Where we shall find him most conveniently.
+
+    Exeunt
+$t$
+$f$;
+SELECT pg_relation_size((SELECT reltoastrelid FROM pg_class WHERE oid = 'pg_proc'::regclass)) > 0;
+
+VACUUM FULL pg_proc;
+
 -- repeated rewrites that fail
 BEGIN;
 CLUSTER pg_class USING pg_class_oid_index;
@@ -57,6 +455,11 @@ COMMIT;
 CHECKPOINT;
 
 SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
-SELECT pg_drop_replication_slot('regression_slot');
+VACUUM FULL pg_proc;
+VACUUM FULL pg_proc;
 
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+
+SELECT pg_drop_replication_slot('regression_slot');
 DROP TABLE IF EXISTS replication_example;
+DROP FUNCTION speaketh();
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 56f1d82f962..f971705507e 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -2551,7 +2551,7 @@ heap_insert(Relation relation, HeapTuple tup, CommandId cid,
 		 * page write, so make sure it's included even if we take a full-page
 		 * image. (XXX We could alternatively store a pointer into the FPW).
 		 */
-		if (RelationIsLogicallyLogged(relation))
+		if (RelationIsLogicallyLogged(relation) && !(options & HEAP_INSERT_NO_LOGICAL))
 		{
 			xlrec.flags |= XLH_INSERT_CONTAINS_NEW_TUPLE;
 			bufflags |= REGBUF_KEEP_DATA;
diff --git a/src/backend/access/heap/rewriteheap.c b/src/backend/access/heap/rewriteheap.c
index 85f92973c95..6bae129f6a6 100644
--- a/src/backend/access/heap/rewriteheap.c
+++ b/src/backend/access/heap/rewriteheap.c
@@ -652,10 +652,18 @@ raw_heap_insert(RewriteState state, HeapTuple tup)
 		heaptup = tup;
 	}
 	else if (HeapTupleHasExternal(tup) || tup->t_len > TOAST_TUPLE_THRESHOLD)
+	{
+		int options = HEAP_INSERT_SKIP_FSM;
+
+		if (!state->rs_use_wal)
+			options |= HEAP_INSERT_SKIP_WAL;
+
+		if (!RelationIsLogicallyLogged(state->rs_old_rel))
+			options |= HEAP_INSERT_NO_LOGICAL;
+
 		heaptup = toast_insert_or_update(state->rs_new_rel, tup, NULL,
-										 HEAP_INSERT_SKIP_FSM |
-										 (state->rs_use_wal ?
-										  0 : HEAP_INSERT_SKIP_WAL));
+										 options);
+	}
 	else
 		heaptup = tup;
 
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index ca5cad7497f..40e153f71ad 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -29,6 +29,7 @@
 #define HEAP_INSERT_SKIP_FSM	0x0002
 #define HEAP_INSERT_FROZEN		0x0004
 #define HEAP_INSERT_SPECULATIVE 0x0008
+#define HEAP_INSERT_NO_LOGICAL	0x0010
 
 typedef struct BulkInsertStateData *BulkInsertState;

---

From: Tomas Vondra <[email-masked]>
Date: Fri, 14 Sep 2018 16:13:46 +0200

Hi,

> (Tomas, CCing you because you IIRC mentioned encountered an issue like

I might have mentioned an issue with this symptom recently, but that
turned out to be already fixed by da10d6a8a9 (before the minor version
with that fix got released).

> After discarding like 30 different theories, I have found the cause:

Yeah. These issues are not exactly trivial to investigate ;-)

> not trigger, in heap_insert():

Hmm, can't we change that? Recognizing the new TOAST table as a catalog
would fix the issue, no?

> harmless, but could obviously be problematic in other scenarios.

Yeah, that seems like a bad idea. That error already caught a couple of
bugs (including da10d6a8a9 and this one), and I have a hunch those are
not the last ones.

> Now I'm having a beer outside.

After discarding 30 theories? Have two.

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

---

From: Andres Freund <[email-masked]>
Date: Thu, 20 Sep 2018 19:41:24 -0700

Hi,

> bugs (including da10d6a8a9 and this one), and I have a hunch those are

One problem with this is that that means upgrading won't fix an existing
instance of the problem, but turning the ERROR into a WARNING would. I
personally think that's *NOT* enough justification for relaxing the
error, given that recreating the slot would fix the issue, but I see how
other people can reasonably differ.  I can't really see a reasonably
complex approach that solves the issue in a "cake but have it too"
way...

> After discarding 30 theories? Have two.

I will neither confirm nor deny. ;)

Greetings,

Andres Freund