Message-ID: <865a4ff239b6824fd5bbdd4a972390d1e95144b4.camel@gmail.com>
Subject: Re: Progress report of CREATE INDEX for nested partitioned tables
From: Ilya Gladyshev <ilya.v.gladyshev@gmail.com>
To: Justin Pryzby <pryzby@telsasoft.com>
Cc: pgsql-hackers@lists.postgresql.org
Date: Mon, 09 Jan 2023 12:44:22 +0400
In-Reply-To: <20230108164849.GA21731@telsasoft.com>
References: <039564d234fc3d014c555a7ee98be69a9e724836.camel@gmail.com>
	 <20221217143002.GR1153@telsasoft.com>
	 <20230108164849.GA21731@telsasoft.com>
Content-Type: multipart/mixed; boundary="=-HKiUwXiqzN+WsiSp8e6f"
User-Agent: Evolution 3.38.3-1+deb11u1 
MIME-Version: 1.0
Archived-At: 
 <https://www.postgresql.org/message-id/865a4ff239b6824fd5bbdd4a972390d1e95144b4.camel%40gmail.com>
Precedence: bulk


--=-HKiUwXiqzN+WsiSp8e6f
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit

On Sun, 2023-01-08 at 10:48 -0600, Justin Pryzby wrote:
> On Sat, Dec 17, 2022 at 08:30:02AM -0600, Justin Pryzby wrote:
> > We have the common problem of too many patches.
> > 
> > https://www.postgresql.org/message-id/a15f904a70924ffa4ca25c3c744cff31e0e6e143.camel%40gmail.com
> > This changes the progress reporting to show indirect children as
> > "total", and adds a global variable to track recursion into
> > DefineIndex(), allowing it to be incremented without the value
> > being
> > lost to the caller.
> > 
> > https://www.postgresql.org/message-id/20221211063334.GB27893%40telsasoft.com
> > This also counts indirect children, but only increments the
> > progress
> > reporting in the parent.  This has the disadvantage that when
> > intermediate partitions are in use, the done_partitions counter
> > will
> > "jump" from (say) 20 to 30 without ever hitting 21-29.
> > 
> > https://www.postgresql.org/message-id/20221213044331.GJ27893%40telsasoft.com
> > This has two alternate patches:
> > - One patch changes to only update progress reporting of *direct*
> >   children.  This is minimal, but discourages any future plan to
> > track
> >   progress involving intermediate partitions with finer
> > granularity.
> > - A alternative patch adds IndexStmt.nparts_done, and allows
> > reporting
> >   fine-grained progress involving intermediate partitions.
> > 
> > https://www.postgresql.org/message-id/flat/039564d234fc3d014c555a7ee98be69a9e724836.camel@gmail.com
> > This also reports progress of intermediate children.  The first
> > patch
> > does it by adding an argument to DefineIndex() (which isn't okay to
> > backpatch).  And an alternate patch does it by adding to IndexStmt.
> > 
> > @committers: Is it okay to add nparts_done to IndexStmt ?
> 
> Any hint about this ?
> 
> This should be resolved before the "CIC on partitioned tables" patch,
> which I think is otherwise done.

I suggest that we move on with the IndexStmt patch and see what the
committers have to say about it. I have brushed the patch up a bit,
fixing TODOs and adding docs as per our discussion above.


--=-HKiUwXiqzN+WsiSp8e6f
Content-Disposition: attachment;
 filename="v2-0001-parts_done-via-IndexStmt.patch"
Content-Type: text/x-patch; name="v2-0001-parts_done-via-IndexStmt.patch";
	charset="UTF-8"
Content-Transfer-Encoding: base64

RnJvbSA0OTBkOGFmYTdjYjk1MmU1YjM5NDdkODFkODViZjk2OTA0OTllOGEzIE1vbiBTZXAgMTcg
MDA6MDA6MDAgMjAwMQpGcm9tOiBJbHlhIEdsYWR5c2hldiA8aWx5YS52LmdsYWR5c2hldkBnbWFp
bC5jb20+CkRhdGU6IFR1ZSwgMTMgRGVjIDIwMjIgMjI6MDQ6NDUgKzA0MDAKU3ViamVjdDogW1BB
VENIIHYyXSBwYXJ0c19kb25lIHZpYSBJbmRleFN0bXQKCi0tLQogZG9jL3NyYy9zZ21sL21vbml0
b3Jpbmcuc2dtbCAgICAgICB8ICA0ICstLQogc3JjL2JhY2tlbmQvYm9vdHN0cmFwL2Jvb3RwYXJz
ZS55ICB8ICAyICsrCiBzcmMvYmFja2VuZC9jb21tYW5kcy9pbmRleGNtZHMuYyAgIHwgNTcgKysr
KysrKysrKysrKysrKysrKysrKysrKysrLS0tCiBzcmMvYmFja2VuZC9wYXJzZXIvcGFyc2VfdXRp
bGNtZC5jIHwgIDIgKysKIHNyYy9pbmNsdWRlL25vZGVzL3BhcnNlbm9kZXMuaCAgICAgfCAgMSAr
CiA1IGZpbGVzIGNoYW5nZWQsIDU5IGluc2VydGlvbnMoKyksIDcgZGVsZXRpb25zKC0pCgpkaWZm
IC0tZ2l0IGEvZG9jL3NyYy9zZ21sL21vbml0b3Jpbmcuc2dtbCBiL2RvYy9zcmMvc2dtbC9tb25p
dG9yaW5nLnNnbWwKaW5kZXggY2YyMjBjM2JjYi4uMmMzMmE5MjM2NCAxMDA2NDQKLS0tIGEvZG9j
L3NyYy9zZ21sL21vbml0b3Jpbmcuc2dtbAorKysgYi9kb2Mvc3JjL3NnbWwvbW9uaXRvcmluZy5z
Z21sCkBAIC02NTY1LDcgKzY1NjUsNyBAQCBGUk9NIHBnX3N0YXRfZ2V0X2JhY2tlbmRfaWRzZXQo
KSBBUyBiYWNrZW5kaWQ7CiAgICAgICA8L3BhcmE+CiAgICAgICA8cGFyYT4KICAgICAgICBXaGVu
IGNyZWF0aW5nIGFuIGluZGV4IG9uIGEgcGFydGl0aW9uZWQgdGFibGUsIHRoaXMgY29sdW1uIGlz
IHNldCB0bwotICAgICAgIHRoZSB0b3RhbCBudW1iZXIgb2YgcGFydGl0aW9ucyBvbiB3aGljaCB0
aGUgaW5kZXggaXMgdG8gYmUgY3JlYXRlZC4KKyAgICAgICB0aGUgdG90YWwgbnVtYmVyIG9mIGxl
YWYgcGFydGl0aW9ucyBvbiB3aGljaCB0aGUgaW5kZXggaXMgdG8gYmUgY3JlYXRlZCBvciBhdHRh
Y2hlZC4KICAgICAgICBUaGlzIGZpZWxkIGlzIDxsaXRlcmFsPjA8L2xpdGVyYWw+IGR1cmluZyBh
IDxsaXRlcmFsPlJFSU5ERVg8L2xpdGVyYWw+LgogICAgICAgPC9wYXJhPjwvZW50cnk+CiAgICAg
IDwvcm93PgpAQCAtNjU3Niw3ICs2NTc2LDcgQEAgRlJPTSBwZ19zdGF0X2dldF9iYWNrZW5kX2lk
c2V0KCkgQVMgYmFja2VuZGlkOwogICAgICAgPC9wYXJhPgogICAgICAgPHBhcmE+CiAgICAgICAg
V2hlbiBjcmVhdGluZyBhbiBpbmRleCBvbiBhIHBhcnRpdGlvbmVkIHRhYmxlLCB0aGlzIGNvbHVt
biBpcyBzZXQgdG8KLSAgICAgICB0aGUgbnVtYmVyIG9mIHBhcnRpdGlvbnMgb24gd2hpY2ggdGhl
IGluZGV4IGhhcyBiZWVuIGNyZWF0ZWQuCisgICAgICAgdGhlIG51bWJlciBvZiBsZWFmIHBhcnRp
dGlvbnMgb24gd2hpY2ggdGhlIGluZGV4IGhhcyBiZWVuIGNyZWF0ZWQgb3IgYXR0YWNoZWQuCiAg
ICAgICAgVGhpcyBmaWVsZCBpcyA8bGl0ZXJhbD4wPC9saXRlcmFsPiBkdXJpbmcgYSA8bGl0ZXJh
bD5SRUlOREVYPC9saXRlcmFsPi4KICAgICAgIDwvcGFyYT48L2VudHJ5PgogICAgICA8L3Jvdz4K
ZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5kL2Jvb3RzdHJhcC9ib290cGFyc2UueSBiL3NyYy9iYWNr
ZW5kL2Jvb3RzdHJhcC9ib290cGFyc2UueQppbmRleCA4NjgwNGJiNTk4Li5kYWQ5YzM4ZDkwIDEw
MDY0NAotLS0gYS9zcmMvYmFja2VuZC9ib290c3RyYXAvYm9vdHBhcnNlLnkKKysrIGIvc3JjL2Jh
Y2tlbmQvYm9vdHN0cmFwL2Jvb3RwYXJzZS55CkBAIC0yOTYsNiArMjk2LDcgQEAgQm9vdF9EZWNs
YXJlSW5kZXhTdG10OgogCQkJCQlzdG10LT5jb25jdXJyZW50ID0gZmFsc2U7CiAJCQkJCXN0bXQt
PmlmX25vdF9leGlzdHMgPSBmYWxzZTsKIAkJCQkJc3RtdC0+cmVzZXRfZGVmYXVsdF90YmxzcGMg
PSBmYWxzZTsKKwkJCQkJc3RtdC0+cGFydHNfZG9uZSA9IC0xOwogCiAJCQkJCS8qIGxvY2tzIGFu
ZCByYWNlcyBuZWVkIG5vdCBjb25jZXJuIHVzIGluIGJvb3RzdHJhcCBtb2RlICovCiAJCQkJCXJl
bGF0aW9uSWQgPSBSYW5nZVZhckdldFJlbGlkKHN0bXQtPnJlbGF0aW9uLCBOb0xvY2ssCkBAIC0z
NDgsNiArMzQ5LDcgQEAgQm9vdF9EZWNsYXJlVW5pcXVlSW5kZXhTdG10OgogCQkJCQlzdG10LT5j
b25jdXJyZW50ID0gZmFsc2U7CiAJCQkJCXN0bXQtPmlmX25vdF9leGlzdHMgPSBmYWxzZTsKIAkJ
CQkJc3RtdC0+cmVzZXRfZGVmYXVsdF90YmxzcGMgPSBmYWxzZTsKKwkJCQkJc3RtdC0+cGFydHNf
ZG9uZSA9IC0xOwogCiAJCQkJCS8qIGxvY2tzIGFuZCByYWNlcyBuZWVkIG5vdCBjb25jZXJuIHVz
IGluIGJvb3RzdHJhcCBtb2RlICovCiAJCQkJCXJlbGF0aW9uSWQgPSBSYW5nZVZhckdldFJlbGlk
KHN0bXQtPnJlbGF0aW9uLCBOb0xvY2ssCmRpZmYgLS1naXQgYS9zcmMvYmFja2VuZC9jb21tYW5k
cy9pbmRleGNtZHMuYyBiL3NyYy9iYWNrZW5kL2NvbW1hbmRzL2luZGV4Y21kcy5jCmluZGV4IDhh
ZmMwMDZmODkuLmJlNGQxYzg4NGUgMTAwNjQ0Ci0tLSBhL3NyYy9iYWNrZW5kL2NvbW1hbmRzL2lu
ZGV4Y21kcy5jCisrKyBiL3NyYy9iYWNrZW5kL2NvbW1hbmRzL2luZGV4Y21kcy5jCkBAIC0xMzAs
NiArMTMwLDI5IEBAIHR5cGVkZWYgc3RydWN0IFJlaW5kZXhFcnJvckluZm8KIAljaGFyCQlyZWxr
aW5kOwogfSBSZWluZGV4RXJyb3JJbmZvOwogCisKKy8qCisgKiBDb3VudCB0aGUgbnVtYmVyIG9m
IGRpcmVjdCBhbmQgaW5kaXJlY3QgbGVhZiBwYXJ0aXRpb25zLCBleGNsdWRpbmcgZm9yZWlnbgor
ICogdGFibGVzLgorICovCitzdGF0aWMgaW50CitudW1fbGVhZl9wYXJ0aXRpb25zKE9pZCByZWxp
ZCkKK3sKKwlpbnQJCW5sZWF2ZXMgPSAwOworCUxpc3QJKmNoaWxkcyA9IGZpbmRfYWxsX2luaGVy
aXRvcnMocmVsaWQsIE5vTG9jaywgTlVMTCk7CisJTGlzdENlbGwgKmxjOworCisJZm9yZWFjaChs
YywgY2hpbGRzKQorCXsKKwkJT2lkCXBhcnRyZWxpZCA9IGxmaXJzdF9vaWQobGMpOworCQlpZiAo
UkVMS0lORF9IQVNfU1RPUkFHRShnZXRfcmVsX3JlbGtpbmQocGFydHJlbGlkKSkpCisJCQlubGVh
dmVzKys7CisJfQorCisJbGlzdF9mcmVlKGNoaWxkcyk7CisJcmV0dXJuIG5sZWF2ZXM7Cit9CisK
IC8qCiAgKiBDaGVja0luZGV4Q29tcGF0aWJsZQogICoJCURldGVybWluZSB3aGV0aGVyIGFuIGV4
aXN0aW5nIGluZGV4IGRlZmluaXRpb24gaXMgY29tcGF0aWJsZSB3aXRoIGEKQEAgLTEyMTksOCAr
MTI0MiwxNiBAQCBEZWZpbmVJbmRleChPaWQgcmVsYXRpb25JZCwKIAkJCVJlbGF0aW9uCXBhcmVu
dEluZGV4OwogCQkJVHVwbGVEZXNjCXBhcmVudERlc2M7CiAKLQkJCXBnc3RhdF9wcm9ncmVzc191
cGRhdGVfcGFyYW0oUFJPR1JFU1NfQ1JFQVRFSURYX1BBUlRJVElPTlNfVE9UQUwsCi0JCQkJCQkJ
CQkJIG5wYXJ0cyk7CisJCQlpZiAoIU9pZElzVmFsaWQocGFyZW50SW5kZXhJZCkpCisJCQl7CisJ
CQkJaW50IHRvdGFsX3BhcnRzOworCisJCQkJLyogSW5pdCBjb3VudGVyIG9mIGRvbmUgcGFydGl0
aW9ucyB3aGVuIHByb2Nlc3Npbmcgcm9vdCBpbmRleCAqLworCQkJCXN0bXQtPnBhcnRzX2RvbmUg
PSAwOworCQkJCXRvdGFsX3BhcnRzID0gbnVtX2xlYWZfcGFydGl0aW9ucyhyZWxhdGlvbklkKTsK
KwkJCQlwZ3N0YXRfcHJvZ3Jlc3NfdXBkYXRlX3BhcmFtKFBST0dSRVNTX0NSRUFURUlEWF9QQVJU
SVRJT05TX1RPVEFMLAorCQkJCQkJCQkJCQkgdG90YWxfcGFydHMpOworCQkJfQogCiAJCQkvKiBN
YWtlIGEgbG9jYWwgY29weSBvZiBwYXJ0ZGVzYy0+b2lkc1tdLCBqdXN0IGZvciBzYWZldHkgKi8K
IAkJCW1lbWNweShwYXJ0X29pZHMsIHBhcnRkZXNjLT5vaWRzLCBzaXplb2YoT2lkKSAqIG5wYXJ0
cyk7CkBAIC0xNDMwLDEwICsxNDYxLDIwIEBAIERlZmluZUluZGV4KE9pZCByZWxhdGlvbklkLAog
CQkJCQkJCQlza2lwX2J1aWxkLCBxdWlldCk7CiAJCQkJCVNldFVzZXJJZEFuZFNlY0NvbnRleHQo
Y2hpbGRfc2F2ZV91c2VyaWQsCiAJCQkJCQkJCQkJICAgY2hpbGRfc2F2ZV9zZWNfY29udGV4dCk7
CisJCQkJCS8qIGNoaWxkU3RtdCBoYXMgbW9yZSByZWNlbnQgZGF0YSBvbiBkb25lIHBhcnRzLCB1
cGRhdGUgdGhlIHBhcmVudCAqLworCQkJCQlzdG10LT5wYXJ0c19kb25lID0gY2hpbGRTdG10LT5w
YXJ0c19kb25lOworCQkJCX0KKwkJCQllbHNlCisJCQkJeworCQkJCQkvKgorCQkJCQkgKiBJZiB0
aGUgaW5kZXggaGFzIGJlZW4gYXR0YWNoZWQsIGNvdW50IGFsbCBvZiBpdHMgbGVhZiBpbmRleGVz
IGFzIGF0dGFjaGVkLAorCQkJCQkgKiBiZWNhdXNlIFBST0dSRVNTX0NSRUFURUlEWF9QQVJUSVRJ
T05TX1RPVEFMIGluY2x1ZGVzIHRoZW0gYWxsIGluaXRpYWxseS4KKwkJCQkJICovCisJCQkJCXN0
bXQtPnBhcnRzX2RvbmUgKz0gbnVtX2xlYWZfcGFydGl0aW9ucyhjaGlsZFJlbGlkKTsKKwkJCQkJ
cGdzdGF0X3Byb2dyZXNzX3VwZGF0ZV9wYXJhbShQUk9HUkVTU19DUkVBVEVJRFhfUEFSVElUSU9O
U19ET05FLAorCQkJCQkJCQkJCQkJIHN0bXQtPnBhcnRzX2RvbmUpOwogCQkJCX0KIAotCQkJCXBn
c3RhdF9wcm9ncmVzc191cGRhdGVfcGFyYW0oUFJPR1JFU1NfQ1JFQVRFSURYX1BBUlRJVElPTlNf
RE9ORSwKLQkJCQkJCQkJCQkJIGkgKyAxKTsKIAkJCQlmcmVlX2F0dHJtYXAoYXR0bWFwKTsKIAkJ
CX0KIApAQCAtMTQ4NCw5ICsxNTI1LDE1IEBAIERlZmluZUluZGV4KE9pZCByZWxhdGlvbklkLAog
CQkvKiBDbG9zZSB0aGUgaGVhcCBhbmQgd2UncmUgZG9uZSwgaW4gdGhlIG5vbi1jb25jdXJyZW50
IGNhc2UgKi8KIAkJdGFibGVfY2xvc2UocmVsLCBOb0xvY2spOwogCi0JCS8qIElmIHRoaXMgaXMg
dGhlIHRvcC1sZXZlbCBpbmRleCwgd2UncmUgZG9uZS4gKi8KKwkJLyoKKwkJICogSWYgdGhpcyBp
cyB0aGUgdG9wLWxldmVsIGluZGV4LCB3ZSdyZSBkb25lLCBvdGhlcndpc2UsIGluY3JlbWVudAor
CQkgKiBkb25lIHBhcnRpdGlvbiBjb3VudGVyLCBpZiB0aGUgY291bnRlciBpcyBpbml0aWFsaXpl
ZC4KKwkJICovCiAJCWlmICghT2lkSXNWYWxpZChwYXJlbnRJbmRleElkKSkKIAkJCXBnc3RhdF9w
cm9ncmVzc19lbmRfY29tbWFuZCgpOworCQllbHNlIGlmIChzdG10LT5wYXJ0c19kb25lID49IDAp
CisJCQlwZ3N0YXRfcHJvZ3Jlc3NfdXBkYXRlX3BhcmFtKFBST0dSRVNTX0NSRUFURUlEWF9QQVJU
SVRJT05TX0RPTkUsCisJCQkJCQkJCQkJICsrc3RtdC0+cGFydHNfZG9uZSk7CiAKIAkJcmV0dXJu
IGFkZHJlc3M7CiAJfQpkaWZmIC0tZ2l0IGEvc3JjL2JhY2tlbmQvcGFyc2VyL3BhcnNlX3V0aWxj
bWQuYyBiL3NyYy9iYWNrZW5kL3BhcnNlci9wYXJzZV91dGlsY21kLmMKaW5kZXggYmZmYTlmOGRk
MC4uMWEyODY2OTRkYSAxMDA2NDQKLS0tIGEvc3JjL2JhY2tlbmQvcGFyc2VyL3BhcnNlX3V0aWxj
bWQuYworKysgYi9zcmMvYmFja2VuZC9wYXJzZXIvcGFyc2VfdXRpbGNtZC5jCkBAIC0xNTk5LDYg
KzE1OTksNyBAQCBnZW5lcmF0ZUNsb25lZEluZGV4U3RtdChSYW5nZVZhciAqaGVhcFJlbCwgUmVs
YXRpb24gc291cmNlX2lkeCwKIAlpbmRleC0+Y29uY3VycmVudCA9IGZhbHNlOwogCWluZGV4LT5p
Zl9ub3RfZXhpc3RzID0gZmFsc2U7CiAJaW5kZXgtPnJlc2V0X2RlZmF1bHRfdGJsc3BjID0gZmFs
c2U7CisJaW5kZXgtPnBhcnRzX2RvbmUgPSAtMTsKIAogCS8qCiAJICogV2UgZG9uJ3QgdHJ5IHRv
IHByZXNlcnZlIHRoZSBuYW1lIG9mIHRoZSBzb3VyY2UgaW5kZXg7IGluc3RlYWQsIGp1c3QKQEAg
LTIyMTcsNiArMjIxOCw3IEBAIHRyYW5zZm9ybUluZGV4Q29uc3RyYWludChDb25zdHJhaW50ICpj
b25zdHJhaW50LCBDcmVhdGVTdG10Q29udGV4dCAqY3h0KQogCWluZGV4LT5jb25jdXJyZW50ID0g
ZmFsc2U7CiAJaW5kZXgtPmlmX25vdF9leGlzdHMgPSBmYWxzZTsKIAlpbmRleC0+cmVzZXRfZGVm
YXVsdF90YmxzcGMgPSBjb25zdHJhaW50LT5yZXNldF9kZWZhdWx0X3RibHNwYzsKKwlpbmRleC0+
cGFydHNfZG9uZSA9IC0xOwogCiAJLyoKIAkgKiBJZiBpdCdzIEFMVEVSIFRBQkxFIEFERCBDT05T
VFJBSU5UIFVTSU5HIElOREVYLCBsb29rIHVwIHRoZSBpbmRleCBhbmQKZGlmZiAtLWdpdCBhL3Ny
Yy9pbmNsdWRlL25vZGVzL3BhcnNlbm9kZXMuaCBiL3NyYy9pbmNsdWRlL25vZGVzL3BhcnNlbm9k
ZXMuaAppbmRleCBjZmVjYTk2ZDUzLi4yMTkxZDI4ZTRhIDEwMDY0NAotLS0gYS9zcmMvaW5jbHVk
ZS9ub2Rlcy9wYXJzZW5vZGVzLmgKKysrIGIvc3JjL2luY2x1ZGUvbm9kZXMvcGFyc2Vub2Rlcy5o
CkBAIC0zMDA0LDYgKzMwMDQsNyBAQCB0eXBlZGVmIHN0cnVjdCBJbmRleFN0bXQKIAlib29sCQlp
Zl9ub3RfZXhpc3RzOwkvKiBqdXN0IGRvIG5vdGhpbmcgaWYgaW5kZXggYWxyZWFkeSBleGlzdHM/
ICovCiAJYm9vbAkJcmVzZXRfZGVmYXVsdF90YmxzcGM7CS8qIHJlc2V0IGRlZmF1bHRfdGFibGVz
cGFjZSBwcmlvciB0bwogCQkJCQkJCQkJCSAqIGV4ZWN1dGluZyAqLworCWludCBwYXJ0c19kb25l
OyAvKiBjb3VudGVyIG9mIGNyZWF0ZWQvYXR0YWNoZWQgaW5kZXhlcyBmb3IgYSBwYXJ0aXRpb25l
ZCBpbmRleCAqLwogfSBJbmRleFN0bXQ7CiAKIC8qIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KLS0g
CjIuMzAuMgoK


--=-HKiUwXiqzN+WsiSp8e6f--