public inbox for [email protected]help / color / mirror / Atom feed
Re: Parallel tuplesort (for parallel B-Tree index creation) 67+ messages / 8 participants [nested] [flat]
* Re: Parallel tuplesort (for parallel B-Tree index creation) @ 2016-11-08 04:28 Peter Geoghegan <[email protected]> 0 siblings, 2 replies; 67+ messages in thread From: Peter Geoghegan @ 2016-11-08 04:28 UTC (permalink / raw) To: Heikki Linnakangas <[email protected]>; +Cc: Claudio Freire <[email protected]>; Robert Haas <[email protected]>; pgsql-hackers; Corey Huinker <[email protected]>; Tom Lane <[email protected]> On Mon, Oct 24, 2016 at 6:17 PM, Peter Geoghegan <[email protected]> wrote: >> * Cost model. Should probably attempt to guess final index size, and >> derive calculation of number of workers from that. Also, I'm concerned >> that I haven't given enough thought to the low end, where with default >> settings most CREATE INDEX statements will use at least one parallel >> worker. > While I haven't made progress on any of these open items, I should > still get a version out that applies cleanly on top of git tip -- > commit b75f467b6eec0678452fd8d7f8d306e6df3a1076 caused the patch to > bitrot. I attach V4, which is a fairly mechanical rebase of V3, with > no notable behavioral changes or bug fixes. I attach V5. Changes: * A big cost model overhaul. Workers are logarithmically scaled based on projected final *index* size, not current heap size, as was the case in V4. A new nbtpage.c routine is added to estimate a not-yet-built B-Tree index's size, now called by the optimizer. This involves getting average item width for indexed attributes from pg_attribute for the heap relation. There are some subtleties here with partial indexes, null_frac, etc. I also refined the cap applied on the number of workers that limits too many workers being launched when there isn't so much maintenance_work_mem. The cost model is much improved now -- it is now more than just a placeholder, at least. It doesn't do things like launch a totally inappropriate number of workers to build a very small partial index. Granted, those workers would still have something to do -- scan the heap -- but not enough to justify launching so many (that is, launching as many as would be launched for an equivalent non-partial index). That having been said, things are still quite fudged here, and I struggle to find any guiding principle around doing better on average. I think that that's because of the inherent difficulty of modeling what's going on, but I'd be happy to be proven wrong on that. In any case, I think it's going to be fairly common for DBAs to want to use the storage parameter to force the use of a particular number of parallel workers. (See also: my remarks below on how the new bt_estimate_nblocks() SQL-callable function can give insight into the new cost model's decisions.) * Overhauled leader_mergeruns() further, to make it closer to mergeruns(). We now always rewind input tapes. This simplification involved refining some of the assertions within logtape.c, which is also slightly simplified. * 2 new testing tools are added to the final commit in the patch series (not actually proposed for commit). I've added 2 new SQL-callable functions to contrib/pageinspect. The 2 new testing functions are: bt_estimate_nblocks ------------------- bt_estimate_nblocks() provides an easy way to see the optimizer's projection of how large the final index will be. It returns an estimate in blocks. Example: mgd=# analyze; ANALYZE mgd=# select oid::regclass as rel, bt_estimated_nblocks(oid), relpages, to_char(bt_estimated_nblocks(oid)::numeric / relpages, 'FM990.990') as estimate_actual from pg_class where relkind = 'i' order by relpages desc limit 20; rel │ bt_estimated_nblocks │ relpages │ estimate_actual ────────────────────────────────────────────────────┼──────────────────────┼──────────┼───────────────── mgd.acc_accession_idx_accid │ 107,091 │ 106,274 │ 1.008 mgd.acc_accession_0 │ 169,024 │ 106,274 │ 1.590 mgd.acc_accession_1 │ 169,024 │ 80,382 │ 2.103 mgd.acc_accession_idx_prefixpart │ 76,661 │ 80,382 │ 0.954 mgd.acc_accession_idx_mgitype_key │ 76,661 │ 76,928 │ 0.997 mgd.acc_accession_idx_clustered │ 76,661 │ 76,928 │ 0.997 mgd.acc_accession_idx_createdby_key │ 76,661 │ 76,928 │ 0.997 mgd.acc_accession_idx_numericpart │ 76,661 │ 76,928 │ 0.997 mgd.acc_accession_idx_logicaldb_key │ 76,661 │ 76,928 │ 0.997 mgd.acc_accession_idx_modifiedby_key │ 76,661 │ 76,928 │ 0.997 mgd.acc_accession_pkey │ 76,661 │ 76,928 │ 0.997 mgd.mgi_relationship_property_idx_propertyname_key │ 74,197 │ 74,462 │ 0.996 mgd.mgi_relationship_property_idx_modifiedby_key │ 74,197 │ 74,462 │ 0.996 mgd.mgi_relationship_property_pkey │ 74,197 │ 74,462 │ 0.996 mgd.mgi_relationship_property_idx_clustered │ 74,197 │ 74,462 │ 0.996 mgd.mgi_relationship_property_idx_createdby_key │ 74,197 │ 74,462 │ 0.996 mgd.seq_sequence_idx_clustered │ 50,051 │ 50,486 │ 0.991 mgd.seq_sequence_raw_pkey │ 35,826 │ 35,952 │ 0.996 mgd.seq_sequence_raw_idx_modifiedby_key │ 35,826 │ 35,952 │ 0.996 mgd.seq_source_assoc_idx_clustered │ 35,822 │ 35,952 │ 0.996 (20 rows) I haven't tried to make the underlying logic as close to perfect as possible, but it tends to be accurate in practice, as is evident from this real-world example (this shows larger indexes following a restoration of the mouse genome sample database [1]). Perhaps there could be a role for a refined bt_estimate_nblocks() function in determining when B-Tree indexes become bloated/unbalanced (maybe have pgstatindex() estimate index bloat based on a difference between projected and actual fan-in?). That has nothing to do with parallel CREATE INDEX, though. bt_main_forks_identical ----------------------- bt_main_forks_identical() checks if 2 specific relations have bitwise identical main forks. If they do, it returns the number of blocks in the main fork of each. Otherwise, an error is raised. Unlike any approach involving *writing* the index in parallel (e.g., any worthwhile approach based on data partitioning), the proposed parallel CREATE INDEX implementation creates an identical index representation to that created by any serial process (including, for example, the master branch when CREATE INDEX uses an internal sort). The index that you end up with when parallelism is used ought to be 100% identical in all cases. (This is true because there is a TID tie-breaker when sorting B-Tree index tuples, and because LSNs are set to 0 by CREATE INDEX. Why not exploit that fact to test the implementation?) If anyone can demonstrate that parallel CREATE INDEX fails to create a non-bitwise-identical index representation to a "known good" implementation, or can demonstrate that it doesn't consistently produce exactly the same final index representation given the same underlying table as input, then there *must* be a bug. bt_main_forks_identical() gives reviewers an easy way to verify this, perhaps just in passing during benchmarking. pg_restore ========== It occurs to me that parallel CREATE INDEX receives no special consideration by pg_restore. This leaves it so that the use of parallel CREATE INDEX can come down to whether or not pg_class.reltuples is accidentally updated by something like an initial CREATE INDEX. This is not ideal. There is also the questions of how pg_restore -j cases ought to give special consideration to parallel CREATE INDEX, if at all -- it's probably true that concurrent index builds on the same relation do go together well with parallel CREATE INDEX, but even in V5 pg_restore remains totally naive about this. That having been said, pg_restore currently does nothing clever with maintenance_work_mem when multiple jobs are used, even though that seems at least as useful as what I outline for parallel CREATE INDEX. It's not clear how to judge this. What do we need to teach pg_restore about parallel CREATE INDEX, if anything at all? Could this be as simple as a blanket disabling of parallelism for CREATE INDEX from pg_restore? Or, does it need to be more sophisticated than that? I suppose that tools like reindexdb and pgbench must be considered in a similar way. Maybe we could get the number of blocks in the heap relation when its pg_class.reltupes is 0, from the smgr, and then extrapolate the reltuples using simple, generic logic, in the style of vac_estimate_reltuples() (its "old_rel_pages" == 0 case). For now, I've avoided doing that out of concern for the overhead in cases where there are many small tables to be restored, and because it may be better to err on the side of not using parallelism. [1] https://wiki.postgresql.org/wiki/Sample_Databases -- Peter Geoghegan -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers Attachments: [application/x-gzip] 0001-Cap-the-number-of-tapes-used-by-external-sorts.patch.gz (1.8K, ../../CAM3SWZSA_+CBsaYyb22cdqwkfNgD9vQL74NVYnbu62ZqhLD4Og@mail.gmail.com/2-0001-Cap-the-number-of-tapes-used-by-external-sorts.patch.gz) download [application/x-gzip] 0003-Add-temporary-testing-tools.patch.gz (4.7K, ../../CAM3SWZSA_+CBsaYyb22cdqwkfNgD9vQL74NVYnbu62ZqhLD4Og@mail.gmail.com/3-0003-Add-temporary-testing-tools.patch.gz) download [application/x-gzip] 0002-Add-parallel-B-tree-index-build-sorting.patch.gz (57.0K, ../../CAM3SWZSA_+CBsaYyb22cdqwkfNgD9vQL74NVYnbu62ZqhLD4Og@mail.gmail.com/4-0002-Add-parallel-B-tree-index-build-sorting.patch.gz) download ^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Parallel tuplesort (for parallel B-Tree index creation) @ 2016-11-10 00:01 Robert Haas <[email protected]> parent: Peter Geoghegan <[email protected]> 1 sibling, 1 reply; 67+ messages in thread From: Robert Haas @ 2016-11-10 00:01 UTC (permalink / raw) To: Peter Geoghegan <[email protected]>; +Cc: Heikki Linnakangas <[email protected]>; Claudio Freire <[email protected]>; pgsql-hackers; Corey Huinker <[email protected]>; Tom Lane <[email protected]> On Mon, Nov 7, 2016 at 11:28 PM, Peter Geoghegan <[email protected]> wrote: > I attach V5. I gather that 0001, which puts a cap on the number of tapes, is not actually related to the subject of this thread; it's an independent change that you think is a good idea. I reviewed the previous discussion on this topic upthread, between you and Heikki, which seems to me to contain more heat than light. At least in my opinion, the question is not whether a limit on the number of tapes is the best possible system, but rather whether it's better than the status quo. It's silly to refuse to make a simple change on the grounds that some much more complex change might be better, because if somebody writes that patch and it is better we can always revert 0001 then. If 0001 involved hundreds of lines of invasive code changes, that argument wouldn't apply, but it doesn't; it's almost a one-liner. Now, on the other hand, as far as I can see, the actual amount of evidence that 0001 is a good idea which has been presented in this forum is pretty near zero. You've argued for it on theoretical grounds several times, but theoretical arguments are not a substitute for test results. Therefore, I decided that the best thing to do was test it myself. I wrote a little patch to add a GUC for max_sort_tapes, which actually turns out not to work as I thought: setting max_sort_tapes = 501 seems to limit the highest tape number to 501 rather than the number of tapes to 501, so there's a sort of off-by-one error. But that doesn't really matter. The patch is attached here for the convenience of anyone else who may want to fiddle with this. Next, I tried to set things up so that I'd get a large enough number of tapes for the cap to matter. To do that, I initialized with "pgbench -i --unlogged-tables -s 20000" so that I had 2 billion tuples. Then I used this SQL query: "select sum(w+abalance) from (select (aid::numeric * 7123000217)%1000000000 w, * from pgbench_accounts order by 1) x". The point of the math is to perturb the ordering of the tuples so that they actually need to be sorted instead of just passed through unchanged. The use of abalance in the outer sum prevents an index-only-scan from being used, which makes the sort wider; perhaps I should have tried to make it wider still, but this is what I did. I wanted to have more than 501 tapes because, obviously, a concern with a change like this is that things might get slower in the case where it forces a polyphase merge rather than a single merge pass. And, of course, I set trace_sort = on. Here's what my initial run looked like, in brief: 2016-11-09 15:37:52 UTC [44026] LOG: begin tuple sort: nkeys = 1, workMem = 262144, randomAccess = f 2016-11-09 15:37:59 UTC [44026] LOG: switching to external sort with 937 tapes: CPU: user: 5.51 s, system: 0.27 s, elapsed: 6.56 s 2016-11-09 16:48:31 UTC [44026] LOG: finished writing run 616 to tape 615: CPU: user: 4029.17 s, system: 152.72 s, elapsed: 4238.54 s 2016-11-09 16:48:31 UTC [44026] LOG: using 246719 KB of memory for read buffers among 616 input tapes 2016-11-09 16:48:39 UTC [44026] LOG: performsort done (except 616-way final merge): CPU: user: 4030.30 s, system: 152.98 s, elapsed: 4247.41 s 2016-11-09 18:33:30 UTC [44026] LOG: external sort ended, 6255145 disk blocks used: CPU: user: 10214.64 s, system: 175.24 s, elapsed: 10538.06 s And according to psql: Time: 10538068.225 ms (02:55:38.068) Then I set max_sort_tapes = 501 and ran it again. This time: 2016-11-09 19:05:22 UTC [44026] LOG: begin tuple sort: nkeys = 1, workMem = 262144, randomAccess = f 2016-11-09 19:05:28 UTC [44026] LOG: switching to external sort with 502 tapes: CPU: user: 5.69 s, system: 0.26 s, elapsed: 6.13 s 2016-11-09 20:15:20 UTC [44026] LOG: finished writing run 577 to tape 75: CPU: user: 3993.81 s, system: 153.42 s, elapsed: 4198.52 s 2016-11-09 20:15:20 UTC [44026] LOG: using 249594 KB of memory for read buffers among 501 input tapes 2016-11-09 20:21:19 UTC [44026] LOG: finished 77-way merge step: CPU: user: 4329.50 s, system: 160.67 s, elapsed: 4557.22 s 2016-11-09 20:21:19 UTC [44026] LOG: performsort done (except 501-way final merge): CPU: user: 4329.50 s, system: 160.67 s, elapsed: 4557.22 s 2016-11-09 21:38:12 UTC [44026] LOG: external sort ended, 6255484 disk blocks used: CPU: user: 8848.81 s, system: 182.64 s, elapsed: 9170.62 s And this one, according to psql: Time: 9170629.597 ms (02:32:50.630) That looks very good. On a test that runs for almost 3 hours, we saved more than 20 minutes. The overall runtime improvement is 23% in a case where we would not expect this patch to do particularly well; after all, without limiting the number of runs, we are able to complete the sort with a single merge pass, whereas when we reduce the number of runs, we now require a polyphase merge. Nevertheless, we come out way ahead, because the final merge pass gets way faster, presumably because there are fewer tapes involved. The first test does a 616-way final merge and takes 6184.34 seconds to do it. The second test does a 501-way final merge and takes 4519.31 seconds to do. This increased final merge speed accounts for practically all of the speedup, and the reason it's faster pretty much has to be that it's merging fewer tapes. That, in turn, happens for two reasons. First, because limiting the number of tapes increases slightly the memory available for storing the tuples belonging to each run, we end up with fewer runs in the first place. The number of runs drops from from 616 to 577, about a 7% reduction. Second, because we have more runs than tapes in the second case, it does a 77-way merge prior to the final merge. Because of that 77-way merge, the time at which the second run starts producing tuples is slightly later. Instead of producing the first tuple at 70:47.71, we have to wait until 75:72.22. That's a small disadvantage in this case, because it's hypothetically possible that a query like this could have a LIMIT and we'd end up worse off overall. However, that's pretty unlikely, for three reasons. Number one, LIMIT isn't likely to be used on queries of this type in the first place. Number two, if it were used, we'd probably end up with a bounded sort plan which would be way faster anyway. Number three, if somehow we still sorted the data set we'd still win in this case if the limit were more than about 20% of the total number of tuples. The much faster run time to produce the whole data set is a small price to pay for possibly needing to wait a little longer for the first tuple. Admittedly, this is only one test, and some other test might show a different result. However, I believe that there aren't likely to be many losing cases. If the increased number of tapes doesn't force a polyphase merge, we're almost certain to win, because in that case the only thing that changes is that we have more memory with which to produce each run. On small sorts, this may not help much, but it won't hurt. Even if the increased number of tapes *does* force a polyphase merge, the reduction in the number of initial runs and/or the reduction in the number of runs in any single merge may add up to a win, as in this example. In fact, it may well be the case that the optimal number of tapes is significantly less than 501. It's hard to tell for sure, but it sure looks like that 77-way non-final merge is significantly more efficient than the final merge. So, I'm now feeling pretty bullish about this patch, except for one thing, which is that I think the comments are way off-base. Peter writes: $$When allowedMem is significantly lower than what is required for an internal sort, it is unlikely that there are benefits to increasing the number of tapes beyond Knuth's "sweet spot" of 7.$$ I'm pretty sure that's totally wrong, first of all because commit df700e6b40195d28dc764e0c694ac8cef90d4638 improved performance by doing precisely the thing which this comment says we shouldn't, secondly because 501 is most definitely significantly higher than 7 so the code and the comment don't even match, and thirdly because, as the comment added in the commit says, each extra tape doesn't really cost that much. In this example, going from 501 tapes up to 937 tapes only reduces memory available for tuples by about 7%, even though the number of tapes have almost doubled. If we had a sort with, say, 30 runs, do we really want to do a polyphase merge just to get a sub-1% increase in the amount of memory per run? I doubt it. Given all that, what I'm inclined to do is rewrite the comment to say, basically, that even though we can afford lots of tapes, it's better not to allow too ridiculously many because (1) that eats away at the amount of memory available for tuples in each initial run and (2) very high-order final merges are not very efficient. And then commit that. If somebody wants to fine-tune the tape limit later after more extensive testing or replacing it by some other system that is better, great. Sound OK? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers Attachments: [binary/octet-stream] max-sort-tapes.patch (1.7K, ../../CA+Tgmobi5zSYAVO9usxMWzfJxnkoh0p2mgP-ZeO=+o1iDOv-EQ@mail.gmail.com/2-max-sort-tapes.patch) download | inline diff: diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index 3c695c1..4cf5825 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -1673,6 +1673,15 @@ static struct config_bool ConfigureNamesBool[] = static struct config_int ConfigureNamesInt[] = { { + {"max_sort_tapes", PGC_USERSET, DEVELOPER_OPTIONS, + gettext_noop("Limits number of sort tapes."), + NULL + }, + &max_sort_tapes, + 0, 0, INT_MAX / 2, + NULL, NULL, NULL + }, + { {"archive_timeout", PGC_SIGHUP, WAL_ARCHIVING, gettext_noop("Forces a switch to the next xlog file if a " "new file has not been started within N seconds."), diff --git a/src/backend/utils/sort/tuplesort.c b/src/backend/utils/sort/tuplesort.c index 46512cf..d1869e2 100644 --- a/src/backend/utils/sort/tuplesort.c +++ b/src/backend/utils/sort/tuplesort.c @@ -152,6 +152,7 @@ #ifdef TRACE_SORT bool trace_sort = false; #endif +int max_sort_tapes = 0; #ifdef DEBUG_BOUNDED_SORT bool optimize_bounded_sort = true; @@ -2316,6 +2317,10 @@ tuplesort_merge_order(int64 allowedMem) /* Even in minimum memory, use at least a MINORDER merge */ mOrder = Max(mOrder, MINORDER); + /* If max_sort_tapes is non-zero, limit sorts to that number of tapes. */ + if (max_sort_tapes != 0 && mOrder > max_sort_tapes) + mOrder = max_sort_tapes; + return mOrder; } diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index 78545da..13858f2 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -466,4 +466,6 @@ extern bool has_rolreplication(Oid roleid); extern bool BackupInProgress(void); extern void CancelBackup(void); +extern int max_sort_tapes; + #endif /* MISCADMIN_H */ ^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Parallel tuplesort (for parallel B-Tree index creation) @ 2016-11-10 00:54 Peter Geoghegan <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 2 replies; 67+ messages in thread From: Peter Geoghegan @ 2016-11-10 00:54 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Heikki Linnakangas <[email protected]>; Claudio Freire <[email protected]>; pgsql-hackers; Corey Huinker <[email protected]>; Tom Lane <[email protected]> On Wed, Nov 9, 2016 at 4:01 PM, Robert Haas <[email protected]> wrote: > I gather that 0001, which puts a cap on the number of tapes, is not > actually related to the subject of this thread; it's an independent > change that you think is a good idea. I reviewed the previous > discussion on this topic upthread, between you and Heikki, which seems > to me to contain more heat than light. FWIW, I don't remember it that way. Heikki seemed to be uncomfortable with the quasi-arbitrary choice of constant, rather than disagreeing with the general idea of a cap. Or, maybe he thought I didn't go far enough, by completely removing polyphase merge. I think that removing polyphase merge would be an orthogonal change to this, though. > Now, on the other hand, as far as I can see, the actual amount of > evidence that 0001 is a good idea which has been presented in this > forum is pretty near zero. You've argued for it on theoretical > grounds several times, but theoretical arguments are not a substitute > for test results. See the illustration in TAOCP, vol III, page 273 in the second edition -- "Fig. 70. Efficiency of Polyphase merge using Algorithm D". I think that it's actually a real-world benchmark. I guess I felt that no one ever argued that as many tapes as possible was sound on any grounds, even theoretical, and so didn't feel obligated to test it until asked to do so. I think that the reason that a cap like this didn't go in around the time that the growth logic went in (2006) was because nobody followed up on it. If you look at the archives, there is plenty of discussion of a cap like this at the time. > That looks very good. On a test that runs for almost 3 hours, we > saved more than 20 minutes. The overall runtime improvement is 23% in > a case where we would not expect this patch to do particularly well; > after all, without limiting the number of runs, we are able to > complete the sort with a single merge pass, whereas when we reduce the > number of runs, we now require a polyphase merge. Nevertheless, we > come out way ahead, because the final merge pass gets way faster, > presumably because there are fewer tapes involved. The first test > does a 616-way final merge and takes 6184.34 seconds to do it. The > second test does a 501-way final merge and takes 4519.31 seconds to > do. This increased final merge speed accounts for practically all of > the speedup, and the reason it's faster pretty much has to be that > it's merging fewer tapes. It's CPU cache efficiency -- has to be. > That, in turn, happens for two reasons. First, because limiting the > number of tapes increases slightly the memory available for storing > the tuples belonging to each run, we end up with fewer runs in the > first place. The number of runs drops from from 616 to 577, about a > 7% reduction. Second, because we have more runs than tapes in the > second case, it does a 77-way merge prior to the final merge. Because > of that 77-way merge, the time at which the second run starts > producing tuples is slightly later. Instead of producing the first > tuple at 70:47.71, we have to wait until 75:72.22. That's a small > disadvantage in this case, because it's hypothetically possible that a > query like this could have a LIMIT and we'd end up worse off overall. > However, that's pretty unlikely, for three reasons. Number one, LIMIT > isn't likely to be used on queries of this type in the first place. > Number two, if it were used, we'd probably end up with a bounded sort > plan which would be way faster anyway. Number three, if somehow we > still sorted the data set we'd still win in this case if the limit > were more than about 20% of the total number of tuples. The much > faster run time to produce the whole data set is a small price to pay > for possibly needing to wait a little longer for the first tuple. Cool. > So, I'm now feeling pretty bullish about this patch, except for one > thing, which is that I think the comments are way off-base. Peter > writes: $When allowedMem is significantly lower than what is required > for an internal sort, it is unlikely that there are benefits to > increasing the number of tapes beyond Knuth's "sweet spot" of 7.$ > I'm pretty sure that's totally wrong, first of all because commit > df700e6b40195d28dc764e0c694ac8cef90d4638 improved performance by doing > precisely the thing which this comment says we shouldn't It's more complicated than that. As I said, I think that Knuth basically had it right with his sweet spot of 7. I think that commit df700e6b40195d28dc764e0c694ac8cef90d4638 was effective in large part because a one-pass merge avoided certain overheads not inherent to polyphase merge, like all that memory accounting stuff, extra palloc() traffic, etc. The expanded use of per tape buffering we have even in multi-pass cases likely makes that much less true for us these days. The reason I haven't actually gone right back down to 7 with this cap is that it's possible that the added I/O costs outweigh the CPU costs in extreme cases, even though I think that polyphase merge doesn't have all that much to do with I/O costs, even with its 1970s perspective. Knuth doesn't say much about I/O costs -- it's more about using an extremely small amount of memory effectively (minimizing CPU costs with very little available main memory). Furthermore, not limiting ourselves to 7 tapes and seeing a benefit (benefitting from a few dozen or hundred instead) seems more possible with the improved merge heap maintenance logic added recently, where there could be perhaps hundreds of runs merged with very low CPU cost in the event of presorted input (or, input that is inversely logically/physically correlated). That would be true because we'd only examine the top of the heap through, and so I/O costs may matter much more. Depending on the exact details, I bet you could see a benefit with only 7 tapes due to CPU cache efficiency in a case like the one you describe. Perhaps when sorting integers, but not when sorting collated text. There are many competing considerations, which I've tried my best to balance here with a merge order of 500. > Sound OK? I'm fine with not mentioning Knuth's sweet spot once more. I guess it's not of much practical value that he was on to something with that. I realize, on reflection, that my understanding of what's going on is very nuanced. Thanks -- Peter Geoghegan -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers ^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Parallel tuplesort (for parallel B-Tree index creation) @ 2016-11-10 01:03 Peter Geoghegan <[email protected]> parent: Peter Geoghegan <[email protected]> 1 sibling, 0 replies; 67+ messages in thread From: Peter Geoghegan @ 2016-11-10 01:03 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Heikki Linnakangas <[email protected]>; Claudio Freire <[email protected]>; pgsql-hackers; Corey Huinker <[email protected]>; Tom Lane <[email protected]> On Wed, Nov 9, 2016 at 4:54 PM, Peter Geoghegan <[email protected]> wrote: > It's more complicated than that. As I said, I think that Knuth > basically had it right with his sweet spot of 7. I think that commit > df700e6b40195d28dc764e0c694ac8cef90d4638 was effective in large part > because a one-pass merge avoided certain overheads not inherent to > polyphase merge, like all that memory accounting stuff, extra palloc() > traffic, etc. The expanded use of per tape buffering we have even in > multi-pass cases likely makes that much less true for us these days. Also, logtape.c fragmentation made multiple merge pass cases experience increased random I/O in a way that was only an accident of our implementation. We've fixed that now, but that problem must have added further cost that df700e6b40195d28dc764e0c694ac8cef90d4638 *masked* when it was commited in 2006. (I do think that the problem with the merge heap maintenance fixed recently in 24598337c8d214ba8dcf354130b72c49636bba69 was the biggest problem that the 2006 work masked, though). -- Peter Geoghegan -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers ^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Parallel tuplesort (for parallel B-Tree index creation) @ 2016-11-10 02:57 Robert Haas <[email protected]> parent: Peter Geoghegan <[email protected]> 1 sibling, 1 reply; 67+ messages in thread From: Robert Haas @ 2016-11-10 02:57 UTC (permalink / raw) To: Peter Geoghegan <[email protected]>; +Cc: Heikki Linnakangas <[email protected]>; Claudio Freire <[email protected]>; pgsql-hackers; Corey Huinker <[email protected]>; Tom Lane <[email protected]> On Wed, Nov 9, 2016 at 7:54 PM, Peter Geoghegan <[email protected]> wrote: >> Now, on the other hand, as far as I can see, the actual amount of >> evidence that 0001 is a good idea which has been presented in this >> forum is pretty near zero. You've argued for it on theoretical >> grounds several times, but theoretical arguments are not a substitute >> for test results. > > See the illustration in TAOCP, vol III, page 273 in the second edition > -- "Fig. 70. Efficiency of Polyphase merge using Algorithm D". I think > that it's actually a real-world benchmark. I don't have that publication, and I'm guessing that's not based on PostgreSQL's implementation. There's no substitute for tests using the code we've actually got. >> So, I'm now feeling pretty bullish about this patch, except for one >> thing, which is that I think the comments are way off-base. Peter >> writes: $When allowedMem is significantly lower than what is required >> for an internal sort, it is unlikely that there are benefits to >> increasing the number of tapes beyond Knuth's "sweet spot" of 7.$ >> I'm pretty sure that's totally wrong, first of all because commit >> df700e6b40195d28dc764e0c694ac8cef90d4638 improved performance by doing >> precisely the thing which this comment says we shouldn't > > It's more complicated than that. As I said, I think that Knuth > basically had it right with his sweet spot of 7. I think that commit > df700e6b40195d28dc764e0c694ac8cef90d4638 was effective in large part > because a one-pass merge avoided certain overheads not inherent to > polyphase merge, like all that memory accounting stuff, extra palloc() > traffic, etc. The expanded use of per tape buffering we have even in > multi-pass cases likely makes that much less true for us these days. > > The reason I haven't actually gone right back down to 7 with this cap > is that it's possible that the added I/O costs outweigh the CPU costs > in extreme cases, even though I think that polyphase merge doesn't > have all that much to do with I/O costs, even with its 1970s > perspective. Knuth doesn't say much about I/O costs -- it's more about > using an extremely small amount of memory effectively (minimizing CPU > costs with very little available main memory). > > Furthermore, not limiting ourselves to 7 tapes and seeing a benefit > (benefitting from a few dozen or hundred instead) seems more possible > with the improved merge heap maintenance logic added recently, where > there could be perhaps hundreds of runs merged with very low CPU cost > in the event of presorted input (or, input that is inversely > logically/physically correlated). That would be true because we'd only > examine the top of the heap through, and so I/O costs may matter much > more. > > Depending on the exact details, I bet you could see a benefit with > only 7 tapes due to CPU cache efficiency in a case like the one you > describe. Perhaps when sorting integers, but not when sorting collated > text. There are many competing considerations, which I've tried my > best to balance here with a merge order of 500. I guess that's possible, but the problem with polyphase merge is that the increased I/O becomes a pretty significant cost in a hurry. Here's the same test with max_sort_tapes = 100: 2016-11-09 23:02:49 UTC [48551] LOG: begin tuple sort: nkeys = 1, workMem = 262144, randomAccess = f 2016-11-09 23:02:55 UTC [48551] LOG: switching to external sort with 101 tapes: CPU: user: 5.72 s, system: 0.25 s, elapsed: 6.04 s 2016-11-10 00:13:00 UTC [48551] LOG: finished writing run 544 to tape 49: CPU: user: 4003.00 s, system: 156.89 s, elapsed: 4211.33 s 2016-11-10 00:16:52 UTC [48551] LOG: finished 51-way merge step: CPU: user: 4214.84 s, system: 161.94 s, elapsed: 4442.98 s 2016-11-10 00:25:41 UTC [48551] LOG: finished 100-way merge step: CPU: user: 4704.14 s, system: 170.83 s, elapsed: 4972.47 s 2016-11-10 00:36:47 UTC [48551] LOG: finished 99-way merge step: CPU: user: 5333.12 s, system: 179.94 s, elapsed: 5638.52 s 2016-11-10 00:45:32 UTC [48551] LOG: finished 99-way merge step: CPU: user: 5821.13 s, system: 189.00 s, elapsed: 6163.53 s 2016-11-10 01:01:29 UTC [48551] LOG: finished 100-way merge step: CPU: user: 6691.10 s, system: 210.60 s, elapsed: 7120.58 s 2016-11-10 01:01:29 UTC [48551] LOG: performsort done (except 100-way final merge): CPU: user: 6691.10 s, system: 210.60 s, elapsed: 7120.58 s 2016-11-10 01:45:40 UTC [48551] LOG: external sort ended, 6255949 disk blocks used: CPU: user: 9271.07 s, system: 232.26 s, elapsed: 9771.49 s This is already worse than max_sort_tapes = 501, though the total runtime is still better than no cap (the time-to-first-tuple is way worse, though). I'm going to try max_sort_tapes = 10 next, but I think the basic pattern is already fairly clear. As you reduce the cap on the number of tapes, (a) the time to build the initial runs doesn't change very much, (b) the time to perform the final merge decreases significantly, and (c) the time to perform the non-final merges increases even faster. In this particular test configuration on this particular hardware, rewriting 77 tapes in the 501-tape configuration wasn't too bad, but now that we're down to 100 tapes, we have to rewrite 449 tapes out of a total of 544, and that's actually a loss: rewriting the bulk of your data an extra time to save on cache misses doesn't pay. It would probably be even less good if there were other concurrent activity on the system. It's possible that if your polyphase merge is actually being done all in memory, cache efficiency might remain the dominant consideration, but I think we should assume that a polyphase merge is doing actual I/O, because it's sort of pointless to use that algorithm in the first place if there's no real I/O involved. At the moment, at least, it looks to me as though we don't need to be afraid of a *little* bit of polyphase merging, but a *lot* of polyphase merging is actually pretty bad. In other words, by imposing a limit of the number of tapes, we're going to improve sorts that are smaller than work_mem * num_tapes * ~1.5 -- because cache efficiency will be better -- but above that things will probably get worse because of the increased I/O cost. From that point of view, a 500-tape limit is the same as saying that it's we don't think it's entirely reasonable to try to perform a sort that exceeds work_mem by a factor of more than ~750, whereas a 7-tape limit is the same as saying that we don't think it's entirely reasonable to perform a sort that exceeds work_mem by a factor of more than ~10. That latter proposition seems entirely untenable. Our default work_mem setting is 4MB, and people will certainly expect to be able to get away with, say, an 80MB sort without changing settings. On the other hand, if they're sorting more than 3GB with work_mem = 4MB, I think we'll be justified in making a gentle suggestion that they reconsider that setting. Among other arguments, it's going to be pretty slow in that case no matter what we do here. Maybe another way of putting this is that, while there's clearly a benefit to having some kind of a cap, it's appropriate to pick a large value, such as 500. Having no cap at all risks creating many extra tapes that just waste memory, and also risks an unduly cache-inefficient final merge. Reigning that in makes sense. However, we can't reign it in too far or we'll create slow polyphase merges in case that are reasonably likely to occur in real life. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers ^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Parallel tuplesort (for parallel B-Tree index creation) @ 2016-11-10 03:18 Peter Geoghegan <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 1 reply; 67+ messages in thread From: Peter Geoghegan @ 2016-11-10 03:18 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Heikki Linnakangas <[email protected]>; Claudio Freire <[email protected]>; pgsql-hackers; Corey Huinker <[email protected]>; Tom Lane <[email protected]> On Wed, Nov 9, 2016 at 6:57 PM, Robert Haas <[email protected]> wrote: > I guess that's possible, but the problem with polyphase merge is that > the increased I/O becomes a pretty significant cost in a hurry. Not if you have a huge RAID array. :-) Obviously I'm not seriously suggesting that we revise the cap from 500 to 7. We're only concerned about the constant factors here. There is a clearly a need to make some simplifying assumptions. I think that you understand this very well, though. > Maybe another way of putting this is that, while there's clearly a > benefit to having some kind of a cap, it's appropriate to pick a large > value, such as 500. Having no cap at all risks creating many extra > tapes that just waste memory, and also risks an unduly > cache-inefficient final merge. Reigning that in makes sense. > However, we can't reign it in too far or we'll create slow polyphase > merges in case that are reasonably likely to occur in real life. I completely agree with your analysis. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers ^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Parallel tuplesort (for parallel B-Tree index creation) @ 2016-11-15 15:44 Robert Haas <[email protected]> parent: Peter Geoghegan <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Robert Haas @ 2016-11-15 15:44 UTC (permalink / raw) To: Peter Geoghegan <[email protected]>; +Cc: Heikki Linnakangas <[email protected]>; Claudio Freire <[email protected]>; pgsql-hackers; Corey Huinker <[email protected]>; Tom Lane <[email protected]> On Wed, Nov 9, 2016 at 10:18 PM, Peter Geoghegan <[email protected]> wrote: >> Maybe another way of putting this is that, while there's clearly a >> benefit to having some kind of a cap, it's appropriate to pick a large >> value, such as 500. Having no cap at all risks creating many extra >> tapes that just waste memory, and also risks an unduly >> cache-inefficient final merge. Reigning that in makes sense. >> However, we can't reign it in too far or we'll create slow polyphase >> merges in case that are reasonably likely to occur in real life. > > I completely agree with your analysis. Cool. BTW, my run with 10 tapes completed in 10696528.377 ms (02:58:16.528) - i.e. almost 3 minutes slower than with no tape limit. Building runs took 4260.16 s, and the final merge pass began at 8239.12 s. That's certainly better than I expected, and it seems to show that even if the number of tapes is grossly inadequate for the number of runs, you can still make up most of the time that you lose to I/O with improved cache efficiency -- at least under favorable circumstances. Of course, on many systems I/O bandwidth will be a scarce resource, so that argument can be overdone -- and even if not, a 10-tape sort version takes FAR longer to deliver the first tuple. I also tried this out with work_mem = 512MB. Doubling work_mem reduces the number of runs enough that we don't get a polyphase merge in any case. With no limit on tapes: 2016-11-10 11:24:45 UTC [54042] LOG: switching to external sort with 1873 tapes: CPU: user: 11.34 s, system: 0.48 s, elapsed: 12.13 s 2016-11-10 12:36:22 UTC [54042] LOG: finished writing run 308 to tape 307: CPU: user: 4096.63 s, system: 156.88 s, elapsed: 4309.66 s 2016-11-10 12:36:22 UTC [54042] LOG: using 516563 KB of memory for read buffers among 308 input tapes 2016-11-10 12:36:30 UTC [54042] LOG: performsort done (except 308-way final merge): CPU: user: 4097.75 s, system: 157.24 s, elapsed: 4317.76 s 2016-11-10 13:54:07 UTC [54042] LOG: external sort ended, 6255577 disk blocks used: CPU: user: 8638.72 s, system: 177.42 s, elapsed: 8974.44 s With a max_sort_tapes = 501: 2016-11-10 14:23:50 UTC [54042] LOG: switching to external sort with 502 tapes: CPU: user: 10.99 s, system: 0.54 s, elapsed: 11.57 s 2016-11-10 15:36:47 UTC [54042] LOG: finished writing run 278 to tape 277: CPU: user: 4190.31 s, system: 155.33 s, elapsed: 4388.86 s 2016-11-10 15:36:47 UTC [54042] LOG: using 517313 KB of memory for read buffers among 278 input tapes 2016-11-10 15:36:54 UTC [54042] LOG: performsort done (except 278-way final merge): CPU: user: 4191.36 s, system: 155.68 s, elapsed: 4395.66 s 2016-11-10 16:53:39 UTC [54042] LOG: external sort ended, 6255699 disk blocks used: CPU: user: 8673.07 s, system: 175.93 s, elapsed: 9000.80 s 0.3% slower with the tape limit, but that might be noise. Even if not, it seems pretty silly to create 1873 tapes when we only need ~300. At work_mem = 2GB: 2016-11-10 18:08:00 UTC [54042] LOG: switching to external sort with 7490 tapes: CPU: user: 44.28 s, system: 1.99 s, elapsed: 46.33 s 2016-11-10 19:23:06 UTC [54042] LOG: finished writing run 77 to tape 76: CPU: user: 4342.10 s, system: 156.21 s, elapsed: 4551.95 s 2016-11-10 19:23:06 UTC [54042] LOG: using 2095202 KB of memory for read buffers among 77 input tapes 2016-11-10 19:23:12 UTC [54042] LOG: performsort done (except 77-way final merge): CPU: user: 4343.36 s, system: 157.07 s, elapsed: 4558.79 s 2016-11-10 20:24:24 UTC [54042] LOG: external sort ended, 6255946 disk blocks used: CPU: user: 7894.71 s, system: 176.36 s, elapsed: 8230.13 s At work_mem = 2GB, max_sort_tapes = 501: 2016-11-10 21:28:23 UTC [54042] LOG: switching to external sort with 502 tapes: CPU: user: 44.09 s, system: 1.94 s, elapsed: 46.07 s 2016-11-10 22:42:28 UTC [54042] LOG: finished writing run 68 to tape 67: CPU: user: 4278.49 s, system: 154.39 s, elapsed: 4490.25 s 2016-11-10 22:42:28 UTC [54042] LOG: using 2095427 KB of memory for read buffers among 68 input tapes 2016-11-10 22:42:34 UTC [54042] LOG: performsort done (except 68-way final merge): CPU: user: 4279.60 s, system: 155.21 s, elapsed: 4496.83 s 2016-11-10 23:42:10 UTC [54042] LOG: external sort ended, 6255983 disk blocks used: CPU: user: 7733.98 s, system: 173.85 s, elapsed: 8072.55 s Roughly 2% faster. Maybe still noise, but less likely. 7490 tapes certainly seems over the top. At work_mem = 8GB: 2016-11-14 19:17:28 UTC [54042] LOG: switching to external sort with 29960 tapes: CPU: user: 183.80 s, system: 7.71 s, elapsed: 191.61 s 2016-11-14 20:32:02 UTC [54042] LOG: finished writing run 20 to tape 19: CPU: user: 4431.44 s, system: 176.82 s, elapsed: 4665.16 s 2016-11-14 20:32:02 UTC [54042] LOG: using 8388083 KB of memory for read buffers among 20 input tapes 2016-11-14 20:32:26 UTC [54042] LOG: performsort done (except 20-way final merge): CPU: user: 4432.99 s, system: 181.29 s, elapsed: 4689.52 s 2016-11-14 21:30:56 UTC [54042] LOG: external sort ended, 6256003 disk blocks used: CPU: user: 7835.83 s, system: 199.01 s, elapsed: 8199.29 s At work_mem = 8GB, max_sort_tapes = 501: 2016-11-14 21:52:43 UTC [54042] LOG: switching to external sort with 502 tapes: CPU: user: 181.08 s, system: 7.66 s, elapsed: 189.05 s 2016-11-14 23:06:06 UTC [54042] LOG: finished writing run 17 to tape 16: CPU: user: 4381.56 s, system: 161.82 s, elapsed: 4591.63 s 2016-11-14 23:06:06 UTC [54042] LOG: using 8388158 KB of memory for read buffers among 17 input tapes 2016-11-14 23:06:36 UTC [54042] LOG: performsort done (except 17-way final merge): CPU: user: 4383.45 s, system: 165.32 s, elapsed: 4622.04 s 2016-11-14 23:54:00 UTC [54042] LOG: external sort ended, 6256002 disk blocks used: CPU: user: 7124.49 s, system: 182.16 s, elapsed: 7466.18 s Roughly 9% faster. Building runs seems to be very slowly degrading as we increase work_mem, but the final merge is speeding up somewhat more quickly. Intuitively that makes sense to me: if merging were faster than quicksorting, we could just merge-sort all the time instead of using quicksort for internal sorts. Also, we've got 29960 tapes now, better than three orders of magnitude more than what we actually need. At this work_mem setting, 501 tapes is enough to efficiently sort at least 4TB of data and quite possibly a good bit more. So, committed 0001, with comment changes along the lines I proposed before. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers ^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Parallel tuplesort (for parallel B-Tree index creation) @ 2016-12-04 01:29 Peter Geoghegan <[email protected]> parent: Peter Geoghegan <[email protected]> 1 sibling, 1 reply; 67+ messages in thread From: Peter Geoghegan @ 2016-12-04 01:29 UTC (permalink / raw) To: Heikki Linnakangas <[email protected]>; +Cc: Claudio Freire <[email protected]>; Robert Haas <[email protected]>; pgsql-hackers; Corey Huinker <[email protected]>; Tom Lane <[email protected]> On Mon, Nov 7, 2016 at 8:28 PM, Peter Geoghegan <[email protected]> wrote: > What do we need to teach pg_restore about parallel CREATE INDEX, if > anything at all? Could this be as simple as a blanket disabling of > parallelism for CREATE INDEX from pg_restore? Or, does it need to be > more sophisticated than that? I suppose that tools like reindexdb and > pgbench must be considered in a similar way. I still haven't resolved this question, which seems like the most important outstanding question, but I attach V6. Changes: * tuplesort.c was adapted to use the recently committed condition variables stuff. This made things cleaner. No more ad-hoc WaitLatch() looping. * Adapted docs to mention the newly committed max_parallel_workers GUC in the context of discussing proposed max_parallel_workers_maintenance GUC. * Fixed trivial assertion failure bug that could be tripped when a conventional sort uses very little memory. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers Attachments: [application/x-gzip] 0002-Add-temporary-testing-tools.patch.gz (4.7K, ../../CAM3SWZQ0EgCxs2NoEUwB6BACi-pp5qVmp432i+Wh_XyUrg1FbQ@mail.gmail.com/2-0002-Add-temporary-testing-tools.patch.gz) download [application/x-gzip] 0001-Add-parallel-B-tree-index-build-sorting.patch.gz (56.9K, ../../CAM3SWZQ0EgCxs2NoEUwB6BACi-pp5qVmp432i+Wh_XyUrg1FbQ@mail.gmail.com/3-0001-Add-parallel-B-tree-index-build-sorting.patch.gz) download ^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Parallel tuplesort (for parallel B-Tree index creation) @ 2016-12-04 01:45 Alvaro Herrera <[email protected]> parent: Peter Geoghegan <[email protected]> 0 siblings, 1 reply; 67+ messages in thread From: Alvaro Herrera @ 2016-12-04 01:45 UTC (permalink / raw) To: Peter Geoghegan <[email protected]>; +Cc: Heikki Linnakangas <[email protected]>; Claudio Freire <[email protected]>; Robert Haas <[email protected]>; pgsql-hackers; Corey Huinker <[email protected]>; Tom Lane <[email protected]> Peter Geoghegan wrote: > On Mon, Nov 7, 2016 at 8:28 PM, Peter Geoghegan <[email protected]> wrote: > > What do we need to teach pg_restore about parallel CREATE INDEX, if > > anything at all? Could this be as simple as a blanket disabling of > > parallelism for CREATE INDEX from pg_restore? Or, does it need to be > > more sophisticated than that? I suppose that tools like reindexdb and > > pgbench must be considered in a similar way. > > I still haven't resolved this question, which seems like the most > important outstanding question, I don't think a patch must necessarily consider all possible uses that the new feature may have. If we introduce parallel index creation, that's great; if pg_restore doesn't start using it right away, that's okay. You, or somebody else, can still patch it later. The patch is still a step forward. -- Álvaro Herrera https://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers ^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Parallel tuplesort (for parallel B-Tree index creation) @ 2016-12-04 02:37 Peter Geoghegan <[email protected]> parent: Alvaro Herrera <[email protected]> 0 siblings, 1 reply; 67+ messages in thread From: Peter Geoghegan @ 2016-12-04 02:37 UTC (permalink / raw) To: Alvaro Herrera <[email protected]>; +Cc: Heikki Linnakangas <[email protected]>; Claudio Freire <[email protected]>; Robert Haas <[email protected]>; pgsql-hackers; Corey Huinker <[email protected]>; Tom Lane <[email protected]> On Sat, Dec 3, 2016 at 5:45 PM, Alvaro Herrera <[email protected]> wrote: > I don't think a patch must necessarily consider all possible uses that > the new feature may have. If we introduce parallel index creation, > that's great; if pg_restore doesn't start using it right away, that's > okay. You, or somebody else, can still patch it later. The patch is > still a step forward. While I agree, right now pg_restore will tend to use or not use parallelism for CREATE INDEX more or less by accident, based on whether or not pg_class.reltuples has already been set by something else (e.g., an earlier CREATE INDEX against the same table in the restoration). That seems unacceptable. I haven't just suppressed the use of parallel CREATE INDEX within pg_restore because that would be taking a position on something I have a hard time defending any particular position on. And so, I am slightly concerned about the entire ecosystem of tools that could implicitly use parallel CREATE INDEX, with undesirable consequences. Especially pg_restore. It's not so much a hard question as it is an awkward one. I want to handle any possible objection about there being future compatibility issues with going one way or the other ("This paints us into a corner with..."). And, there is no existing, simple way for pg_restore and other tools to disable the use of parallelism due to the cost model automatically kicking in, while still allowing the proposed new index storage parameter ("parallel_workers") to force the use of parallelism, which seems like something that should happen. (I might have to add a new GUC like "enable_maintenance_paralleism", since "max_parallel_workers_maintenance = 0" disables parallelism no matter how it might be invoked). In general, I have a positive outlook on this patch, since it appears to compete well with similar implementations in other systems scalability-wise. It does what it's supposed to do. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers ^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Parallel tuplesort (for parallel B-Tree index creation) @ 2016-12-04 03:23 Tomas Vondra <[email protected]> parent: Peter Geoghegan <[email protected]> 0 siblings, 1 reply; 67+ messages in thread From: Tomas Vondra @ 2016-12-04 03:23 UTC (permalink / raw) To: Peter Geoghegan <[email protected]>; Alvaro Herrera <[email protected]>; +Cc: Heikki Linnakangas <[email protected]>; Claudio Freire <[email protected]>; Robert Haas <[email protected]>; pgsql-hackers; Corey Huinker <[email protected]>; Tom Lane <[email protected]> On Sat, 2016-12-03 at 18:37 -0800, Peter Geoghegan wrote: > On Sat, Dec 3, 2016 at 5:45 PM, Alvaro Herrera <alvherre@2ndquadrant. > com> wrote: > > > > I don't think a patch must necessarily consider all possible uses > > that > > the new feature may have. If we introduce parallel index creation, > > that's great; if pg_restore doesn't start using it right away, > > that's > > okay. You, or somebody else, can still patch it later. The patch > > is > > still a step forward. > While I agree, right now pg_restore will tend to use or not use > parallelism for CREATE INDEX more or less by accident, based on > whether or not pg_class.reltuples has already been set by something > else (e.g., an earlier CREATE INDEX against the same table in the > restoration). That seems unacceptable. I haven't just suppressed the > use of parallel CREATE INDEX within pg_restore because that would be > taking a position on something I have a hard time defending any > particular position on. And so, I am slightly concerned about the > entire ecosystem of tools that could implicitly use parallel CREATE > INDEX, with undesirable consequences. Especially pg_restore. > > It's not so much a hard question as it is an awkward one. I want to > handle any possible objection about there being future compatibility > issues with going one way or the other ("This paints us into a corner > with..."). And, there is no existing, simple way for pg_restore and > other tools to disable the use of parallelism due to the cost model > automatically kicking in, while still allowing the proposed new index > storage parameter ("parallel_workers") to force the use of > parallelism, which seems like something that should happen. (I might > have to add a new GUC like "enable_maintenance_paralleism", since > "max_parallel_workers_maintenance = 0" disables parallelism no matter > how it might be invoked). I do share your concerns about unpredictable behavior - that's particularly worrying for pg_restore, which may be used for time- sensitive use cases (DR, migrations between versions), so unpredictable changes in behavior / duration are unwelcome. But isn't this more a deficiency in pg_restore, than in CREATE INDEX? The issue seems to be that the reltuples value may or may not get updated, so maybe forcing ANALYZE (even very low statistics_target values would do the trick, I think) would be more appropriate solution? Or maybe it's time add at least some rudimentary statistics into the dumps (the reltuples field seems like a good candidate). Trying to fix this by adding more GUCs seems a bit strange to me. > > In general, I have a positive outlook on this patch, since it appears > to compete well with similar implementations in other systems > scalability-wise. It does what it's supposed to do. > +1 to that -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers ^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Parallel tuplesort (for parallel B-Tree index creation) @ 2016-12-04 20:44 Peter Geoghegan <[email protected]> parent: Tomas Vondra <[email protected]> 0 siblings, 1 reply; 67+ messages in thread From: Peter Geoghegan @ 2016-12-04 20:44 UTC (permalink / raw) To: Tomas Vondra <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Heikki Linnakangas <[email protected]>; Claudio Freire <[email protected]>; Robert Haas <[email protected]>; pgsql-hackers; Corey Huinker <[email protected]>; Tom Lane <[email protected]> On Sat, Dec 3, 2016 at 7:23 PM, Tomas Vondra <[email protected]> wrote: > I do share your concerns about unpredictable behavior - that's > particularly worrying for pg_restore, which may be used for time- > sensitive use cases (DR, migrations between versions), so unpredictable > changes in behavior / duration are unwelcome. Right. > But isn't this more a deficiency in pg_restore, than in CREATE INDEX? > The issue seems to be that the reltuples value may or may not get > updated, so maybe forcing ANALYZE (even very low statistics_target > values would do the trick, I think) would be more appropriate solution? > Or maybe it's time add at least some rudimentary statistics into the > dumps (the reltuples field seems like a good candidate). I think that there is a number of reasonable ways of looking at it. It might also be worthwhile to have a minimal ANALYZE performed by CREATE INDEX directly, iff there are no preexisting statistics (there is definitely going to be something pg_restore-like that we cannot fix -- some ETL tool, for example). Perhaps, as an additional condition to proceeding with such an ANALYZE, it should also only happen when there is any chance at all of parallelism being used (but then you get into having to establish the relation size reliably in the absence of any pg_class.relpages, which isn't very appealing when there are many tiny indexes). In summary, I would really like it if a consensus emerged on how parallel CREATE INDEX should handle the ecosystem of tools like pg_restore, reindexdb, and so on. Personally, I'm neutral on which general approach should be taken. Proposals from other hackers about what to do here are particularly welcome. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers ^ permalink raw reply [nested|flat] 67+ messages in thread
* Re: Parallel tuplesort (for parallel B-Tree index creation) @ 2016-12-05 05:08 Haribabu Kommi <[email protected]> parent: Peter Geoghegan <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Haribabu Kommi @ 2016-12-05 05:08 UTC (permalink / raw) To: Peter Geoghegan <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Heikki Linnakangas <[email protected]>; Claudio Freire <[email protected]>; Robert Haas <[email protected]>; pgsql-hackers; Corey Huinker <[email protected]>; Tom Lane <[email protected]> On Mon, Dec 5, 2016 at 7:44 AM, Peter Geoghegan <[email protected]> wrote: > On Sat, Dec 3, 2016 at 7:23 PM, Tomas Vondra > <[email protected]> wrote: > > I do share your concerns about unpredictable behavior - that's > > particularly worrying for pg_restore, which may be used for time- > > sensitive use cases (DR, migrations between versions), so unpredictable > > changes in behavior / duration are unwelcome. > > Right. > > > But isn't this more a deficiency in pg_restore, than in CREATE INDEX? > > The issue seems to be that the reltuples value may or may not get > > updated, so maybe forcing ANALYZE (even very low statistics_target > > values would do the trick, I think) would be more appropriate solution? > > Or maybe it's time add at least some rudimentary statistics into the > > dumps (the reltuples field seems like a good candidate). > > I think that there is a number of reasonable ways of looking at it. It > might also be worthwhile to have a minimal ANALYZE performed by CREATE > INDEX directly, iff there are no preexisting statistics (there is > definitely going to be something pg_restore-like that we cannot fix -- > some ETL tool, for example). Perhaps, as an additional condition to > proceeding with such an ANALYZE, it should also only happen when there > is any chance at all of parallelism being used (but then you get into > having to establish the relation size reliably in the absence of any > pg_class.relpages, which isn't very appealing when there are many tiny > indexes). > > In summary, I would really like it if a consensus emerged on how > parallel CREATE INDEX should handle the ecosystem of tools like > pg_restore, reindexdb, and so on. Personally, I'm neutral on which > general approach should be taken. Proposals from other hackers about > what to do here are particularly welcome. > > Moved to next CF with "needs review" status. Regards, Hari Babu Fujitsu Australia ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 02/10] review @ 2021-03-16 16:29 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-16 16:29 UTC (permalink / raw) --- src/backend/optimizer/path/allpaths.c | 2 ++ src/backend/optimizer/plan/initsplan.c | 9 ++++++++- src/backend/optimizer/util/plancat.c | 3 +++ 3 files changed, 13 insertions(+), 1 deletion(-) diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..fc1a3a68a2 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -1065,8 +1065,10 @@ set_append_rel_size(PlannerInfo *root, RelOptInfo *rel, /* Whole row is not null, so must be same for child */ childrel->notnullattrs = bms_add_member(childrel->notnullattrs, attno - FirstLowInvalidHeapAttributeNumber); + /* XXX shouldn't this be a continue, instead of a break? */ break; } + /* XXX isn't this missing 'else'? */ if (attno < 0 ) /* no need to translate system column */ child_attno = attno; diff --git a/src/backend/optimizer/plan/initsplan.c b/src/backend/optimizer/plan/initsplan.c index d27167dc76..4ef876cf7b 100644 --- a/src/backend/optimizer/plan/initsplan.c +++ b/src/backend/optimizer/plan/initsplan.c @@ -829,7 +829,14 @@ deconstruct_recurse(PlannerInfo *root, Node *jtnode, bool below_outer_join, { Node *qual = (Node *) lfirst(l); - /* Set the not null info now */ + /* Set the not null info now + * + * XXX Why now? Why is this the right place to do this? Does it need + * to happen before distribute_qual_to_rels, for example? + * + * XXX Not clear to me why this looks at non-nullable vars? Shouldn't + * we already have the bitmap built from atnums (from get_relation_info)? + */ ListCell *lc; List *non_nullable_vars = find_nonnullable_vars(qual); foreach(lc, non_nullable_vars) diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index eebabcfccf..cd703e41ba 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -481,6 +481,9 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, if (inhparent && relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) set_relation_partition_info(root, rel, relation); + /* + * Build information about which attributes are marked as NOT NULL. + */ Assert(rel->notnullattrs == NULL); for(i = 0; i < relation->rd_att->natts; i++) { -- 2.30.2 --------------F495F6B18672582269F00FF9 Content-Type: text/x-patch; charset=UTF-8; name="0003-Introduce-UniqueKey-attributes-on-RelOptInf-20210317.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0003-Introduce-UniqueKey-attributes-on-RelOptInf-20210317.pa"; filename*1="tch" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 02/10] review @ 2021-03-16 16:29 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-16 16:29 UTC (permalink / raw) --- src/backend/optimizer/path/allpaths.c | 2 ++ src/backend/optimizer/plan/initsplan.c | 9 ++++++++- src/backend/optimizer/util/plancat.c | 3 +++ 3 files changed, 13 insertions(+), 1 deletion(-) diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..fc1a3a68a2 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -1065,8 +1065,10 @@ set_append_rel_size(PlannerInfo *root, RelOptInfo *rel, /* Whole row is not null, so must be same for child */ childrel->notnullattrs = bms_add_member(childrel->notnullattrs, attno - FirstLowInvalidHeapAttributeNumber); + /* XXX shouldn't this be a continue, instead of a break? */ break; } + /* XXX isn't this missing 'else'? */ if (attno < 0 ) /* no need to translate system column */ child_attno = attno; diff --git a/src/backend/optimizer/plan/initsplan.c b/src/backend/optimizer/plan/initsplan.c index d27167dc76..4ef876cf7b 100644 --- a/src/backend/optimizer/plan/initsplan.c +++ b/src/backend/optimizer/plan/initsplan.c @@ -829,7 +829,14 @@ deconstruct_recurse(PlannerInfo *root, Node *jtnode, bool below_outer_join, { Node *qual = (Node *) lfirst(l); - /* Set the not null info now */ + /* Set the not null info now + * + * XXX Why now? Why is this the right place to do this? Does it need + * to happen before distribute_qual_to_rels, for example? + * + * XXX Not clear to me why this looks at non-nullable vars? Shouldn't + * we already have the bitmap built from atnums (from get_relation_info)? + */ ListCell *lc; List *non_nullable_vars = find_nonnullable_vars(qual); foreach(lc, non_nullable_vars) diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index eebabcfccf..cd703e41ba 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -481,6 +481,9 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, if (inhparent && relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) set_relation_partition_info(root, rel, relation); + /* + * Build information about which attributes are marked as NOT NULL. + */ Assert(rel->notnullattrs == NULL); for(i = 0; i < relation->rd_att->natts; i++) { -- 2.30.2 --------------F495F6B18672582269F00FF9 Content-Type: text/x-patch; charset=UTF-8; name="0003-Introduce-UniqueKey-attributes-on-RelOptInf-20210317.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0003-Introduce-UniqueKey-attributes-on-RelOptInf-20210317.pa"; filename*1="tch" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 04/10] review @ 2021-03-16 18:26 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-16 18:26 UTC (permalink / raw) --- src/backend/nodes/copyfuncs.c | 4 + src/backend/nodes/makefuncs.c | 1 - src/backend/nodes/outfuncs.c | 1 + src/backend/nodes/readfuncs.c | 2 + src/backend/optimizer/path/README.uniquekey | 285 +++--- src/backend/optimizer/path/allpaths.c | 10 +- src/backend/optimizer/path/joinrels.c | 7 + src/backend/optimizer/path/uniquekeys.c | 906 ++++++++++++++------ src/backend/optimizer/plan/planner.c | 10 + src/backend/optimizer/prep/prepunion.c | 1 + src/backend/optimizer/util/inherit.c | 1 + 11 files changed, 863 insertions(+), 365 deletions(-) diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 75c1c5e824..9d832ddc03 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,9 @@ _copyPathKey(const PathKey *from) return newnode; } +/* + * _copyUniqueKey + */ static UniqueKey * _copyUniqueKey(const UniqueKey *from) { @@ -2306,6 +2309,7 @@ _copyUniqueKey(const UniqueKey *from) return newnode; } + /* * _copyRestrictInfo */ diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 40415d0f5b..e156c9cdf8 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -816,7 +816,6 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) return v; } - /* * makeUniqueKey */ diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 44154cde6a..13905e6037 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2460,6 +2460,7 @@ static void _outUniqueKey(StringInfo str, const UniqueKey *node) { WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); WRITE_BOOL_FIELD(multi_nullvals); } diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index b3e212bf1c..8830c8df99 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -496,8 +496,10 @@ static UniqueKey * _readUniqueKey(void) { READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); } diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey index 5eac761995..31cdb5ed65 100644 --- a/src/backend/optimizer/path/README.uniquekey +++ b/src/backend/optimizer/path/README.uniquekey @@ -1,131 +1,208 @@ -1. What is UniqueKey? -We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure -that doesn't yields same result among all the rows. The simplest UniqueKey -format is primary key. +review comments: +XXX Maybe move this to src/backend/optimizer/README.uniquekey? +XXX multi_nullvals name seems a bit weird +XXX no info about populate_distinctrel_uniquekeys, populate_grouprel_uniquekeys, populate_unionrel_uniquekeys +----- +src/backend/optimizer/path/README.uniquekey + +UniqueKey +========= + +UniqueKey is a set of exprs for a RelOptInfo, which are known to have unique +values on all the rows in the relation. A trivial example is a primary key +defined on a relation - each attributes of the constraint is a unique key. + +We can use this knowledge to perform optimization in a number of places. Some +of the optimizations are fairly obvious, others are less so: + +1. remove DISTINCT node if the clause is unique +2. remove aggregation if group by clause is unique +3. remove_useless_joins +4. reduce_semianti_joins +5. Index Skip Scan (WIP) +6. Aggregation Push-Down without 2-phase aggregation if the join can't + duplicate the aggregated rows. (WIP) + -However we define the UnqiueKey as below. +UniqueKey struct +---------------- -typedef struct UniqueKey -{ +A UnqiueKey is represented by the following struct: + + typedef struct UniqueKey + { NodeTag type; List *exprs; bool multi_nullvals; -} UniqueKey; - -exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL -is a special case of UniqueKey, which means there is only one row in that -relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is -normal unique key and may have different values. SELECT colx FROM t WHERE uk = -const. colx is unique AND we have only 1 value. This field can used for -innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow -function. - -multi_nullvals: true means multi null values may exist in these exprs, so the -uniqueness is not guaranteed in this case. This field is necessary for -remove_useless_join & reduce_unique_semijoins where we don't mind these -duplicated NULL values. It is set to true for 2 cases. One is a unique key -from a unique index but the related column is nullable. The other one is for -outer join. see populate_joinrel_uniquekeys for detail. - - -The UniqueKey can be used at the following cases at least: -1. remove_useless_joins. -2. reduce_semianti_joins -3. remove distinct node if distinct clause is unique. -4. remove aggnode if group by clause is unique. -5. Index Skip Scan (WIP) -6. Aggregation Push Down without 2 phase aggregation if the join can't - duplicated the aggregated rows. (work in progress feature) - -2. How is it maintained? - -We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on -various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, -groupedrel, unionrel. and we also need to convert the uniquekey from subquery -to outer relation, which is what convert_subquery_uniquekeys does. - -1. The first part is about baserel. We handled 3 cases. suppose we have Unique -Index on (a, b). - -1. SELECT a, b FROM t. UniqueKey (a, b) -2. SELECT a FROM t WHERE b = 1; UniqueKey (a) -3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every - column is Unique. - -2. The next part is joinrel, this part is most error-prone, we simplified the rules -like below: -1. If the relation's UniqueKey can't be duplicated after join, then is will be - still valid for the join rel. The function we used here is - innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. - -2. If the UnqiueKey can't keep valid via the rule 1, the combination of the - UniqueKey from both sides are valid for sure. We can prove this as: if the - unique exprs from rel1 is duplicated by rel2, the duplicated rows must - contains different unique exprs from rel2. - -More considerations about onerow: -1. If relation with one row and it can't be duplicated, it is still possible - contains mulit_nullvas after outer join. -2. If the either UniqueKey can be duplicated after join, the can get one row - only when both side is one row AND there is no outer join. -3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one - row UniqueKey to normal unique key since we don't store exprs for one-row - relation. get_exprs_from_uniquekeys will be used here. - - -More considerations about multi_nullvals after join: + } UniqueKey; + +exprs is a list of exprs which are know to be unique on current RelOptInfo. + +exprs = NIL is a special case, meaning there is only one row in the relation. +This has has a stronger semantic than others. Consider for example + + SELECT uk FROM t + +where 'uk' is a unique key. This guarantees uniqueness, but there may be mamy +rows in the relation. On the other hand, consider this query + + SELECT colx FROM t WHERE uk = const + +In this case we know there's only a single matching row (thanks to a condition +on the unique key), which in turn guarantees uniqueness of the colx value, even +if there is no constraint on the column itself. + +This knowledge is used in innerrel_is_unique, and is handled as a special case +in add_uniquekey_for_onerow. + + +The multi_nullvals field tracks whether the expressions may contain multiple +NULL values. This can happen for example when the unique key is derived from +a unique index with nullable columns, or because of outer joins (which may add +NULL values to a known-unique list - see populate_joinrel_uniquekeys). + +In this case uniqueness is not guaranteed, but we can still use the information +in places places where NULL values are harmless - when removing useless joins, +reducing semijoins, and so on. + + +How is it maintained? +--------------------- + +Deducing the unique keys depends on the type of the relation - for each case +there's a separate "populate" function: + + +populate_baserel_uniquekeys +--------------------------- + +There are three cases, all assuming there's a unique index (e.g. on (a,b)): + +1. SELECT a, b FROM t => UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1 => UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; => UniqueKey (NIL) + +The last query is the "one row" case, in which case every column is Unique. + + +populate_joinrel_uniquekeys +--------------------------- + +For joins, deducing the unique keys may be fairly complex and error-prone. +We've simplified the rules like this: + +1. If the UniqueKey on an input relation can't be duplicated by the join, then +it will be valid for the join rel. A typical example is a join like this: + + inner_rel.any_col = outer_rel.unique_key + +The function used to detect this is innerrel_keeps_unique. + +2. Any combination of unique keys on each side of the join is a unique key +for the join relation. This can be proved by contradiction - assume we have +unique key on either side of the join - uk1 and uk2. If the values in uk1 get +duplicated by the join with uk2 (by matching the row to multiple rows), the +duplicated rows must have different values in the uk2. + +We can also leverage information about the "one row" case: + +1. If one of the input relations is known to have a single row, and the join +can't duplicate the row (e.g. semi/anti join), we can keep the unique keys. +It may however contain multi_nullvals after an outer join. + +XXX Not sure I understand the original logic/wording :-( + +2. If either UniqueKey can be duplicated after a join, there can be only one +row only when both sides are "one row" AND there is no outer join. + +XXX Why the restriction on not allowing outer joins? + +3. Whenever the one row UniqueKey is not a valid any more, we need to convert +UniqueKey to normal unique key since we don't store exprs for one-row relation. +This is done by get_exprs_from_uniquekeys. + +The join case needs to be careful about multi_nullvals too: + 1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have - mulit_nullvals in any case. -2. If a unique key doesn't allow mulit_nullvals, after some outer join, it - allows some outer join. +mulit_nullvals in any case too. + +2. If the original unique key doesn't allow multi_nullvals, the unique key for +the join relation may allow multi_nullvals after an outer join. + + +subqueries +---------- + +It's necessary to "translate" unique keys between a subquery and the outer rels, +which is what convert_subquery_uniquekeys does. This does almost exactly what +convert_subquery_pathkeys does for pathkeys. It keeps only unique keys matching +Vars in the outer relation. The relationship between outerrel.Var and +subquery.exprs is built from outerel->subroot->processed_tlist. -3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like -convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as -a Var in outer relation will be reused. The relationship between the outerrel.Var -and subquery.exprs is built with outerel->subroot->processed_tlist. +set-returning functions +------------------------ +As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. So +we will maintain the UniqueKey until there and reset it to NIL at that place. -4. As for the SRF functions, it will break the uniqueness of uniquekey, However it -is handled in adjust_paths_for_srfs, which happens after the query_planner. so -we will maintain the UniqueKey until there and reset it to NIL at that -places. This can't help on distinct/group by elimination cases but probably help -in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is -semantic correctly. +This can't help on distinct/group by elimination cases but probably help in some +other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is correct. -5. As for inherit table, we first main the UnqiueKey on childrel as well. But for -partitioned table we need to maintain 2 different kinds of -UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child -relation for partition wise query. +populate_partitionedrel_uniquekeys +---------------------------------- + +As for inherit table, we first build the UnqiueKey on childrel as well. But for +partitioned table we need to maintain two different kinds of UniqueKey: + +1) UniqueKey on the parent relation + +2) UniqueKey on child + +This is needed because a unique key from the partition may not be be unique key +on the partitioned table. + Example: -CREATE TABLE p (a int not null, b int not null) partition by list (a); + +CREATE TABLE p (a INT NOT NULL, b INT NOT NULL) PARTITION BY LIST (a); + CREATE TABLE p0 partition of p for values in (1); CREATE TABLE p1 partition of p for values in (2); -create unique index p0_b on p0(b); -create unique index p1_b on p1(b); +CREATE UNIQUE INDEX p0_b ON p0(b); +CREATE UNIQUE INDEX p1_b ON p1(b); -Now b is only unique on partition level, so the distinct can't be removed on -the following cases. SELECT DISTINCT b FROM p; +SELECT DISTINCT b FROM p; -Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one -partition is chosen, the UniqueKey on child relation is same as the UniqueKey on -parent relation. +Now "b" is only unique on partition level, but the two partitions may contain +duplicate values for the "b" column (with different values in "a"). That means +the DISTINCT clause can't be removed. -Another usage of UniqueKey on partition level is it be helpful for -partition-wise join. +Now consider: -As for the UniqueKey on parent table level, it comes with 2 different ways, -1). the UniqueKey is also derived in UniqueKey index, but the index must be same -in all the related children relations and the unique index must contains -Partition Key in it. Example: +SELECT DISTINCT a, b FROM p WHERE a = 1 + +In this case, the optimizer eliminates all partitions except for one, so that +the UniqueKey is valid for the parent relation too. + +UniqueKey at a partition level is useful for partition-wise join too. + +XXX Explain why is it useful? + +A UniqueKey from a partition can be transferred to the parent relation, in two +cases. A trivial case is if there's a single child relation (e.g. thanks to +partition elimination). In that case all unique keys on the child relation are +automatically valid for the parent relation. If there are multiple relations, +the unique key must be defived from an index present in all partitions, and the +index has to include the partition key. + +Example: CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. -- Query SELECT a, b FROM p; the (a, b) is a UniqueKey of p. -2). If the parent relation has only one childrel, the UniqueKey on childrel is - the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 66bf6f19f7..a801707eaa 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -581,7 +581,8 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) /* * Now that we've marked which partial indexes are suitable, we can now - * build the relation's unique keys. + * build the relation's unique keys. We need to do it in this order, + * so that we don't deduce unique keys from inapplicable partial indexes. */ populate_baserel_uniquekeys(root, rel, rel->indexlist); @@ -1305,6 +1306,12 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + + /* + * XXX Maybe move the check into populate populate_partitionedrel_uniquekeys? + * XXX What if it's append rel (but not partitioned one), but there's only one + * child relation? We could still deduce unique keys, no? + */ if (IS_PARTITIONED_REL(rel)) populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2314,6 +2321,7 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + /* Convert subpath's unique keys to outer representation */ convert_subquery_uniquekeys(root, rel, sub_final_rel); /* If outer rel allows parallelism, do same for partial paths. */ diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 7271f044ec..eefba449d6 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -925,6 +925,13 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + /* + * Determine which of the unique keys from input relations are applicable + * for the join result. + * + * XXX We do this after trying the partitionwise join, because that may allow + * using additional unique keys. + */ populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c index 77ed2b2eff..114e8334f5 100644 --- a/src/backend/optimizer/path/uniquekeys.c +++ b/src/backend/optimizer/path/uniquekeys.c @@ -36,7 +36,7 @@ typedef struct UniqueKeyContextData bool useful; } *UniqueKeyContext; -static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static List *initialize_uniquecontext_for_joinrel(RelOptInfo *inputrel); static bool innerrel_keeps_unique(PlannerInfo *root, RelOptInfo *outerrel, RelOptInfo *innerrel, @@ -80,8 +80,20 @@ static void add_uniquekey_from_sortgroups(PlannerInfo *root, /* * populate_baserel_uniquekeys - * Populate 'baserel' uniquekeys list by looking at the rel's unique index - * and baserestrictinfo + * Build list of unique keys for the base relation. + * + * Inspects unique indexes defined on the relation and determines what + * unique keys are valid. Partial indexes are considered too, if the + * predicate is valid. + * + * This also inspects baserestrictinfo, because we need to determine + * which opclass families are interesting when inspecting indexes. If we + * have a unique index and distinct clause with a mismatching opclasses, + * we should not use that. + * + * XXX Why does this look at baserestrictinfo? + * + * XXX What about collations? */ void populate_baserel_uniquekeys(PlannerInfo *root, @@ -99,22 +111,48 @@ populate_baserel_uniquekeys(PlannerInfo *root, Assert(baserel->rtekind == RTE_RELATION); + if (!indexlist) + return; + + /* + * Determine which unique indexes to use to build the unique keys. + * We have to skip partial with predicates not matched by the query, + * and unique indexes that are not immediately enforced. + * + * XXX Do we actually skip indexes that are not immediate? + * XXX What about hypothetical indexes? + */ foreach(lc, indexlist) { IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || (ind->indpred != NIL && !ind->predOK)) continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); } + /* If there are not applicable unique indexes, we're done. */ if (matched_uniq_indexes == NIL) return; - /* Check which attrs is used in baserel->reltarget */ - pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + /* + * Determine which attrs are referenced in baserel->reltarget. To use the + * unique key info, we need all the columns - a unique index on (a,b) may + * not be unique on (a). If a column is missing in reltarget, the nodes + * above can't possibly use it, and we can just ignore any matching index. + */ + pull_varattnos((Node *) baserel->reltarget->exprs, baserel->relid, &used_attrs); - /* Check which attrno is used at a mergeable const filter */ + /* + * Check which attrno is used at a mergeable const filter + * + * XXX This is not lookint att attrno at all, maybe obsolete comment? + * + * Seems the primary purpose of this is determining which opclass + * families to use when matching unique indexes in the next loop? + */ foreach(lc, baserel->baserestrictinfo) { RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); @@ -122,6 +160,10 @@ populate_baserel_uniquekeys(PlannerInfo *root, if (rinfo->mergeopfamilies == NIL) continue; + /* + * XXX What if bms_is_empty is true for both left_relids/right_relids? + * Or what if it's false in both cases? + */ if (bms_is_empty(rinfo->left_relids)) { const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); @@ -136,40 +178,69 @@ populate_baserel_uniquekeys(PlannerInfo *root, expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); } + /* + * Now try to match unique indexes to attributes in reltarget, and to + * merge operator families. The index may be on the right attributes, + * but if it's not matching the opfamily it's useless. + * + * XXX Can we have multiple baserestrictinfo for the same attribute, + * with different opfamilies? Probably not. + */ foreach(lc, matched_uniq_indexes) { - bool multi_nullvals, useful; - List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), - const_exprs, - expr_opfamilies, - used_attrs, - &useful, - &multi_nullvals); - if (useful) + bool multi_nullvals, + useful; + + IndexOptInfo *index_info = (IndexOptInfo *) lfirst_node(IndexOptInfo, lc); + + List *exprs = get_exprs_from_uniqueindex(index_info, + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + + if (!useful) + continue; + + /* + * All the columns in Unique Index matched with a restrictinfo, so + * that we know there's just a one row in the result. If we find + * such index, we're done - we discard all other unique keys and + * keep just this special one. In principle, this is a stronger + * guarantee, because all subsets of one row are still unique. + * + * XXX Is it correct to just return? Doesn't that prevent some + * optimizations that might be possible with the other keys? + */ + if (exprs == NIL) { - if (exprs == NIL) - { - /* All the columns in Unique Index matched with a restrictinfo */ - add_uniquekey_for_onerow(baserel); - return; - } - baserel->uniquekeys = lappend(baserel->uniquekeys, - makeUniqueKey(exprs, multi_nullvals)); + /* discards all previous uniquekeys */ + add_uniquekey_for_onerow(baserel); + return; } + + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); } } /* * populate_partitionedrel_uniquekeys - * The UniqueKey on partitionrel comes from 2 cases: - * 1). Only one partition is involved in this query, the unique key can be - * copied to parent rel from childrel. - * 2). There are some unique index which includes partition key and exists - * in all the related partitions. - * We never mind rule 2 if we hit rule 1. + * Determine unique keys for a partitioned relation. + * + * Inspects unique keys for all partitions and derives unique keys that + * are valid for the whole partitioned table. There are two basic cases: + * + * 1) There's only one remaining partition (thanks to pruning all other + * partitions). In this case all the unique keys from the partition are + * trivially valid for the partitioned table. + * + * 2) All the partitions have the same unique index (on the same set of + * columns), and the index includes the partition key. This ensures the + * combination of values is unique for the whole partitioned table. */ - void populate_partitionedrel_uniquekeys(PlannerInfo *root, RelOptInfo *rel, @@ -180,110 +251,181 @@ populate_partitionedrel_uniquekeys(PlannerInfo *root, RelOptInfo *childrel; bool is_first = true; + /* XXX What about append rels? At least for the one-child case? */ Assert(IS_PARTITIONED_REL(rel)); + /* if there are no child relations, we're done. */ if (childrels == NIL) return; /* - * If there is only one partition used in this query, the UniqueKey in childrel is - * still valid in parent level, but we need convert the format from child expr to - * parent expr. + * If there is only one partition used in this query, the UniqueKey for + * a child relation is still valid for the parent level. We need to + * convert the format from child expr to parent expr. */ if (list_length(childrels) == 1) { - /* Check for Rule 1 */ RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + + /* If the partition has a single row, so does the parent. */ if (relation_is_onerow(childrel)) { add_uniquekey_for_onerow(rel); return; } + /* + * Inspect the unique keys one by one, try reusing them for the + * parent relation. + * + * FIXME This needs more work to handle expressions and not just + * simple Vars. + */ foreach(lc, childrel->uniquekeys) { + ListCell *lc2; + List *parent_exprs = NIL; + bool can_reuse = true; + UniqueKey *ukey = lfirst_node(UniqueKey, lc); AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); - List *parent_exprs = NIL; - bool can_reuse = true; - ListCell *lc2; + + /* + * XXX Not sure what exactly we do here. Surely we deal with + * expressions at child/parent level elsewhere? Can't we just + * copy the code from there? + */ foreach(lc2, ukey->exprs) { - Var *var = (Var *)lfirst(lc2); + Var *var = (Var *) lfirst(lc2); + /* - * If the expr comes from a expression, it is hard to build the expression - * in parent so ignore that case for now. + * XXX For now this only supports simple Var expressions, + * so if there's a more complex expression we'll not copy + * the unique key to the parent. */ if(!IsA(var, Var)) { can_reuse = false; break; } + /* Convert it to parent var */ - parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + parent_exprs = lappend(parent_exprs, + find_parent_var(appinfo, var)); } - if (can_reuse) - rel->uniquekeys = lappend(rel->uniquekeys, - makeUniqueKey(parent_exprs, - ukey->multi_nullvals)); + + /* ignore unique keys with complex expressions */ + if (!can_reuse) + continue; + + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); } + + return; } - else + + /* + * A parent with multiple child relations. We only care about indexes that + * are in all child relations, so we loop through indexes on the first one + * and check that they exist in the other child relations too. + */ + + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) { - /* Check for rule 2 */ - childrel = linitial_node(RelOptInfo, childrels); - foreach(lc, childrel->indexlist) - { - IndexOptInfo *ind = lfirst(lc); - IndexOptInfo *modified_index; - if (!ind->unique || !ind->immediate || - (ind->indpred != NIL && !ind->predOK)) - continue; + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; - /* - * During simple_copy_indexinfo_to_parent, we need to convert var from - * child var to parent var, index on expression is too complex to handle. - * so ignore it for now. - */ - if (ind->indexprs != NIL) - continue; + /* + * Ignore indexes that are not unique, immediately enforced. Partial + * indexes with mismatched predicate are useless too. + */ + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; - modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); - /* - * If the unique index doesn't contain partkey, then it is unique - * on this partition only, so it is useless for us. - */ - if (!index_constains_partkey(rel, modified_index)) - continue; + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + * + * FIXME We should support indexes on expressions. + */ + if (ind->indexprs != NIL) + continue; - global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); - } + /* + * Adopt the index definition for the parent. + * + * XXX This seems rather weird. We're constructing "artificial" index + * for the partitioned table (kinda like a global index). Can't we + * just have some simpler struct representing it? + */ + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + * + * XXX Can't we do this check before simple_copy_indexinfo_to_parent? + */ + if (!index_constains_partkey(rel, modified_index)) + continue; - if (global_uniq_indexlist != NIL) + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + /* if there are no applicable unique indexes, we're done */ + if (!global_uniq_indexlist) + return; + + /* + * We iterate over the child relations first, and inspect the unique + * indexes for each hild, because this way we can stop early if we + * happen to eliminate all the unique indexes. + */ + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + + /* skip the first index, which is where we got the list from */ + if (is_first) { - foreach(lc, childrels) - { - RelOptInfo *child = lfirst(lc); - if (is_first) - { - is_first = false; - continue; - } - adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); - } - /* Now we have a list of unique index which are exactly same on all childrels, - * Set the UniqueKey just like it is non-partition table - */ - populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + is_first = false; + continue; } + + /* match the unique keys to indexes on this child */ + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + + /* + * If we have eliminated all unique indexes, no point in looking at + * the remaining child relations. + */ + if (!global_uniq_indexlist) + break; } + + /* Now we have a list of unique index which are exactly same on all child + * relations. Set the UniqueKey just like it is non-partition table. + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); } /* * populate_distinctrel_uniquekeys + * Update unique keys for relation produced by DISTINCT. + * + * We can keep all unique keys from the input relations, because DISTINCT + * can only remove rows - it can't duplicate them. Also, the DISTINCT clause + * itself is a unique key, so add that. */ void populate_distinctrel_uniquekeys(PlannerInfo *root, @@ -292,11 +434,13 @@ populate_distinctrel_uniquekeys(PlannerInfo *root, { /* The unique key before the distinct is still valid. */ distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); } /* * populate_grouprel_uniquekeys + * */ void populate_grouprel_uniquekeys(PlannerInfo *root, @@ -305,54 +449,76 @@ populate_grouprel_uniquekeys(PlannerInfo *root, { Query *parse = root->parse; - bool input_ukey_added = false; ListCell *lc; + /* + * XXX Is this actually valid, before checking fro grouping sets? + * The grouping sets may produce duplicate row even with just a single + * input row, I think. + */ if (relation_is_onerow(inputrel)) { add_uniquekey_for_onerow(grouprel); return; } + + /* + * Bail out if there are grouping sets. + * + * XXX Could we maybe inspect the grouping sets and determine if this + * generates distinct combinations? In some cases that's clearly not + * the case (rollup, cube), but for some simple cases it might. + */ if (parse->groupingSets) return; - /* A Normal group by without grouping set. */ - if (parse->groupClause) + /* It has aggregation but without a group by, so only one row returned */ + if (!parse->groupClause) + add_uniquekey_for_onerow(grouprel); + + /* + * A regular group by, without grouping sets. + * + * Obviously, the whole group clause determines a unique key. But if + * there are smaller unique keys on the input rel, we prefer those + * because those are more flexible. If (a,b) is unique, (a,b,c) is + * unique too. Only when there are no such smaller unique keys, we + * add the unique key derived from the group clause. + */ + foreach(lc, inputrel->uniquekeys) { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + /* - * Current even the groupby clause is Unique already, but if query has aggref - * We have to create grouprel still. To keep the UnqiueKey short, we will check - * the UniqueKey of input_rel still valid, if so we reuse it. + * Ignore unique keys on the input that are not subset of the + * group clause. We can't use incomplete unique keys. */ - foreach(lc, inputrel->uniquekeys) - { - UniqueKey *ukey = lfirst_node(UniqueKey, lc); - if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) - { - grouprel->uniquekeys = lappend(grouprel->uniquekeys, - ukey); - input_ukey_added = true; - } - } - if (!input_ukey_added) - /* - * group by clause must be a super-set of grouprel->reltarget->exprs except the - * aggregation expr, so if such exprs is unique already, no bother to generate - * new uniquekey for group by exprs. - */ - add_uniquekey_from_sortgroups(root, - grouprel, - root->parse->groupClause); + if (!list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + continue; + + grouprel->uniquekeys = lappend(grouprel->uniquekeys, ukey); } - else - /* It has aggregation but without a group by, so only one row returned */ - add_uniquekey_for_onerow(grouprel); + + /* + * Group clause must be a super-set of of grouprel->reltarget->exprs, + * except for the aggregation expressions. So if we found a smaller + * unique key on the input relation, don't bother adding a unique key + * for the group clause. + */ + if (!grouprel->uniquekeys) + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); } /* * simple_copy_uniquekeys - * Using a function for the one-line code makes us easy to check where we simply - * copied the uniquekey. + * Copy yhe unique keys between relations. + * + * Using a function for the one-line code makes us easy to check where we + * simply copied the uniquekey. + * + * XXX Seems like an overkill, not sure what's the purpose? */ void simple_copy_uniquekeys(RelOptInfo *oldrel, @@ -362,24 +528,27 @@ simple_copy_uniquekeys(RelOptInfo *oldrel, } /* - * populate_unionrel_uniquekeys + * populate_unionrel_uniquekeys + * Determine unique keys for UNION relation. + * + * XXX Does this need to care about UNION vs. UNION ALL? At least in the + * one-row code path? */ void populate_unionrel_uniquekeys(PlannerInfo *root, - RelOptInfo *unionrel) + RelOptInfo *unionrel) { - ListCell *lc; - List *exprs = NIL; + ListCell *lc; + List *exprs = NIL; Assert(unionrel->uniquekeys == NIL); + /* XXX Why are we copying the expressions? */ foreach(lc, unionrel->reltarget->exprs) - { exprs = lappend(exprs, lfirst(lc)); - } + /* SQL: select union select; is valid, we need to handle it here. */ if (exprs == NIL) - /* SQL: select union select; is valid, we need to handle it here. */ add_uniquekey_for_onerow(unionrel); else unionrel->uniquekeys = lappend(unionrel->uniquekeys, @@ -389,6 +558,7 @@ populate_unionrel_uniquekeys(PlannerInfo *root, /* * populate_joinrel_uniquekeys + * Determine unique keys for a join relation. * * populate uniquekeys for joinrel. We will check each relation to see if its * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to @@ -404,70 +574,99 @@ populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, List *restrictlist, JoinType jointype) { - ListCell *lc, *lc2; - List *clause_list = NIL; - List *outerrel_ukey_ctx; - List *innerrel_ukey_ctx; - bool inner_onerow, outer_onerow; - bool mergejoin_allowed; - - /* Care about the outerrel relation only for SEMI/ANTI join */ + ListCell *lc, + *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, + outer_onerow; + bool mergejoin_allowed; + + /* For SEMI/ANTI join, we care only about the outerrel unique keys. */ if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) { foreach(lc, outerrel->uniquekeys) { UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + + /* Keep the unique key if it's included in the joinrel. */ if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); } + return; } + /* XXX What about JOIN_RIGHT? */ Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); - /* Fast path */ + /* + * For regular joins, we need to combine unique keys from both sides + * of the join, to get a new unique key for the join relation. So if + * either side does not have a unique key, bail out. + */ if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) return; + /* XXX maybe move to the if blocks? Not needed outside. */ inner_onerow = relation_is_onerow(innerrel); outer_onerow = relation_is_onerow(outerrel); - outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); - innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + outerrel_ukey_ctx = initialize_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initialize_uniquecontext_for_joinrel(innerrel); - clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + clause_list = select_mergejoin_clauses(root, + joinrel, outerrel, innerrel, restrictlist, jointype, &mergejoin_allowed); - if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + /* + * XXX Seems a bit weird that it's called innerrel_keeps_unique but we + * seem to use it in both directions. Or what's the "reverse" for? The + * "reverse" name is not particularly descriptive. + */ + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true)) { - bool outer_impact = jointype == JOIN_FULL; + bool outer_impact = (jointype == JOIN_FULL); + + /* Inspect unique keys on the outer relation. */ foreach(lc, outerrel_ukey_ctx) { UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + /* + * If the output of the join does not include all the parts of the + * unique key, it's useless, so mark it accordingly and ignore it. + */ if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) { ctx->useful = false; continue; } - /* Outer relation has one row, and the unique key is not duplicated after join, - * the joinrel will still has one row unless the jointype == JOIN_FULL. + /* + * When the outer relation has one row, and the unique key is not + * duplicated after join, so the joinrel will still have just one + * row unless the jointype == JOIN_FULL. In that case we're done, + * it's the strictest unique key possible. + * + * If it's one-row with a JOIN_FULL, it might produce multiple + * rows with NULLs, so set multi_nullvals. We also need to set + * the exprs correctly since it can't be NIL any more. + * + * For other cases (not one-row relation), we just reuse the + * unique key, but we may need to tweak the multi_nullvals. */ if (outer_onerow && !outer_impact) { add_uniquekey_for_onerow(joinrel); return; } - else if (outer_onerow) + else if (outer_onerow) /* one-row and FULL join */ { - /* - * The onerow outerrel becomes multi rows and multi_nullvals - * will be changed to true. We also need to set the exprs correctly since it - * can't be NIL any more. - */ ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) { joinrel->uniquekeys = lappend(joinrel->uniquekeys, @@ -485,18 +684,38 @@ populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, joinrel->uniquekeys = lappend(joinrel->uniquekeys, ctx->uniquekey); } + + /* + * Mark the unique key as added, so that we can ignore it later + * when combining unique keys from both sides of the join. + */ ctx->added_to_joinrel = true; } } + /* + * XXX Seems this actually checks if "outerrel keeps unique" so the name + * is misleading. Of maybe it's the previous block, not sure. + * + * XXX So why does this consider JOIN_FULL and JOIN_LEFT, while the previous + * block only cares about JOIN_FULL? + * + * XXX This is almost exact copy of the previous block, so maybe make it + * a separate function and just call it twice? + */ if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) { - bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + bool outer_impact = (jointype == JOIN_FULL || jointype == JOIN_LEFT); + /* Inspect unique keys on the inner relation. */ foreach(lc, innerrel_ukey_ctx) { UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + /* + * If the output of the join does not include all the parts of the + * unique key, it's useless, so mark it accordingly and ignore it. + */ if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) { ctx->useful = false; @@ -529,29 +748,52 @@ populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, ctx->uniquekey); } + + /* + * Mark the unique key as added, so that we can ignore it later + * when combining unique keys from both sides of the join. + */ ctx->added_to_joinrel = true; } } /* - * The combination of the UniqueKey from both sides is unique as well regardless - * of join type, but no bother to add it if its subset has been added to joinrel - * already or it is not useful for the joinrel. + * XXX What if either of the previous two conditions did not match? In + * that case we haven't updated the useful flag, and maybe the unique + * key is not useful, but we don't know, right? So we should not be + * using it in the next loop. Or maybe we should evaluate the flag + * before the loops. + */ + + /* + * The combination of the UniqueKey from both sides is unique as well, + * regardless of the join type. But don't bother to add it if its + * subset has been added to joinrel already or when it's not useful for + * the joinrel. + * + * XXX Maybe we should have a flag that both sides have useful keys? + * Or maybe the loops are short/cheap? */ foreach(lc, outerrel_ukey_ctx) { UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + + /* when not useful or already added to the joinrel, skip it */ if (ctx1->added_to_joinrel || !ctx1->useful) continue; + foreach(lc2, innerrel_ukey_ctx) { UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + + /* when not useful or already added to the joinrel, skip it */ if (ctx2->added_to_joinrel || !ctx2->useful) continue; + + /* If we add a onerow UniqueKey, we don't need another key. */ if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, ctx1->uniquekey, ctx2->uniquekey, jointype)) - /* If we set a onerow UniqueKey to joinrel, we don't need other. */ return; } } @@ -560,8 +802,9 @@ populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, /* * convert_subquery_uniquekeys + * Covert the UniqueKey in subquery to outer relation. * - * Covert the UniqueKey in subquery to outer relation. + * XXX Explain what exactly does the conversion do? */ void convert_subquery_uniquekeys(PlannerInfo *root, RelOptInfo *currel, @@ -618,12 +861,14 @@ void convert_subquery_uniquekeys(PlannerInfo *root, /* * innerrel_keeps_unique + * Check if Unique key on the innerrel is valid after join. * - * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey - * will be still valid if innerrel's any-column mergeop outrerel's uniquekey - * exists in clause_list. + * innerrel's UniqueKey will be still valid if innerrel's any-column mergeop + * outrerel's uniquekey exists in clause_list * * Note: the clause_list must be a list of mergeable restrictinfo already. + * + * XXX Misleading name? We seem to use it for "outerrel_keeps_unique" too. */ static bool innerrel_keeps_unique(PlannerInfo *root, @@ -634,26 +879,32 @@ innerrel_keeps_unique(PlannerInfo *root, { ListCell *lc, *lc2, *lc3; + /* XXX probably not needed, duplicate with the check in the caller + * (populate_joinrel_uniquekeys). But it's cheap. */ if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) return false; /* Check if there is outerrel's uniquekey in mergeable clause. */ foreach(lc, outerrel->uniquekeys) { - List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; - bool clauselist_matchs_all_exprs = true; + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) { Node *outer_uq_expr = lfirst(lc2); bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) { RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); Node *outer_expr; + if (reverse) outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); else outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) { find_uq_expr_in_clauselist = true; @@ -677,22 +928,37 @@ innerrel_keeps_unique(PlannerInfo *root, /* * relation_is_onerow - * Check if it is a one-row relation by checking UniqueKey. + * Check if it is a one-row relation by checking UniqueKey. + * + * The one-row is a special case - there has to be just a single unique key, + * with no expressions. */ bool relation_is_onerow(RelOptInfo *rel) { UniqueKey *ukey; - if (rel->uniquekeys == NIL) + + /* there has to be exactly one unique key */ + if (list_length(rel->uniquekeys) != 1) return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); - return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; + + /* the unique key must have no expressions */ + return (ukey->exprs == NIL); } /* * relation_has_uniquekeys_for - * Returns true if we have proofs that 'rel' cannot return multiple rows with - * the same values in each of 'exprs'. Otherwise returns false. + * Determines if the relation has unique key for a list of expressions. + * + * Returns true iff we can prove that the relation cannot return multiple rows + * with the same values in the provided expression. + * + * allow_multinulls determines whether we allow multiple NULL values or not. + * + * The special "one-row" unique key is considered incompatible with all + * possible expressions. */ bool relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, @@ -710,20 +976,39 @@ relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, foreach(lc, rel->uniquekeys) { UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) continue; + if (list_is_subset(ukey->exprs, exprs)) return true; } + return false; } /* * get_exprs_from_uniqueindex + * Return a list of expressions from a unique index. + * + * Provided with a list of expressions and opclass families, we try to match + * it to the index. If useful, we produce a list of index expressions (subset + * of the list we provided). + * + * We simply walk through the index expressions, and for each expression we + * check three things: * - * Return a list of exprs which is unique. set useful to false if this - * unique index is not useful for us. + * 1) If there's a matching (expr = Const) clause, we can simply ignore the + * expressions. Unique index on (a,b,c) guarantees uniqueness on (a,b) when + * there's condition (c=1). + * + * 2) Check that the index expression is present in the relation we're + * dealing with. If not, the unique key would be useless anyway, and the + * index can't produce unique key. + * + * XXX Shouldn't it be enough to return NULL when the index is not useful? + * The extra flag seems a bit unnecessary. */ static List * get_exprs_from_uniqueindex(IndexOptInfo *unique_index, @@ -743,18 +1028,19 @@ get_exprs_from_uniqueindex(IndexOptInfo *unique_index, indexpr_item = list_head(unique_index->indexprs); for(c = 0; c < unique_index->ncolumns; c++) { - int attr = unique_index->indexkeys[c]; - Expr *expr; - bool matched_const = false; - ListCell *lc1, *lc2; + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; - if(attr > 0) + if (attr > 0) { + /* regular attribute, just use the expression from index tlist */ expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; } else if (attr == 0) { - /* Expression index */ + /* expression from the index */ expr = lfirst(indexpr_item); indexpr_item = lnext(unique_index->indexprs, indexpr_item); } @@ -764,29 +1050,43 @@ get_exprs_from_uniqueindex(IndexOptInfo *unique_index, Assert(false); } + /* should have a valid expression now */ + Assert(expr); + /* - * Check index_col = Const case with regarding to opfamily checking - * If we can remove the index_col from the final UniqueKey->exprs. + * Check if there's (index_col = Const) condition, and that it's using + * a compatible opfamily. If yes, we can remove the index_col from the + * final UniqueKey->exprs, because the value is constant (so removing + * it can't introduce duplicities). */ forboth(lc1, const_exprs, lc2, const_expr_opfamilies) { - if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) - && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + List *opfamilies = (List *) lfirst(lc2); + Node *cexpr = (Node *) lfirst(lc1); + + if (list_member_oid(opfamilies, unique_index->opfamily[c]) && + match_index_to_operand(cexpr, c, unique_index)) { matched_const = true; break; } } + /* it's constant, so ignore the expression */ if (matched_const) continue; - /* Check if the indexed expr is used in rel */ + /* + * Check if the indexed expr is used in rel. We do this after the + * (col = Const) check, because nn expression may be in a a restrict + * clause and not in the reltarget. So we don't want to rule out an + * index unnecessarily. + */ if (attr > 0) { /* - * Normal Indexed column, if the col is not used, then the index is useless - * for uniquekey. + * Normal indexed column, if the col is not used, then the index + * is useless for uniquekey. */ attr -= FirstLowInvalidHeapAttributeNumber; @@ -806,67 +1106,85 @@ get_exprs_from_uniqueindex(IndexOptInfo *unique_index, /* check not null property. */ if (attr == 0) { - /* We never know if a expression yields null or not */ + /* We never know if an expression yields null or not */ *multi_nullvals = true; } - else if (!bms_is_member(attr, unique_index->rel->notnullattrs) - && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, - unique_index->rel->notnullattrs)) + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) && + !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) { *multi_nullvals = true; } exprs = lappend(exprs, expr); } + return exprs; } /* * add_uniquekey_for_onerow - * If we are sure that the relation only returns one row, then all the columns - * are unique. However we don't need to create UniqueKey for every column, we - * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo - * since this one has strongest semantics. + * Create a special unique key signifying that the rel has one row. + * + * If we are sure that the relation only returns one row (it might return + * no rows, but we still consider that unique), then all the columns are + * trivially unique. + * + * However we don't need to create UniqueKey with every column, we just + * set exprs = NIL, because that's easier to identify. We don't want to + * add unnecessary unique keys (such that we already have a unique key + * for a subset of the expressions), and with (exprs == NIL) we can just + * assume we have one unique key for each column in the rel. + * + * We discard all other unique keys, since it has the strongest semantics. */ void add_uniquekey_for_onerow(RelOptInfo *rel) { /* - * We overwrite the previous UniqueKey on purpose since this one has the - * strongest semantic. + * We overwrite the previous UniqueKey on purpose since this one has + * the strongest semantic (all other unique keys are implied by it). */ rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); } /* - * initililze_uniquecontext_for_joinrel - * Return a List of UniqueKeyContext for an inputrel + * initialize_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel. */ static List * -initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +initialize_uniquecontext_for_joinrel(RelOptInfo *inputrel) { - List *res = NIL; - ListCell *lc; - foreach(lc, inputrel->uniquekeys) + List *res = NIL; + ListCell *lc; + + foreach(lc, inputrel->uniquekeys) { UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); context->uniquekey = lfirst_node(UniqueKey, lc); context->added_to_joinrel = false; context->useful = true; + res = lappend(res, context); } + return res; } - /* * get_exprs_from_uniquekey - * Unify the way of get List of exprs from a one-row UniqueKey or - * normal UniqueKey. for the onerow case, every expr in rel1 is a valid - * UniqueKey. Return a List of exprs. + * Extract expressions that are part of a unique key. + * + * The meaning of the result is a bit different in regular and one-row cases. + * For the regular case, the list of expressions form a single unique key, + * i.e. the combination of values is unique. + * + * For the one-row case, each individual expression is known to be unique + * (simply because in a single row everything is unique). * * rel1: The relation which you want to get the exprs. * ukey: The UniqueKey you want to get the exprs. @@ -875,27 +1193,29 @@ static List * get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *rel1, UniqueKey *ukey) { - ListCell *lc; - bool onerow = rel1 != NULL && relation_is_onerow(rel1); + ListCell *lc; + List *res = NIL; + bool onerow = (rel1 != NULL) && relation_is_onerow(rel1); - List *res = NIL; + /* We require at least one of those to be true. */ Assert(onerow || ukey); - if (onerow) - { - /* Only cares about the exprs still exist in joinrel */ - foreach(lc, joinrel->reltarget->exprs) - { - Bitmapset *relids = pull_varnos(root, lfirst(lc)); - if (bms_is_subset(relids, rel1->relids)) - { - res = lappend(res, list_make1(lfirst(lc))); - } - } - } - else + + /* if not a one-row unique key, just return the key's expressions */ + if (!onerow) + return list_make1(ukey->exprs); + + /* + * If it's a one-row relation, we simply extract the expressions that + * still exist in the reltarget. + */ + foreach(lc, joinrel->reltarget->exprs) { - res = list_make1(ukey->exprs); + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + + if (bms_is_subset(relids, rel1->relids)) + res = lappend(res, list_make1(lfirst(lc))); } + return res; } @@ -910,55 +1230,67 @@ get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, /* * index_constains_partkey - * return true if the index contains the partiton key. + * Determines if the index includes a partition key. + * + * XXX Surely we already have a code doing this already? E.g. when creating + * a unique index on a partitioned table we define that. */ static bool -index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) { ListCell *lc; int i; + Assert(IS_PARTITIONED_REL(partrel)); Assert(partrel->part_scheme->partnatts > 0); for(i = 0; i < partrel->part_scheme->partnatts; i++) { - Node *part_expr = linitial(partrel->partexprs[i]); - bool found_in_index = false; + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) { - Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) { found_in_index = true; break; } } + if (!found_in_index) return false; } + return true; } /* * simple_indexinfo_equal + * Compare two indexes to determine if they are the same. + * + * We need to do this because simple_copy_indexinfo_to_parent does change + * some elements. So this is not exactly the same as calling equal(). * - * Used to check if the 2 index is same as each other. The index here - * is COPIED from childrel and did some tiny changes(see - * simple_copy_indexinfo_to_parent) + * XXX I wonder if we could simply use equal(), somehow? In fact, we should + * probably build something much simpler than IndexOptInfo, just enough to + * do the checks. */ static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) { Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; - return ind1->ncolumns == ind2->ncolumns && - ind1->unique == ind2->unique && - memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && - memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && - memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && - memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && - equal(get_tlist_exprs(ind1->indextlist, true), - get_tlist_exprs(ind2->indextlist, true)); + return ((ind1->ncolumns == ind2->ncolumns) && + (ind1->unique == ind2->unique) && + (memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0) && + (memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0) && + (memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0) && + (memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0) && + (equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)))); } @@ -981,11 +1313,21 @@ simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) /* - * simple_copy_indexinfo_to_parent (from partition) - * Copy the IndexInfo from child relation to parent relation with some modification, - * which is used to test: - * 1. If the same index exists in all the childrels. + * simple_copy_indexinfo_to_parent + * Copy index info from child to parent, with necessary tweaks. + * + * We use this copy to check: + * + * 1. If the same/matching index exists in all the childrels. * 2. If the parentrel->reltarget/basicrestrict info matches this index. + * + * XXX IMHO we should probably build something much simpler than a full + * IndexOptInfo copy, just enough to do the checks. + * + * XXX The fact that we copy so much data seems wrong, and having to + * define macros from copyfuncs.c seems like a very suspicious thing. + * One reason is that IndeOptInfo is fairly large struct, especially + * with all the fields, and we allocate it very often. */ static IndexOptInfo * simple_copy_indexinfo_to_parent(PlannerInfo *root, @@ -1027,20 +1369,24 @@ simple_copy_indexinfo_to_parent(PlannerInfo *root, /* * adjust_partition_unique_indexlist + * Checks and eliminates indexes that do not exist on the child relation. * - * global_unique_indexes: At the beginning, it contains the copy & modified - * unique index from the first partition. And then check if each index in it still - * exists in the following partitions. If no, remove it. at last, it has an - * index list which exists in all the partitions. + * Walks the list of unique indexes, and eliminates those that don't match + * the child relation (i.e. where a matching child index does not exist). + * This is used to iteratively filter the list of candidate unique keys. + * + * After processing all child relations, the list contains only indexes that + * exist in all the child relations. */ static void adjust_partition_unique_indexlist(PlannerInfo *root, RelOptInfo *parentrel, RelOptInfo *childrel, - List **global_unique_indexes) + List **indexes) { ListCell *lc, *lc2; - foreach(lc, *global_unique_indexes) + + foreach(lc, *indexes) { IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); bool found_in_child = false; @@ -1049,23 +1395,45 @@ adjust_partition_unique_indexlist(PlannerInfo *root, { IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); IndexOptInfo *p_ind_copy; - if (!p_ind->unique || !p_ind->immediate || - (p_ind->indpred != NIL && !p_ind->predOK)) + + /* + * Ignore child indexes that can't possibly match (not unique or + * immediate, etc.) + * + * XXX We do these checks in many places, so maybe turn it into + * a reusable macro? + */ + if ((!p_ind->unique) || (!p_ind->immediate) || + (p_ind->indpred != NIL) && (!p_ind->predOK)) continue; + + /* + * XXX This seems possibly quite expensive. Imagine there are many + * child relations, with a bunch of unique indexes each. Then this + * generates a copy for each unique index in each child relation, + * something like O(N^2/2) copies. + */ p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + + /* Found a matching index for the child relation, we're done. */ if (simple_indexinfo_equal(p_ind_copy, g_ind)) { found_in_child = true; break; } } + + /* No matching index in the child, so remove it from the list. */ if (!found_in_child) - /* The index doesn't exist in childrel, remove it from global_unique_indexes */ - *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + *indexes = foreach_delete_current(*indexes, lc); } } -/* Helper function for groupres/distinctrel */ +/* + * Helper function for groupres/distinctrel + * + * FIXME Not sure about this. + */ static void add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) { @@ -1073,27 +1441,32 @@ add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgrou List *exprs; /* - * XXX: If there are some vars which is not in current levelsup, the semantic is - * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to - * check every level other than 0, if so, looks we have to write another - * pull_var_walker. + * XXX: If there are some vars which are not in the current levelsup, the + * semantic is imprecise, should we avoid it or not? levelsup = 1 is just + * a demo, maybe we need to check every level other than 0, if so, looks + * we have to write another pull_var_walker. */ List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); if (upper_vars != NIL) return; + /* sortgroupclause can't be multi_nullvals */ exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); rel->uniquekeys = lappend(rel->uniquekeys, - makeUniqueKey(exprs, - false /* sortgroupclause can't be multi_nullvals */)); + makeUniqueKey(exprs, false)); } /* * add_combined_uniquekey - * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter - * the jointype. + * Add a unique key for a join, combined from keys on inner/outer side. + * + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no + * matter what's the exact jointype. + * + * Returns true if the unique key is "one-row" variant, so that the caller + * can stop considering further combinations. */ bool add_combined_uniquekey(PlannerInfo *root, @@ -1104,32 +1477,47 @@ add_combined_uniquekey(PlannerInfo *root, UniqueKey *inner_ukey, JoinType jointype) { + bool multi_nullvals; + ListCell *lc1, *lc2; - ListCell *lc1, *lc2; - - /* Either side has multi_nullvals or we have outer join, - * the combined UniqueKey has multi_nullvals */ - bool multi_nullvals = outer_ukey->multi_nullvals || + /* + * If either side has multi_nullvals, or we are dealing with an outer join, + * the combined UniqueKey has multi_nullvals too. + */ + multi_nullvals = outer_ukey->multi_nullvals || inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); /* The only case we can get onerow joinrel after join */ - if (relation_is_onerow(outer_rel) - && relation_is_onerow(inner_rel) - && jointype == JOIN_INNER) + if (relation_is_onerow(outer_rel) && + relation_is_onerow(inner_rel) && + jointype == JOIN_INNER) { add_uniquekey_for_onerow(joinrel); return true; } + /* + * XXX Isn't this wrong? Why is it combining expressions that are part + * of the two unique keys? Imagine we have outer unique key on (a1, a2) + * and inner outer key on (b1, b2). Then this adds four unique keys + * for the join (a1,b1), (a1,b2), (a2,b1) and (a2,b2). Shouldn't it + * just add (a1,a2,b1,b2)? + */ foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) { + /* + * XXX This calls get_exprs_from_uniquekey repeatedly for each outer + * loop. Maybe we should calculate it just once before the loop. + */ foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) { List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, makeUniqueKey(exprs, multi_nullvals)); } } + return false; } diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 8d8e493f5c..f29b65c07b 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,7 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + /* XXX comment? can we simply just copy the unique keys to the final relation? */ simple_copy_uniquekeys(current_rel, final_rel); /* @@ -3902,7 +3903,9 @@ create_grouping_paths(PlannerInfo *root, set_cheapest(grouped_rel); + /* XXX does this apply to grouping sets too? */ populate_grouprel_uniquekeys(root, grouped_rel, input_rel); + return grouped_rel; } @@ -4625,7 +4628,10 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); + + /* XXX comment? */ simple_copy_uniquekeys(input_rel, window_rel); + return window_rel; } @@ -4939,7 +4945,10 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); + + /* XXX comment */ populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); + return distinct_rel; } @@ -5200,6 +5209,7 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + /* XXX comment */ simple_copy_uniquekeys(input_rel, ordered_rel); return ordered_rel; diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index b7626545bf..72a3f3c598 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -691,6 +691,7 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Add the UniqueKeys */ populate_unionrel_uniquekeys(root, result_rel); + return result_rel; } diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index 3eec1f4d74..c9829c5fc4 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -755,6 +755,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, pseudoconstant, rinfo->security_level, NULL, NULL, NULL); + /* XXX This is a bit weird, doing this outside make_restrictinfo */ child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ -- 2.30.2 --------------F495F6B18672582269F00FF9 Content-Type: text/x-patch; charset=UTF-8; name="0005-Extend-UniqueKeys-20210317.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0005-Extend-UniqueKeys-20210317.patch" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 04/10] review @ 2021-03-16 18:26 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-16 18:26 UTC (permalink / raw) --- src/backend/nodes/copyfuncs.c | 4 + src/backend/nodes/makefuncs.c | 1 - src/backend/nodes/outfuncs.c | 1 + src/backend/nodes/readfuncs.c | 2 + src/backend/optimizer/path/README.uniquekey | 285 +++--- src/backend/optimizer/path/allpaths.c | 10 +- src/backend/optimizer/path/joinrels.c | 7 + src/backend/optimizer/path/uniquekeys.c | 906 ++++++++++++++------ src/backend/optimizer/plan/planner.c | 10 + src/backend/optimizer/prep/prepunion.c | 1 + src/backend/optimizer/util/inherit.c | 1 + 11 files changed, 863 insertions(+), 365 deletions(-) diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 75c1c5e824..9d832ddc03 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,9 @@ _copyPathKey(const PathKey *from) return newnode; } +/* + * _copyUniqueKey + */ static UniqueKey * _copyUniqueKey(const UniqueKey *from) { @@ -2306,6 +2309,7 @@ _copyUniqueKey(const UniqueKey *from) return newnode; } + /* * _copyRestrictInfo */ diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 40415d0f5b..e156c9cdf8 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -816,7 +816,6 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) return v; } - /* * makeUniqueKey */ diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 44154cde6a..13905e6037 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2460,6 +2460,7 @@ static void _outUniqueKey(StringInfo str, const UniqueKey *node) { WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); WRITE_BOOL_FIELD(multi_nullvals); } diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index b3e212bf1c..8830c8df99 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -496,8 +496,10 @@ static UniqueKey * _readUniqueKey(void) { READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); } diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey index 5eac761995..31cdb5ed65 100644 --- a/src/backend/optimizer/path/README.uniquekey +++ b/src/backend/optimizer/path/README.uniquekey @@ -1,131 +1,208 @@ -1. What is UniqueKey? -We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure -that doesn't yields same result among all the rows. The simplest UniqueKey -format is primary key. +review comments: +XXX Maybe move this to src/backend/optimizer/README.uniquekey? +XXX multi_nullvals name seems a bit weird +XXX no info about populate_distinctrel_uniquekeys, populate_grouprel_uniquekeys, populate_unionrel_uniquekeys +----- +src/backend/optimizer/path/README.uniquekey + +UniqueKey +========= + +UniqueKey is a set of exprs for a RelOptInfo, which are known to have unique +values on all the rows in the relation. A trivial example is a primary key +defined on a relation - each attributes of the constraint is a unique key. + +We can use this knowledge to perform optimization in a number of places. Some +of the optimizations are fairly obvious, others are less so: + +1. remove DISTINCT node if the clause is unique +2. remove aggregation if group by clause is unique +3. remove_useless_joins +4. reduce_semianti_joins +5. Index Skip Scan (WIP) +6. Aggregation Push-Down without 2-phase aggregation if the join can't + duplicate the aggregated rows. (WIP) + -However we define the UnqiueKey as below. +UniqueKey struct +---------------- -typedef struct UniqueKey -{ +A UnqiueKey is represented by the following struct: + + typedef struct UniqueKey + { NodeTag type; List *exprs; bool multi_nullvals; -} UniqueKey; - -exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL -is a special case of UniqueKey, which means there is only one row in that -relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is -normal unique key and may have different values. SELECT colx FROM t WHERE uk = -const. colx is unique AND we have only 1 value. This field can used for -innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow -function. - -multi_nullvals: true means multi null values may exist in these exprs, so the -uniqueness is not guaranteed in this case. This field is necessary for -remove_useless_join & reduce_unique_semijoins where we don't mind these -duplicated NULL values. It is set to true for 2 cases. One is a unique key -from a unique index but the related column is nullable. The other one is for -outer join. see populate_joinrel_uniquekeys for detail. - - -The UniqueKey can be used at the following cases at least: -1. remove_useless_joins. -2. reduce_semianti_joins -3. remove distinct node if distinct clause is unique. -4. remove aggnode if group by clause is unique. -5. Index Skip Scan (WIP) -6. Aggregation Push Down without 2 phase aggregation if the join can't - duplicated the aggregated rows. (work in progress feature) - -2. How is it maintained? - -We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on -various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, -groupedrel, unionrel. and we also need to convert the uniquekey from subquery -to outer relation, which is what convert_subquery_uniquekeys does. - -1. The first part is about baserel. We handled 3 cases. suppose we have Unique -Index on (a, b). - -1. SELECT a, b FROM t. UniqueKey (a, b) -2. SELECT a FROM t WHERE b = 1; UniqueKey (a) -3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every - column is Unique. - -2. The next part is joinrel, this part is most error-prone, we simplified the rules -like below: -1. If the relation's UniqueKey can't be duplicated after join, then is will be - still valid for the join rel. The function we used here is - innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. - -2. If the UnqiueKey can't keep valid via the rule 1, the combination of the - UniqueKey from both sides are valid for sure. We can prove this as: if the - unique exprs from rel1 is duplicated by rel2, the duplicated rows must - contains different unique exprs from rel2. - -More considerations about onerow: -1. If relation with one row and it can't be duplicated, it is still possible - contains mulit_nullvas after outer join. -2. If the either UniqueKey can be duplicated after join, the can get one row - only when both side is one row AND there is no outer join. -3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one - row UniqueKey to normal unique key since we don't store exprs for one-row - relation. get_exprs_from_uniquekeys will be used here. - - -More considerations about multi_nullvals after join: + } UniqueKey; + +exprs is a list of exprs which are know to be unique on current RelOptInfo. + +exprs = NIL is a special case, meaning there is only one row in the relation. +This has has a stronger semantic than others. Consider for example + + SELECT uk FROM t + +where 'uk' is a unique key. This guarantees uniqueness, but there may be mamy +rows in the relation. On the other hand, consider this query + + SELECT colx FROM t WHERE uk = const + +In this case we know there's only a single matching row (thanks to a condition +on the unique key), which in turn guarantees uniqueness of the colx value, even +if there is no constraint on the column itself. + +This knowledge is used in innerrel_is_unique, and is handled as a special case +in add_uniquekey_for_onerow. + + +The multi_nullvals field tracks whether the expressions may contain multiple +NULL values. This can happen for example when the unique key is derived from +a unique index with nullable columns, or because of outer joins (which may add +NULL values to a known-unique list - see populate_joinrel_uniquekeys). + +In this case uniqueness is not guaranteed, but we can still use the information +in places places where NULL values are harmless - when removing useless joins, +reducing semijoins, and so on. + + +How is it maintained? +--------------------- + +Deducing the unique keys depends on the type of the relation - for each case +there's a separate "populate" function: + + +populate_baserel_uniquekeys +--------------------------- + +There are three cases, all assuming there's a unique index (e.g. on (a,b)): + +1. SELECT a, b FROM t => UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1 => UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; => UniqueKey (NIL) + +The last query is the "one row" case, in which case every column is Unique. + + +populate_joinrel_uniquekeys +--------------------------- + +For joins, deducing the unique keys may be fairly complex and error-prone. +We've simplified the rules like this: + +1. If the UniqueKey on an input relation can't be duplicated by the join, then +it will be valid for the join rel. A typical example is a join like this: + + inner_rel.any_col = outer_rel.unique_key + +The function used to detect this is innerrel_keeps_unique. + +2. Any combination of unique keys on each side of the join is a unique key +for the join relation. This can be proved by contradiction - assume we have +unique key on either side of the join - uk1 and uk2. If the values in uk1 get +duplicated by the join with uk2 (by matching the row to multiple rows), the +duplicated rows must have different values in the uk2. + +We can also leverage information about the "one row" case: + +1. If one of the input relations is known to have a single row, and the join +can't duplicate the row (e.g. semi/anti join), we can keep the unique keys. +It may however contain multi_nullvals after an outer join. + +XXX Not sure I understand the original logic/wording :-( + +2. If either UniqueKey can be duplicated after a join, there can be only one +row only when both sides are "one row" AND there is no outer join. + +XXX Why the restriction on not allowing outer joins? + +3. Whenever the one row UniqueKey is not a valid any more, we need to convert +UniqueKey to normal unique key since we don't store exprs for one-row relation. +This is done by get_exprs_from_uniquekeys. + +The join case needs to be careful about multi_nullvals too: + 1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have - mulit_nullvals in any case. -2. If a unique key doesn't allow mulit_nullvals, after some outer join, it - allows some outer join. +mulit_nullvals in any case too. + +2. If the original unique key doesn't allow multi_nullvals, the unique key for +the join relation may allow multi_nullvals after an outer join. + + +subqueries +---------- + +It's necessary to "translate" unique keys between a subquery and the outer rels, +which is what convert_subquery_uniquekeys does. This does almost exactly what +convert_subquery_pathkeys does for pathkeys. It keeps only unique keys matching +Vars in the outer relation. The relationship between outerrel.Var and +subquery.exprs is built from outerel->subroot->processed_tlist. -3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like -convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as -a Var in outer relation will be reused. The relationship between the outerrel.Var -and subquery.exprs is built with outerel->subroot->processed_tlist. +set-returning functions +------------------------ +As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. So +we will maintain the UniqueKey until there and reset it to NIL at that place. -4. As for the SRF functions, it will break the uniqueness of uniquekey, However it -is handled in adjust_paths_for_srfs, which happens after the query_planner. so -we will maintain the UniqueKey until there and reset it to NIL at that -places. This can't help on distinct/group by elimination cases but probably help -in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is -semantic correctly. +This can't help on distinct/group by elimination cases but probably help in some +other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is correct. -5. As for inherit table, we first main the UnqiueKey on childrel as well. But for -partitioned table we need to maintain 2 different kinds of -UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child -relation for partition wise query. +populate_partitionedrel_uniquekeys +---------------------------------- + +As for inherit table, we first build the UnqiueKey on childrel as well. But for +partitioned table we need to maintain two different kinds of UniqueKey: + +1) UniqueKey on the parent relation + +2) UniqueKey on child + +This is needed because a unique key from the partition may not be be unique key +on the partitioned table. + Example: -CREATE TABLE p (a int not null, b int not null) partition by list (a); + +CREATE TABLE p (a INT NOT NULL, b INT NOT NULL) PARTITION BY LIST (a); + CREATE TABLE p0 partition of p for values in (1); CREATE TABLE p1 partition of p for values in (2); -create unique index p0_b on p0(b); -create unique index p1_b on p1(b); +CREATE UNIQUE INDEX p0_b ON p0(b); +CREATE UNIQUE INDEX p1_b ON p1(b); -Now b is only unique on partition level, so the distinct can't be removed on -the following cases. SELECT DISTINCT b FROM p; +SELECT DISTINCT b FROM p; -Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one -partition is chosen, the UniqueKey on child relation is same as the UniqueKey on -parent relation. +Now "b" is only unique on partition level, but the two partitions may contain +duplicate values for the "b" column (with different values in "a"). That means +the DISTINCT clause can't be removed. -Another usage of UniqueKey on partition level is it be helpful for -partition-wise join. +Now consider: -As for the UniqueKey on parent table level, it comes with 2 different ways, -1). the UniqueKey is also derived in UniqueKey index, but the index must be same -in all the related children relations and the unique index must contains -Partition Key in it. Example: +SELECT DISTINCT a, b FROM p WHERE a = 1 + +In this case, the optimizer eliminates all partitions except for one, so that +the UniqueKey is valid for the parent relation too. + +UniqueKey at a partition level is useful for partition-wise join too. + +XXX Explain why is it useful? + +A UniqueKey from a partition can be transferred to the parent relation, in two +cases. A trivial case is if there's a single child relation (e.g. thanks to +partition elimination). In that case all unique keys on the child relation are +automatically valid for the parent relation. If there are multiple relations, +the unique key must be defived from an index present in all partitions, and the +index has to include the partition key. + +Example: CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. -- Query SELECT a, b FROM p; the (a, b) is a UniqueKey of p. -2). If the parent relation has only one childrel, the UniqueKey on childrel is - the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 66bf6f19f7..a801707eaa 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -581,7 +581,8 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) /* * Now that we've marked which partial indexes are suitable, we can now - * build the relation's unique keys. + * build the relation's unique keys. We need to do it in this order, + * so that we don't deduce unique keys from inapplicable partial indexes. */ populate_baserel_uniquekeys(root, rel, rel->indexlist); @@ -1305,6 +1306,12 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + + /* + * XXX Maybe move the check into populate populate_partitionedrel_uniquekeys? + * XXX What if it's append rel (but not partitioned one), but there's only one + * child relation? We could still deduce unique keys, no? + */ if (IS_PARTITIONED_REL(rel)) populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2314,6 +2321,7 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + /* Convert subpath's unique keys to outer representation */ convert_subquery_uniquekeys(root, rel, sub_final_rel); /* If outer rel allows parallelism, do same for partial paths. */ diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 7271f044ec..eefba449d6 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -925,6 +925,13 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + /* + * Determine which of the unique keys from input relations are applicable + * for the join result. + * + * XXX We do this after trying the partitionwise join, because that may allow + * using additional unique keys. + */ populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c index 77ed2b2eff..114e8334f5 100644 --- a/src/backend/optimizer/path/uniquekeys.c +++ b/src/backend/optimizer/path/uniquekeys.c @@ -36,7 +36,7 @@ typedef struct UniqueKeyContextData bool useful; } *UniqueKeyContext; -static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static List *initialize_uniquecontext_for_joinrel(RelOptInfo *inputrel); static bool innerrel_keeps_unique(PlannerInfo *root, RelOptInfo *outerrel, RelOptInfo *innerrel, @@ -80,8 +80,20 @@ static void add_uniquekey_from_sortgroups(PlannerInfo *root, /* * populate_baserel_uniquekeys - * Populate 'baserel' uniquekeys list by looking at the rel's unique index - * and baserestrictinfo + * Build list of unique keys for the base relation. + * + * Inspects unique indexes defined on the relation and determines what + * unique keys are valid. Partial indexes are considered too, if the + * predicate is valid. + * + * This also inspects baserestrictinfo, because we need to determine + * which opclass families are interesting when inspecting indexes. If we + * have a unique index and distinct clause with a mismatching opclasses, + * we should not use that. + * + * XXX Why does this look at baserestrictinfo? + * + * XXX What about collations? */ void populate_baserel_uniquekeys(PlannerInfo *root, @@ -99,22 +111,48 @@ populate_baserel_uniquekeys(PlannerInfo *root, Assert(baserel->rtekind == RTE_RELATION); + if (!indexlist) + return; + + /* + * Determine which unique indexes to use to build the unique keys. + * We have to skip partial with predicates not matched by the query, + * and unique indexes that are not immediately enforced. + * + * XXX Do we actually skip indexes that are not immediate? + * XXX What about hypothetical indexes? + */ foreach(lc, indexlist) { IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || (ind->indpred != NIL && !ind->predOK)) continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); } + /* If there are not applicable unique indexes, we're done. */ if (matched_uniq_indexes == NIL) return; - /* Check which attrs is used in baserel->reltarget */ - pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + /* + * Determine which attrs are referenced in baserel->reltarget. To use the + * unique key info, we need all the columns - a unique index on (a,b) may + * not be unique on (a). If a column is missing in reltarget, the nodes + * above can't possibly use it, and we can just ignore any matching index. + */ + pull_varattnos((Node *) baserel->reltarget->exprs, baserel->relid, &used_attrs); - /* Check which attrno is used at a mergeable const filter */ + /* + * Check which attrno is used at a mergeable const filter + * + * XXX This is not lookint att attrno at all, maybe obsolete comment? + * + * Seems the primary purpose of this is determining which opclass + * families to use when matching unique indexes in the next loop? + */ foreach(lc, baserel->baserestrictinfo) { RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); @@ -122,6 +160,10 @@ populate_baserel_uniquekeys(PlannerInfo *root, if (rinfo->mergeopfamilies == NIL) continue; + /* + * XXX What if bms_is_empty is true for both left_relids/right_relids? + * Or what if it's false in both cases? + */ if (bms_is_empty(rinfo->left_relids)) { const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); @@ -136,40 +178,69 @@ populate_baserel_uniquekeys(PlannerInfo *root, expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); } + /* + * Now try to match unique indexes to attributes in reltarget, and to + * merge operator families. The index may be on the right attributes, + * but if it's not matching the opfamily it's useless. + * + * XXX Can we have multiple baserestrictinfo for the same attribute, + * with different opfamilies? Probably not. + */ foreach(lc, matched_uniq_indexes) { - bool multi_nullvals, useful; - List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), - const_exprs, - expr_opfamilies, - used_attrs, - &useful, - &multi_nullvals); - if (useful) + bool multi_nullvals, + useful; + + IndexOptInfo *index_info = (IndexOptInfo *) lfirst_node(IndexOptInfo, lc); + + List *exprs = get_exprs_from_uniqueindex(index_info, + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + + if (!useful) + continue; + + /* + * All the columns in Unique Index matched with a restrictinfo, so + * that we know there's just a one row in the result. If we find + * such index, we're done - we discard all other unique keys and + * keep just this special one. In principle, this is a stronger + * guarantee, because all subsets of one row are still unique. + * + * XXX Is it correct to just return? Doesn't that prevent some + * optimizations that might be possible with the other keys? + */ + if (exprs == NIL) { - if (exprs == NIL) - { - /* All the columns in Unique Index matched with a restrictinfo */ - add_uniquekey_for_onerow(baserel); - return; - } - baserel->uniquekeys = lappend(baserel->uniquekeys, - makeUniqueKey(exprs, multi_nullvals)); + /* discards all previous uniquekeys */ + add_uniquekey_for_onerow(baserel); + return; } + + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); } } /* * populate_partitionedrel_uniquekeys - * The UniqueKey on partitionrel comes from 2 cases: - * 1). Only one partition is involved in this query, the unique key can be - * copied to parent rel from childrel. - * 2). There are some unique index which includes partition key and exists - * in all the related partitions. - * We never mind rule 2 if we hit rule 1. + * Determine unique keys for a partitioned relation. + * + * Inspects unique keys for all partitions and derives unique keys that + * are valid for the whole partitioned table. There are two basic cases: + * + * 1) There's only one remaining partition (thanks to pruning all other + * partitions). In this case all the unique keys from the partition are + * trivially valid for the partitioned table. + * + * 2) All the partitions have the same unique index (on the same set of + * columns), and the index includes the partition key. This ensures the + * combination of values is unique for the whole partitioned table. */ - void populate_partitionedrel_uniquekeys(PlannerInfo *root, RelOptInfo *rel, @@ -180,110 +251,181 @@ populate_partitionedrel_uniquekeys(PlannerInfo *root, RelOptInfo *childrel; bool is_first = true; + /* XXX What about append rels? At least for the one-child case? */ Assert(IS_PARTITIONED_REL(rel)); + /* if there are no child relations, we're done. */ if (childrels == NIL) return; /* - * If there is only one partition used in this query, the UniqueKey in childrel is - * still valid in parent level, but we need convert the format from child expr to - * parent expr. + * If there is only one partition used in this query, the UniqueKey for + * a child relation is still valid for the parent level. We need to + * convert the format from child expr to parent expr. */ if (list_length(childrels) == 1) { - /* Check for Rule 1 */ RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + + /* If the partition has a single row, so does the parent. */ if (relation_is_onerow(childrel)) { add_uniquekey_for_onerow(rel); return; } + /* + * Inspect the unique keys one by one, try reusing them for the + * parent relation. + * + * FIXME This needs more work to handle expressions and not just + * simple Vars. + */ foreach(lc, childrel->uniquekeys) { + ListCell *lc2; + List *parent_exprs = NIL; + bool can_reuse = true; + UniqueKey *ukey = lfirst_node(UniqueKey, lc); AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); - List *parent_exprs = NIL; - bool can_reuse = true; - ListCell *lc2; + + /* + * XXX Not sure what exactly we do here. Surely we deal with + * expressions at child/parent level elsewhere? Can't we just + * copy the code from there? + */ foreach(lc2, ukey->exprs) { - Var *var = (Var *)lfirst(lc2); + Var *var = (Var *) lfirst(lc2); + /* - * If the expr comes from a expression, it is hard to build the expression - * in parent so ignore that case for now. + * XXX For now this only supports simple Var expressions, + * so if there's a more complex expression we'll not copy + * the unique key to the parent. */ if(!IsA(var, Var)) { can_reuse = false; break; } + /* Convert it to parent var */ - parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + parent_exprs = lappend(parent_exprs, + find_parent_var(appinfo, var)); } - if (can_reuse) - rel->uniquekeys = lappend(rel->uniquekeys, - makeUniqueKey(parent_exprs, - ukey->multi_nullvals)); + + /* ignore unique keys with complex expressions */ + if (!can_reuse) + continue; + + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); } + + return; } - else + + /* + * A parent with multiple child relations. We only care about indexes that + * are in all child relations, so we loop through indexes on the first one + * and check that they exist in the other child relations too. + */ + + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) { - /* Check for rule 2 */ - childrel = linitial_node(RelOptInfo, childrels); - foreach(lc, childrel->indexlist) - { - IndexOptInfo *ind = lfirst(lc); - IndexOptInfo *modified_index; - if (!ind->unique || !ind->immediate || - (ind->indpred != NIL && !ind->predOK)) - continue; + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; - /* - * During simple_copy_indexinfo_to_parent, we need to convert var from - * child var to parent var, index on expression is too complex to handle. - * so ignore it for now. - */ - if (ind->indexprs != NIL) - continue; + /* + * Ignore indexes that are not unique, immediately enforced. Partial + * indexes with mismatched predicate are useless too. + */ + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; - modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); - /* - * If the unique index doesn't contain partkey, then it is unique - * on this partition only, so it is useless for us. - */ - if (!index_constains_partkey(rel, modified_index)) - continue; + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + * + * FIXME We should support indexes on expressions. + */ + if (ind->indexprs != NIL) + continue; - global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); - } + /* + * Adopt the index definition for the parent. + * + * XXX This seems rather weird. We're constructing "artificial" index + * for the partitioned table (kinda like a global index). Can't we + * just have some simpler struct representing it? + */ + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + * + * XXX Can't we do this check before simple_copy_indexinfo_to_parent? + */ + if (!index_constains_partkey(rel, modified_index)) + continue; - if (global_uniq_indexlist != NIL) + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + /* if there are no applicable unique indexes, we're done */ + if (!global_uniq_indexlist) + return; + + /* + * We iterate over the child relations first, and inspect the unique + * indexes for each hild, because this way we can stop early if we + * happen to eliminate all the unique indexes. + */ + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + + /* skip the first index, which is where we got the list from */ + if (is_first) { - foreach(lc, childrels) - { - RelOptInfo *child = lfirst(lc); - if (is_first) - { - is_first = false; - continue; - } - adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); - } - /* Now we have a list of unique index which are exactly same on all childrels, - * Set the UniqueKey just like it is non-partition table - */ - populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + is_first = false; + continue; } + + /* match the unique keys to indexes on this child */ + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + + /* + * If we have eliminated all unique indexes, no point in looking at + * the remaining child relations. + */ + if (!global_uniq_indexlist) + break; } + + /* Now we have a list of unique index which are exactly same on all child + * relations. Set the UniqueKey just like it is non-partition table. + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); } /* * populate_distinctrel_uniquekeys + * Update unique keys for relation produced by DISTINCT. + * + * We can keep all unique keys from the input relations, because DISTINCT + * can only remove rows - it can't duplicate them. Also, the DISTINCT clause + * itself is a unique key, so add that. */ void populate_distinctrel_uniquekeys(PlannerInfo *root, @@ -292,11 +434,13 @@ populate_distinctrel_uniquekeys(PlannerInfo *root, { /* The unique key before the distinct is still valid. */ distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); } /* * populate_grouprel_uniquekeys + * */ void populate_grouprel_uniquekeys(PlannerInfo *root, @@ -305,54 +449,76 @@ populate_grouprel_uniquekeys(PlannerInfo *root, { Query *parse = root->parse; - bool input_ukey_added = false; ListCell *lc; + /* + * XXX Is this actually valid, before checking fro grouping sets? + * The grouping sets may produce duplicate row even with just a single + * input row, I think. + */ if (relation_is_onerow(inputrel)) { add_uniquekey_for_onerow(grouprel); return; } + + /* + * Bail out if there are grouping sets. + * + * XXX Could we maybe inspect the grouping sets and determine if this + * generates distinct combinations? In some cases that's clearly not + * the case (rollup, cube), but for some simple cases it might. + */ if (parse->groupingSets) return; - /* A Normal group by without grouping set. */ - if (parse->groupClause) + /* It has aggregation but without a group by, so only one row returned */ + if (!parse->groupClause) + add_uniquekey_for_onerow(grouprel); + + /* + * A regular group by, without grouping sets. + * + * Obviously, the whole group clause determines a unique key. But if + * there are smaller unique keys on the input rel, we prefer those + * because those are more flexible. If (a,b) is unique, (a,b,c) is + * unique too. Only when there are no such smaller unique keys, we + * add the unique key derived from the group clause. + */ + foreach(lc, inputrel->uniquekeys) { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + /* - * Current even the groupby clause is Unique already, but if query has aggref - * We have to create grouprel still. To keep the UnqiueKey short, we will check - * the UniqueKey of input_rel still valid, if so we reuse it. + * Ignore unique keys on the input that are not subset of the + * group clause. We can't use incomplete unique keys. */ - foreach(lc, inputrel->uniquekeys) - { - UniqueKey *ukey = lfirst_node(UniqueKey, lc); - if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) - { - grouprel->uniquekeys = lappend(grouprel->uniquekeys, - ukey); - input_ukey_added = true; - } - } - if (!input_ukey_added) - /* - * group by clause must be a super-set of grouprel->reltarget->exprs except the - * aggregation expr, so if such exprs is unique already, no bother to generate - * new uniquekey for group by exprs. - */ - add_uniquekey_from_sortgroups(root, - grouprel, - root->parse->groupClause); + if (!list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + continue; + + grouprel->uniquekeys = lappend(grouprel->uniquekeys, ukey); } - else - /* It has aggregation but without a group by, so only one row returned */ - add_uniquekey_for_onerow(grouprel); + + /* + * Group clause must be a super-set of of grouprel->reltarget->exprs, + * except for the aggregation expressions. So if we found a smaller + * unique key on the input relation, don't bother adding a unique key + * for the group clause. + */ + if (!grouprel->uniquekeys) + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); } /* * simple_copy_uniquekeys - * Using a function for the one-line code makes us easy to check where we simply - * copied the uniquekey. + * Copy yhe unique keys between relations. + * + * Using a function for the one-line code makes us easy to check where we + * simply copied the uniquekey. + * + * XXX Seems like an overkill, not sure what's the purpose? */ void simple_copy_uniquekeys(RelOptInfo *oldrel, @@ -362,24 +528,27 @@ simple_copy_uniquekeys(RelOptInfo *oldrel, } /* - * populate_unionrel_uniquekeys + * populate_unionrel_uniquekeys + * Determine unique keys for UNION relation. + * + * XXX Does this need to care about UNION vs. UNION ALL? At least in the + * one-row code path? */ void populate_unionrel_uniquekeys(PlannerInfo *root, - RelOptInfo *unionrel) + RelOptInfo *unionrel) { - ListCell *lc; - List *exprs = NIL; + ListCell *lc; + List *exprs = NIL; Assert(unionrel->uniquekeys == NIL); + /* XXX Why are we copying the expressions? */ foreach(lc, unionrel->reltarget->exprs) - { exprs = lappend(exprs, lfirst(lc)); - } + /* SQL: select union select; is valid, we need to handle it here. */ if (exprs == NIL) - /* SQL: select union select; is valid, we need to handle it here. */ add_uniquekey_for_onerow(unionrel); else unionrel->uniquekeys = lappend(unionrel->uniquekeys, @@ -389,6 +558,7 @@ populate_unionrel_uniquekeys(PlannerInfo *root, /* * populate_joinrel_uniquekeys + * Determine unique keys for a join relation. * * populate uniquekeys for joinrel. We will check each relation to see if its * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to @@ -404,70 +574,99 @@ populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, List *restrictlist, JoinType jointype) { - ListCell *lc, *lc2; - List *clause_list = NIL; - List *outerrel_ukey_ctx; - List *innerrel_ukey_ctx; - bool inner_onerow, outer_onerow; - bool mergejoin_allowed; - - /* Care about the outerrel relation only for SEMI/ANTI join */ + ListCell *lc, + *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, + outer_onerow; + bool mergejoin_allowed; + + /* For SEMI/ANTI join, we care only about the outerrel unique keys. */ if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) { foreach(lc, outerrel->uniquekeys) { UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + + /* Keep the unique key if it's included in the joinrel. */ if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); } + return; } + /* XXX What about JOIN_RIGHT? */ Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); - /* Fast path */ + /* + * For regular joins, we need to combine unique keys from both sides + * of the join, to get a new unique key for the join relation. So if + * either side does not have a unique key, bail out. + */ if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) return; + /* XXX maybe move to the if blocks? Not needed outside. */ inner_onerow = relation_is_onerow(innerrel); outer_onerow = relation_is_onerow(outerrel); - outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); - innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + outerrel_ukey_ctx = initialize_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initialize_uniquecontext_for_joinrel(innerrel); - clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + clause_list = select_mergejoin_clauses(root, + joinrel, outerrel, innerrel, restrictlist, jointype, &mergejoin_allowed); - if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + /* + * XXX Seems a bit weird that it's called innerrel_keeps_unique but we + * seem to use it in both directions. Or what's the "reverse" for? The + * "reverse" name is not particularly descriptive. + */ + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true)) { - bool outer_impact = jointype == JOIN_FULL; + bool outer_impact = (jointype == JOIN_FULL); + + /* Inspect unique keys on the outer relation. */ foreach(lc, outerrel_ukey_ctx) { UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + /* + * If the output of the join does not include all the parts of the + * unique key, it's useless, so mark it accordingly and ignore it. + */ if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) { ctx->useful = false; continue; } - /* Outer relation has one row, and the unique key is not duplicated after join, - * the joinrel will still has one row unless the jointype == JOIN_FULL. + /* + * When the outer relation has one row, and the unique key is not + * duplicated after join, so the joinrel will still have just one + * row unless the jointype == JOIN_FULL. In that case we're done, + * it's the strictest unique key possible. + * + * If it's one-row with a JOIN_FULL, it might produce multiple + * rows with NULLs, so set multi_nullvals. We also need to set + * the exprs correctly since it can't be NIL any more. + * + * For other cases (not one-row relation), we just reuse the + * unique key, but we may need to tweak the multi_nullvals. */ if (outer_onerow && !outer_impact) { add_uniquekey_for_onerow(joinrel); return; } - else if (outer_onerow) + else if (outer_onerow) /* one-row and FULL join */ { - /* - * The onerow outerrel becomes multi rows and multi_nullvals - * will be changed to true. We also need to set the exprs correctly since it - * can't be NIL any more. - */ ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) { joinrel->uniquekeys = lappend(joinrel->uniquekeys, @@ -485,18 +684,38 @@ populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, joinrel->uniquekeys = lappend(joinrel->uniquekeys, ctx->uniquekey); } + + /* + * Mark the unique key as added, so that we can ignore it later + * when combining unique keys from both sides of the join. + */ ctx->added_to_joinrel = true; } } + /* + * XXX Seems this actually checks if "outerrel keeps unique" so the name + * is misleading. Of maybe it's the previous block, not sure. + * + * XXX So why does this consider JOIN_FULL and JOIN_LEFT, while the previous + * block only cares about JOIN_FULL? + * + * XXX This is almost exact copy of the previous block, so maybe make it + * a separate function and just call it twice? + */ if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) { - bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + bool outer_impact = (jointype == JOIN_FULL || jointype == JOIN_LEFT); + /* Inspect unique keys on the inner relation. */ foreach(lc, innerrel_ukey_ctx) { UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + /* + * If the output of the join does not include all the parts of the + * unique key, it's useless, so mark it accordingly and ignore it. + */ if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) { ctx->useful = false; @@ -529,29 +748,52 @@ populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, ctx->uniquekey); } + + /* + * Mark the unique key as added, so that we can ignore it later + * when combining unique keys from both sides of the join. + */ ctx->added_to_joinrel = true; } } /* - * The combination of the UniqueKey from both sides is unique as well regardless - * of join type, but no bother to add it if its subset has been added to joinrel - * already or it is not useful for the joinrel. + * XXX What if either of the previous two conditions did not match? In + * that case we haven't updated the useful flag, and maybe the unique + * key is not useful, but we don't know, right? So we should not be + * using it in the next loop. Or maybe we should evaluate the flag + * before the loops. + */ + + /* + * The combination of the UniqueKey from both sides is unique as well, + * regardless of the join type. But don't bother to add it if its + * subset has been added to joinrel already or when it's not useful for + * the joinrel. + * + * XXX Maybe we should have a flag that both sides have useful keys? + * Or maybe the loops are short/cheap? */ foreach(lc, outerrel_ukey_ctx) { UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + + /* when not useful or already added to the joinrel, skip it */ if (ctx1->added_to_joinrel || !ctx1->useful) continue; + foreach(lc2, innerrel_ukey_ctx) { UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + + /* when not useful or already added to the joinrel, skip it */ if (ctx2->added_to_joinrel || !ctx2->useful) continue; + + /* If we add a onerow UniqueKey, we don't need another key. */ if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, ctx1->uniquekey, ctx2->uniquekey, jointype)) - /* If we set a onerow UniqueKey to joinrel, we don't need other. */ return; } } @@ -560,8 +802,9 @@ populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, /* * convert_subquery_uniquekeys + * Covert the UniqueKey in subquery to outer relation. * - * Covert the UniqueKey in subquery to outer relation. + * XXX Explain what exactly does the conversion do? */ void convert_subquery_uniquekeys(PlannerInfo *root, RelOptInfo *currel, @@ -618,12 +861,14 @@ void convert_subquery_uniquekeys(PlannerInfo *root, /* * innerrel_keeps_unique + * Check if Unique key on the innerrel is valid after join. * - * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey - * will be still valid if innerrel's any-column mergeop outrerel's uniquekey - * exists in clause_list. + * innerrel's UniqueKey will be still valid if innerrel's any-column mergeop + * outrerel's uniquekey exists in clause_list * * Note: the clause_list must be a list of mergeable restrictinfo already. + * + * XXX Misleading name? We seem to use it for "outerrel_keeps_unique" too. */ static bool innerrel_keeps_unique(PlannerInfo *root, @@ -634,26 +879,32 @@ innerrel_keeps_unique(PlannerInfo *root, { ListCell *lc, *lc2, *lc3; + /* XXX probably not needed, duplicate with the check in the caller + * (populate_joinrel_uniquekeys). But it's cheap. */ if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) return false; /* Check if there is outerrel's uniquekey in mergeable clause. */ foreach(lc, outerrel->uniquekeys) { - List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; - bool clauselist_matchs_all_exprs = true; + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) { Node *outer_uq_expr = lfirst(lc2); bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) { RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); Node *outer_expr; + if (reverse) outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); else outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) { find_uq_expr_in_clauselist = true; @@ -677,22 +928,37 @@ innerrel_keeps_unique(PlannerInfo *root, /* * relation_is_onerow - * Check if it is a one-row relation by checking UniqueKey. + * Check if it is a one-row relation by checking UniqueKey. + * + * The one-row is a special case - there has to be just a single unique key, + * with no expressions. */ bool relation_is_onerow(RelOptInfo *rel) { UniqueKey *ukey; - if (rel->uniquekeys == NIL) + + /* there has to be exactly one unique key */ + if (list_length(rel->uniquekeys) != 1) return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); - return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; + + /* the unique key must have no expressions */ + return (ukey->exprs == NIL); } /* * relation_has_uniquekeys_for - * Returns true if we have proofs that 'rel' cannot return multiple rows with - * the same values in each of 'exprs'. Otherwise returns false. + * Determines if the relation has unique key for a list of expressions. + * + * Returns true iff we can prove that the relation cannot return multiple rows + * with the same values in the provided expression. + * + * allow_multinulls determines whether we allow multiple NULL values or not. + * + * The special "one-row" unique key is considered incompatible with all + * possible expressions. */ bool relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, @@ -710,20 +976,39 @@ relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, foreach(lc, rel->uniquekeys) { UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) continue; + if (list_is_subset(ukey->exprs, exprs)) return true; } + return false; } /* * get_exprs_from_uniqueindex + * Return a list of expressions from a unique index. + * + * Provided with a list of expressions and opclass families, we try to match + * it to the index. If useful, we produce a list of index expressions (subset + * of the list we provided). + * + * We simply walk through the index expressions, and for each expression we + * check three things: * - * Return a list of exprs which is unique. set useful to false if this - * unique index is not useful for us. + * 1) If there's a matching (expr = Const) clause, we can simply ignore the + * expressions. Unique index on (a,b,c) guarantees uniqueness on (a,b) when + * there's condition (c=1). + * + * 2) Check that the index expression is present in the relation we're + * dealing with. If not, the unique key would be useless anyway, and the + * index can't produce unique key. + * + * XXX Shouldn't it be enough to return NULL when the index is not useful? + * The extra flag seems a bit unnecessary. */ static List * get_exprs_from_uniqueindex(IndexOptInfo *unique_index, @@ -743,18 +1028,19 @@ get_exprs_from_uniqueindex(IndexOptInfo *unique_index, indexpr_item = list_head(unique_index->indexprs); for(c = 0; c < unique_index->ncolumns; c++) { - int attr = unique_index->indexkeys[c]; - Expr *expr; - bool matched_const = false; - ListCell *lc1, *lc2; + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; - if(attr > 0) + if (attr > 0) { + /* regular attribute, just use the expression from index tlist */ expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; } else if (attr == 0) { - /* Expression index */ + /* expression from the index */ expr = lfirst(indexpr_item); indexpr_item = lnext(unique_index->indexprs, indexpr_item); } @@ -764,29 +1050,43 @@ get_exprs_from_uniqueindex(IndexOptInfo *unique_index, Assert(false); } + /* should have a valid expression now */ + Assert(expr); + /* - * Check index_col = Const case with regarding to opfamily checking - * If we can remove the index_col from the final UniqueKey->exprs. + * Check if there's (index_col = Const) condition, and that it's using + * a compatible opfamily. If yes, we can remove the index_col from the + * final UniqueKey->exprs, because the value is constant (so removing + * it can't introduce duplicities). */ forboth(lc1, const_exprs, lc2, const_expr_opfamilies) { - if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) - && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + List *opfamilies = (List *) lfirst(lc2); + Node *cexpr = (Node *) lfirst(lc1); + + if (list_member_oid(opfamilies, unique_index->opfamily[c]) && + match_index_to_operand(cexpr, c, unique_index)) { matched_const = true; break; } } + /* it's constant, so ignore the expression */ if (matched_const) continue; - /* Check if the indexed expr is used in rel */ + /* + * Check if the indexed expr is used in rel. We do this after the + * (col = Const) check, because nn expression may be in a a restrict + * clause and not in the reltarget. So we don't want to rule out an + * index unnecessarily. + */ if (attr > 0) { /* - * Normal Indexed column, if the col is not used, then the index is useless - * for uniquekey. + * Normal indexed column, if the col is not used, then the index + * is useless for uniquekey. */ attr -= FirstLowInvalidHeapAttributeNumber; @@ -806,67 +1106,85 @@ get_exprs_from_uniqueindex(IndexOptInfo *unique_index, /* check not null property. */ if (attr == 0) { - /* We never know if a expression yields null or not */ + /* We never know if an expression yields null or not */ *multi_nullvals = true; } - else if (!bms_is_member(attr, unique_index->rel->notnullattrs) - && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, - unique_index->rel->notnullattrs)) + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) && + !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) { *multi_nullvals = true; } exprs = lappend(exprs, expr); } + return exprs; } /* * add_uniquekey_for_onerow - * If we are sure that the relation only returns one row, then all the columns - * are unique. However we don't need to create UniqueKey for every column, we - * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo - * since this one has strongest semantics. + * Create a special unique key signifying that the rel has one row. + * + * If we are sure that the relation only returns one row (it might return + * no rows, but we still consider that unique), then all the columns are + * trivially unique. + * + * However we don't need to create UniqueKey with every column, we just + * set exprs = NIL, because that's easier to identify. We don't want to + * add unnecessary unique keys (such that we already have a unique key + * for a subset of the expressions), and with (exprs == NIL) we can just + * assume we have one unique key for each column in the rel. + * + * We discard all other unique keys, since it has the strongest semantics. */ void add_uniquekey_for_onerow(RelOptInfo *rel) { /* - * We overwrite the previous UniqueKey on purpose since this one has the - * strongest semantic. + * We overwrite the previous UniqueKey on purpose since this one has + * the strongest semantic (all other unique keys are implied by it). */ rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); } /* - * initililze_uniquecontext_for_joinrel - * Return a List of UniqueKeyContext for an inputrel + * initialize_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel. */ static List * -initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +initialize_uniquecontext_for_joinrel(RelOptInfo *inputrel) { - List *res = NIL; - ListCell *lc; - foreach(lc, inputrel->uniquekeys) + List *res = NIL; + ListCell *lc; + + foreach(lc, inputrel->uniquekeys) { UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); context->uniquekey = lfirst_node(UniqueKey, lc); context->added_to_joinrel = false; context->useful = true; + res = lappend(res, context); } + return res; } - /* * get_exprs_from_uniquekey - * Unify the way of get List of exprs from a one-row UniqueKey or - * normal UniqueKey. for the onerow case, every expr in rel1 is a valid - * UniqueKey. Return a List of exprs. + * Extract expressions that are part of a unique key. + * + * The meaning of the result is a bit different in regular and one-row cases. + * For the regular case, the list of expressions form a single unique key, + * i.e. the combination of values is unique. + * + * For the one-row case, each individual expression is known to be unique + * (simply because in a single row everything is unique). * * rel1: The relation which you want to get the exprs. * ukey: The UniqueKey you want to get the exprs. @@ -875,27 +1193,29 @@ static List * get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *rel1, UniqueKey *ukey) { - ListCell *lc; - bool onerow = rel1 != NULL && relation_is_onerow(rel1); + ListCell *lc; + List *res = NIL; + bool onerow = (rel1 != NULL) && relation_is_onerow(rel1); - List *res = NIL; + /* We require at least one of those to be true. */ Assert(onerow || ukey); - if (onerow) - { - /* Only cares about the exprs still exist in joinrel */ - foreach(lc, joinrel->reltarget->exprs) - { - Bitmapset *relids = pull_varnos(root, lfirst(lc)); - if (bms_is_subset(relids, rel1->relids)) - { - res = lappend(res, list_make1(lfirst(lc))); - } - } - } - else + + /* if not a one-row unique key, just return the key's expressions */ + if (!onerow) + return list_make1(ukey->exprs); + + /* + * If it's a one-row relation, we simply extract the expressions that + * still exist in the reltarget. + */ + foreach(lc, joinrel->reltarget->exprs) { - res = list_make1(ukey->exprs); + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + + if (bms_is_subset(relids, rel1->relids)) + res = lappend(res, list_make1(lfirst(lc))); } + return res; } @@ -910,55 +1230,67 @@ get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, /* * index_constains_partkey - * return true if the index contains the partiton key. + * Determines if the index includes a partition key. + * + * XXX Surely we already have a code doing this already? E.g. when creating + * a unique index on a partitioned table we define that. */ static bool -index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) { ListCell *lc; int i; + Assert(IS_PARTITIONED_REL(partrel)); Assert(partrel->part_scheme->partnatts > 0); for(i = 0; i < partrel->part_scheme->partnatts; i++) { - Node *part_expr = linitial(partrel->partexprs[i]); - bool found_in_index = false; + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) { - Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) { found_in_index = true; break; } } + if (!found_in_index) return false; } + return true; } /* * simple_indexinfo_equal + * Compare two indexes to determine if they are the same. + * + * We need to do this because simple_copy_indexinfo_to_parent does change + * some elements. So this is not exactly the same as calling equal(). * - * Used to check if the 2 index is same as each other. The index here - * is COPIED from childrel and did some tiny changes(see - * simple_copy_indexinfo_to_parent) + * XXX I wonder if we could simply use equal(), somehow? In fact, we should + * probably build something much simpler than IndexOptInfo, just enough to + * do the checks. */ static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) { Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; - return ind1->ncolumns == ind2->ncolumns && - ind1->unique == ind2->unique && - memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && - memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && - memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && - memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && - equal(get_tlist_exprs(ind1->indextlist, true), - get_tlist_exprs(ind2->indextlist, true)); + return ((ind1->ncolumns == ind2->ncolumns) && + (ind1->unique == ind2->unique) && + (memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0) && + (memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0) && + (memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0) && + (memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0) && + (equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)))); } @@ -981,11 +1313,21 @@ simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) /* - * simple_copy_indexinfo_to_parent (from partition) - * Copy the IndexInfo from child relation to parent relation with some modification, - * which is used to test: - * 1. If the same index exists in all the childrels. + * simple_copy_indexinfo_to_parent + * Copy index info from child to parent, with necessary tweaks. + * + * We use this copy to check: + * + * 1. If the same/matching index exists in all the childrels. * 2. If the parentrel->reltarget/basicrestrict info matches this index. + * + * XXX IMHO we should probably build something much simpler than a full + * IndexOptInfo copy, just enough to do the checks. + * + * XXX The fact that we copy so much data seems wrong, and having to + * define macros from copyfuncs.c seems like a very suspicious thing. + * One reason is that IndeOptInfo is fairly large struct, especially + * with all the fields, and we allocate it very often. */ static IndexOptInfo * simple_copy_indexinfo_to_parent(PlannerInfo *root, @@ -1027,20 +1369,24 @@ simple_copy_indexinfo_to_parent(PlannerInfo *root, /* * adjust_partition_unique_indexlist + * Checks and eliminates indexes that do not exist on the child relation. * - * global_unique_indexes: At the beginning, it contains the copy & modified - * unique index from the first partition. And then check if each index in it still - * exists in the following partitions. If no, remove it. at last, it has an - * index list which exists in all the partitions. + * Walks the list of unique indexes, and eliminates those that don't match + * the child relation (i.e. where a matching child index does not exist). + * This is used to iteratively filter the list of candidate unique keys. + * + * After processing all child relations, the list contains only indexes that + * exist in all the child relations. */ static void adjust_partition_unique_indexlist(PlannerInfo *root, RelOptInfo *parentrel, RelOptInfo *childrel, - List **global_unique_indexes) + List **indexes) { ListCell *lc, *lc2; - foreach(lc, *global_unique_indexes) + + foreach(lc, *indexes) { IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); bool found_in_child = false; @@ -1049,23 +1395,45 @@ adjust_partition_unique_indexlist(PlannerInfo *root, { IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); IndexOptInfo *p_ind_copy; - if (!p_ind->unique || !p_ind->immediate || - (p_ind->indpred != NIL && !p_ind->predOK)) + + /* + * Ignore child indexes that can't possibly match (not unique or + * immediate, etc.) + * + * XXX We do these checks in many places, so maybe turn it into + * a reusable macro? + */ + if ((!p_ind->unique) || (!p_ind->immediate) || + (p_ind->indpred != NIL) && (!p_ind->predOK)) continue; + + /* + * XXX This seems possibly quite expensive. Imagine there are many + * child relations, with a bunch of unique indexes each. Then this + * generates a copy for each unique index in each child relation, + * something like O(N^2/2) copies. + */ p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + + /* Found a matching index for the child relation, we're done. */ if (simple_indexinfo_equal(p_ind_copy, g_ind)) { found_in_child = true; break; } } + + /* No matching index in the child, so remove it from the list. */ if (!found_in_child) - /* The index doesn't exist in childrel, remove it from global_unique_indexes */ - *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + *indexes = foreach_delete_current(*indexes, lc); } } -/* Helper function for groupres/distinctrel */ +/* + * Helper function for groupres/distinctrel + * + * FIXME Not sure about this. + */ static void add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) { @@ -1073,27 +1441,32 @@ add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgrou List *exprs; /* - * XXX: If there are some vars which is not in current levelsup, the semantic is - * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to - * check every level other than 0, if so, looks we have to write another - * pull_var_walker. + * XXX: If there are some vars which are not in the current levelsup, the + * semantic is imprecise, should we avoid it or not? levelsup = 1 is just + * a demo, maybe we need to check every level other than 0, if so, looks + * we have to write another pull_var_walker. */ List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); if (upper_vars != NIL) return; + /* sortgroupclause can't be multi_nullvals */ exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); rel->uniquekeys = lappend(rel->uniquekeys, - makeUniqueKey(exprs, - false /* sortgroupclause can't be multi_nullvals */)); + makeUniqueKey(exprs, false)); } /* * add_combined_uniquekey - * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter - * the jointype. + * Add a unique key for a join, combined from keys on inner/outer side. + * + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no + * matter what's the exact jointype. + * + * Returns true if the unique key is "one-row" variant, so that the caller + * can stop considering further combinations. */ bool add_combined_uniquekey(PlannerInfo *root, @@ -1104,32 +1477,47 @@ add_combined_uniquekey(PlannerInfo *root, UniqueKey *inner_ukey, JoinType jointype) { + bool multi_nullvals; + ListCell *lc1, *lc2; - ListCell *lc1, *lc2; - - /* Either side has multi_nullvals or we have outer join, - * the combined UniqueKey has multi_nullvals */ - bool multi_nullvals = outer_ukey->multi_nullvals || + /* + * If either side has multi_nullvals, or we are dealing with an outer join, + * the combined UniqueKey has multi_nullvals too. + */ + multi_nullvals = outer_ukey->multi_nullvals || inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); /* The only case we can get onerow joinrel after join */ - if (relation_is_onerow(outer_rel) - && relation_is_onerow(inner_rel) - && jointype == JOIN_INNER) + if (relation_is_onerow(outer_rel) && + relation_is_onerow(inner_rel) && + jointype == JOIN_INNER) { add_uniquekey_for_onerow(joinrel); return true; } + /* + * XXX Isn't this wrong? Why is it combining expressions that are part + * of the two unique keys? Imagine we have outer unique key on (a1, a2) + * and inner outer key on (b1, b2). Then this adds four unique keys + * for the join (a1,b1), (a1,b2), (a2,b1) and (a2,b2). Shouldn't it + * just add (a1,a2,b1,b2)? + */ foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) { + /* + * XXX This calls get_exprs_from_uniquekey repeatedly for each outer + * loop. Maybe we should calculate it just once before the loop. + */ foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) { List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, makeUniqueKey(exprs, multi_nullvals)); } } + return false; } diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 8d8e493f5c..f29b65c07b 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,7 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + /* XXX comment? can we simply just copy the unique keys to the final relation? */ simple_copy_uniquekeys(current_rel, final_rel); /* @@ -3902,7 +3903,9 @@ create_grouping_paths(PlannerInfo *root, set_cheapest(grouped_rel); + /* XXX does this apply to grouping sets too? */ populate_grouprel_uniquekeys(root, grouped_rel, input_rel); + return grouped_rel; } @@ -4625,7 +4628,10 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); + + /* XXX comment? */ simple_copy_uniquekeys(input_rel, window_rel); + return window_rel; } @@ -4939,7 +4945,10 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); + + /* XXX comment */ populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); + return distinct_rel; } @@ -5200,6 +5209,7 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + /* XXX comment */ simple_copy_uniquekeys(input_rel, ordered_rel); return ordered_rel; diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index b7626545bf..72a3f3c598 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -691,6 +691,7 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Add the UniqueKeys */ populate_unionrel_uniquekeys(root, result_rel); + return result_rel; } diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index 3eec1f4d74..c9829c5fc4 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -755,6 +755,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, pseudoconstant, rinfo->security_level, NULL, NULL, NULL); + /* XXX This is a bit weird, doing this outside make_restrictinfo */ child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ -- 2.30.2 --------------F495F6B18672582269F00FF9 Content-Type: text/x-patch; charset=UTF-8; name="0005-Extend-UniqueKeys-20210317.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0005-Extend-UniqueKeys-20210317.patch" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 06/10] review @ 2021-03-17 01:19 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-17 01:19 UTC (permalink / raw) --- src/backend/optimizer/path/uniquekeys.c | 2 ++ src/backend/optimizer/plan/planner.c | 2 ++ 2 files changed, 4 insertions(+) diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c index b9567ca5c8..0e6c826fcd 100644 --- a/src/backend/optimizer/path/uniquekeys.c +++ b/src/backend/optimizer/path/uniquekeys.c @@ -1522,6 +1522,7 @@ add_combined_uniquekey(PlannerInfo *root, return false; } +/* FIXME comment */ List* build_uniquekeys(PlannerInfo *root, List *sortclauses) { @@ -1548,6 +1549,7 @@ build_uniquekeys(PlannerInfo *root, List *sortclauses) return result; } +/* FIXME comment */ bool query_has_uniquekeys_for(PlannerInfo *root, List *pathuniquekeys, bool allow_multinulls) diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 62c0e46c49..f66b9d4898 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -4857,11 +4857,13 @@ create_distinct_paths(PlannerInfo *root, Path *path = (Path *) lfirst(lc); if (pathkeys_contained_in(needed_pathkeys, path->pathkeys)) + { add_path(distinct_rel, (Path *) create_upper_unique_path(root, distinct_rel, path, list_length(root->distinct_pathkeys), numDistinctRows)); + } } foreach(lc, input_rel->unique_pathlist) -- 2.30.2 --------------F495F6B18672582269F00FF9 Content-Type: text/x-patch; charset=UTF-8; name="0007-Index-skip-scan-20210317.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0007-Index-skip-scan-20210317.patch" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 06/10] review @ 2021-03-17 01:19 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-17 01:19 UTC (permalink / raw) --- src/backend/optimizer/path/uniquekeys.c | 2 ++ src/backend/optimizer/plan/planner.c | 2 ++ 2 files changed, 4 insertions(+) diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c index b9567ca5c8..0e6c826fcd 100644 --- a/src/backend/optimizer/path/uniquekeys.c +++ b/src/backend/optimizer/path/uniquekeys.c @@ -1522,6 +1522,7 @@ add_combined_uniquekey(PlannerInfo *root, return false; } +/* FIXME comment */ List* build_uniquekeys(PlannerInfo *root, List *sortclauses) { @@ -1548,6 +1549,7 @@ build_uniquekeys(PlannerInfo *root, List *sortclauses) return result; } +/* FIXME comment */ bool query_has_uniquekeys_for(PlannerInfo *root, List *pathuniquekeys, bool allow_multinulls) diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 62c0e46c49..f66b9d4898 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -4857,11 +4857,13 @@ create_distinct_paths(PlannerInfo *root, Path *path = (Path *) lfirst(lc); if (pathkeys_contained_in(needed_pathkeys, path->pathkeys)) + { add_path(distinct_rel, (Path *) create_upper_unique_path(root, distinct_rel, path, list_length(root->distinct_pathkeys), numDistinctRows)); + } } foreach(lc, input_rel->unique_pathlist) -- 2.30.2 --------------F495F6B18672582269F00FF9 Content-Type: text/x-patch; charset=UTF-8; name="0007-Index-skip-scan-20210317.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0007-Index-skip-scan-20210317.patch" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 08/10] review @ 2021-03-17 01:36 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-17 01:36 UTC (permalink / raw) --- src/backend/commands/explain.c | 1 + src/backend/executor/nodeIndexscan.c | 2 ++ 2 files changed, 3 insertions(+) diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index a160de5493..3b0b38099f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -1110,6 +1110,7 @@ ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es) { if (skipPrefixSize > 0) { + /* FIXME Why not to show this for TEXT output? */ if (es->format != EXPLAIN_FORMAT_TEXT) ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es); } diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c index 71aac4493d..720442b731 100644 --- a/src/backend/executor/nodeIndexscan.c +++ b/src/backend/executor/nodeIndexscan.c @@ -124,6 +124,8 @@ IndexNext(IndexScanState *node) node->iss_ScanDesc = scandesc; + /* FIXME Is the else branch necessary? */ + /* Index skip scan assumes xs_want_itup, so set it to true */ if (indexscan->indexskipprefixsize > 0) node->iss_ScanDesc->xs_want_itup = true; -- 2.30.2 --------------F495F6B18672582269F00FF9 Content-Type: text/x-patch; charset=UTF-8; name="0009-Btree-implementation-of-skipping-20210317.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0009-Btree-implementation-of-skipping-20210317.patch" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH 08/10] review @ 2021-03-17 01:36 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tomas Vondra @ 2021-03-17 01:36 UTC (permalink / raw) --- src/backend/commands/explain.c | 1 + src/backend/executor/nodeIndexscan.c | 2 ++ 2 files changed, 3 insertions(+) diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index a160de5493..3b0b38099f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -1110,6 +1110,7 @@ ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es) { if (skipPrefixSize > 0) { + /* FIXME Why not to show this for TEXT output? */ if (es->format != EXPLAIN_FORMAT_TEXT) ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es); } diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c index 71aac4493d..720442b731 100644 --- a/src/backend/executor/nodeIndexscan.c +++ b/src/backend/executor/nodeIndexscan.c @@ -124,6 +124,8 @@ IndexNext(IndexScanState *node) node->iss_ScanDesc = scandesc; + /* FIXME Is the else branch necessary? */ + /* Index skip scan assumes xs_want_itup, so set it to true */ if (indexscan->indexskipprefixsize > 0) node->iss_ScanDesc->xs_want_itup = true; -- 2.30.2 --------------F495F6B18672582269F00FF9 Content-Type: text/x-patch; charset=UTF-8; name="0009-Btree-implementation-of-skipping-20210317.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0009-Btree-implementation-of-skipping-20210317.patch" ^ permalink raw reply [nested|flat] 67+ messages in thread
* [PATCH] Extend ALTER OPERATOR to support adding commutator, negator, hashes, and merges @ 2023-06-22 16:35 Tommy Pavlicek <[email protected]> 0 siblings, 0 replies; 67+ messages in thread From: Tommy Pavlicek @ 2023-06-22 16:35 UTC (permalink / raw) To: [email protected]; +Cc: Tom Lane <[email protected]>; Tomas Vondra <[email protected]> Hi All, I've attached a couple of patches to allow ALTER OPERATOR to add commutators, negators, hashes and merges to operators that lack them. The need for this arose adding hash functions to the ltree type after the operator had been created without hash support[1]. There are potential issues with modifying these attributes that have been discussed previously[2], but I understand that setting them, if they have not been set before, is ok. I belatedly realised that it may not be desirable or necessary to allow adding commutators and negators in ALTER OPERATOR because the linkage can already be added when creating a new operator. I don't know what's best, so I thought I'd post this here and get feedback before removing anything. The first patch is create_op_fixes_v1.patch and it includes some refactoring in preparation for the ALTER OPERATOR changes and fixes a couple of minor bugs in CREATE OPERATOR: - prevents self negation when filling in/creating an existing shell operator - remove reference to sort operator in the self negation error message as the sort attribute appears to be deprecated in Postgres 8.3 The second patch is alter_op_v1.patch which contains the changes to ALTER OPERATOR and depends on create_op_fixes_v1.patch. Additionally, I wasn't sure whether it was preferred to fail or succeed on ALTERs that have no effect, such as adding hashes on an operator that already allows them or disabling hashes on one that does not. I chose to raise an error when this happens, on the thinking it was more explicit and made the code simpler, even though the end result would be what the user wanted. Comments appreciated. Thanks, Tommy [1] https://www.postgresql.org/message-id/flat/CAEhP-W9ZEoHeaP_nKnPCVd_o1c3BAUvq1gWHrq8EbkNRiS9CvQ%40mai... [2] https://www.postgresql.org/message-id/flat/3348985.V7xMLFDaJO@dinodell Attachments: [application/octet-stream] alter_op_v1.patch (29.7K, ../../CAEhP-W-vGVzf4udhR5M8Bdv88UYnPrhoSkj3ieR3QNrsGQoqdg@mail.gmail.com/3-alter_op_v1.patch) download | inline diff: diff --git a/doc/src/sgml/ref/alter_operator.sgml b/doc/src/sgml/ref/alter_operator.sgml index a4a1af564f..04af611cca 100644 --- a/doc/src/sgml/ref/alter_operator.sgml +++ b/doc/src/sgml/ref/alter_operator.sgml @@ -29,8 +29,12 @@ ALTER OPERATOR <replaceable>name</replaceable> ( { <replaceable>left_type</repla ALTER OPERATOR <replaceable>name</replaceable> ( { <replaceable>left_type</replaceable> | NONE } , <replaceable>right_type</replaceable> ) SET ( { RESTRICT = { <replaceable class="parameter">res_proc</replaceable> | NONE } - | JOIN = { <replaceable class="parameter">join_proc</replaceable> | NONE } - } [, ... ] ) + | JOIN = { <replaceable class="parameter">join_proc</replaceable> | NONE } + | COMMUTATOR = <replaceable class="parameter">com_op</replaceable> + | NEGATOR = <replaceable class="parameter">neg_op</replaceable> + | HASHES + | MERGES + } [, ... ] ) </synopsis> </refsynopsisdiv> @@ -121,6 +125,42 @@ ALTER OPERATOR <replaceable>name</replaceable> ( { <replaceable>left_type</repla </listitem> </varlistentry> + <varlistentry> + <term><replaceable class="parameter">com_op</replaceable></term> + <listitem> + <para> + The commutator of this operator. Can only be set if the operator does not have an existing commutator. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">neg_op</replaceable></term> + <listitem> + <para> + The negator of this operator. Can only be set if the operator does not have an existing negator. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>HASHES</literal></term> + <listitem> + <para> + Indicates this operator can support a hash join. Can only be set when the operator does not support a hash join. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>MERGES</literal></term> + <listitem> + <para> + Indicates this operator can support a merge join. Can only be set when the operator does not support a merge join. + </para> + </listitem> + </varlistentry> + </variablelist> </refsect1> diff --git a/src/backend/commands/operatorcmds.c b/src/backend/commands/operatorcmds.c index cd7f83136f..3787c234d4 100644 --- a/src/backend/commands/operatorcmds.c +++ b/src/backend/commands/operatorcmds.c @@ -54,6 +54,7 @@ static Oid ValidateRestrictionEstimator(List *restrictionName); static Oid ValidateJoinEstimator(List *joinName); +static void ValidateBooleanIsTrue(DefElem *def, const char *paramName); /* * DefineOperator @@ -359,6 +360,18 @@ ValidateJoinEstimator(List *joinName) return joinOid; } +static void +ValidateBooleanIsTrue(DefElem *def, const char *paramName) +{ + if (!defGetBoolean(def)) + { + ereport(ERROR, + (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), + errmsg("alter operator cannot disable \"%s\"", + paramName))); + } +} + /* * Guts of operator deletion. */ @@ -426,6 +439,12 @@ AlterOperator(AlterOperatorStmt *stmt) List *joinName = NIL; /* optional join sel. function */ bool updateJoin = false; Oid joinOid; + List *commutatorName = NIL; /* optional commutator operator name */ + Oid commutatorOid; + List *negatorName = NIL; /* optional negator operator name */ + Oid negatorOid; + bool updateHashes = false; + bool updateMerges = false; /* Look up the operator */ oprId = LookupOperWithArgs(stmt->opername, false); @@ -456,6 +475,24 @@ AlterOperator(AlterOperatorStmt *stmt) joinName = param; updateJoin = true; } + else if (strcmp(defel->defname, "commutator") == 0) + { + commutatorName = defGetQualifiedName(defel); + } + else if (strcmp(defel->defname, "negator") == 0) + { + negatorName = defGetQualifiedName(defel); + } + else if (strcmp(defel->defname, "hashes") == 0) + { + ValidateBooleanIsTrue(defel, "hashes"); + updateHashes = true; + } + else if (strcmp(defel->defname, "merges") == 0) + { + ValidateBooleanIsTrue(defel, "merges"); + updateMerges = true; + } /* * The rest of the options that CREATE accepts cannot be changed. @@ -464,11 +501,7 @@ AlterOperator(AlterOperatorStmt *stmt) else if (strcmp(defel->defname, "leftarg") == 0 || strcmp(defel->defname, "rightarg") == 0 || strcmp(defel->defname, "function") == 0 || - strcmp(defel->defname, "procedure") == 0 || - strcmp(defel->defname, "commutator") == 0 || - strcmp(defel->defname, "negator") == 0 || - strcmp(defel->defname, "hashes") == 0 || - strcmp(defel->defname, "merges") == 0) + strcmp(defel->defname, "procedure") == 0) { ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), @@ -488,7 +521,7 @@ AlterOperator(AlterOperatorStmt *stmt) NameStr(oprForm->oprname)); /* - * Look up restriction and join estimators if specified + * Look up Oid for any parameters specified */ if (restrictionName) restrictionOid = ValidateRestrictionEstimator(restrictionName); @@ -499,28 +532,77 @@ AlterOperator(AlterOperatorStmt *stmt) else joinOid = InvalidOid; - /* Perform additional checks, like OperatorCreate does */ - if (!(OidIsValid(oprForm->oprleft) && OidIsValid(oprForm->oprright))) + if (commutatorName) { - /* If it's not a binary op, these things mustn't be set: */ - if (OidIsValid(joinOid)) - ereport(ERROR, - (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), - errmsg("only binary operators can have join selectivity"))); + commutatorOid = OperatorGetOrCreateValidCommutator(commutatorName, + oprForm->oid, + oprForm->oprname.data, + oprForm->oprnamespace, + oprForm->oprleft, + oprForm->oprright); + + /* + * we don't need to do anything extra for a self commutator as in + * OperatorCreate as there we have to create the operator before + * setting the commutator, but here the operator already exists and + * the commutatorOid above is valid (and is the operator oid). + */ } + else + commutatorOid = InvalidOid; + + if (negatorName) + negatorOid = OperatorGetOrCreateValidNegator(negatorName, + oprForm->oid, + oprForm->oprname.data, + oprForm->oprnamespace, + oprForm->oprleft, + oprForm->oprright); + else + negatorOid = InvalidOid; - if (oprForm->oprresult != BOOLOID) + /* + * check that we're not changing any existing values that can't be changed + */ + if (OidIsValid(commutatorOid) && OidIsValid(oprForm->oprcom)) { - if (OidIsValid(restrictionOid)) - ereport(ERROR, - (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), - errmsg("only boolean operators can have restriction selectivity"))); - if (OidIsValid(joinOid)) - ereport(ERROR, - (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), - errmsg("only boolean operators can have join selectivity"))); + ereport(ERROR, + (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), + errmsg("operator attribute \"commutator\" cannot be changed if it has already been set"))); } + if (OidIsValid(negatorOid) && OidIsValid(oprForm->oprnegate)) + { + ereport(ERROR, + (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), + errmsg("operator attribute \"negator\" cannot be changed if it has already been set"))); + } + + if (updateHashes && oprForm->oprcanhash) + { + ereport(ERROR, + (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), + errmsg("operator attribute \"hashes\" cannot be changed if it has already been set"))); + } + + if (updateMerges && oprForm->oprcanmerge) + { + ereport(ERROR, + (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), + errmsg("operator attribute \"merges\" cannot be changed if it has already been set"))); + } + + /* Perform additional checks, like OperatorCreate does */ + OperatorValidateParams(oprForm->oprleft, + oprForm->oprright, + oprForm->oprresult, + OidIsValid(commutatorOid), + OidIsValid(negatorOid), + OidIsValid(joinOid), + OidIsValid(restrictionOid), + updateMerges, + updateHashes); + /* Update the tuple */ for (i = 0; i < Natts_pg_operator; ++i) { @@ -539,6 +621,30 @@ AlterOperator(AlterOperatorStmt *stmt) values[Anum_pg_operator_oprjoin - 1] = joinOid; } + if (OidIsValid(commutatorOid)) + { + replaces[Anum_pg_operator_oprcom - 1] = true; + values[Anum_pg_operator_oprcom - 1] = ObjectIdGetDatum(commutatorOid); + } + + if (OidIsValid(negatorOid)) + { + replaces[Anum_pg_operator_oprnegate - 1] = true; + values[Anum_pg_operator_oprnegate - 1] = ObjectIdGetDatum(negatorOid); + } + + if (updateMerges) + { + replaces[Anum_pg_operator_oprcanmerge - 1] = true; + values[Anum_pg_operator_oprcanmerge - 1] = true; + } + + if (updateHashes) + { + replaces[Anum_pg_operator_oprcanhash - 1] = true; + values[Anum_pg_operator_oprcanhash - 1] = true; + } + tup = heap_modify_tuple(tup, RelationGetDescr(catalog), values, nulls, replaces); @@ -550,5 +656,8 @@ AlterOperator(AlterOperatorStmt *stmt) table_close(catalog, NoLock); + if (OidIsValid(commutatorOid) || OidIsValid(negatorOid)) + OperatorUpd(oprId, commutatorOid, negatorOid, false); + return address; } diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 39ab7eac0d..84017da3ee 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -10101,6 +10101,8 @@ operator_def_elem: ColLabel '=' NONE { $$ = makeDefElem($1, NULL, @1); } | ColLabel '=' operator_def_arg { $$ = makeDefElem($1, (Node *) $3, @1); } + | ColLabel + { $$ = makeDefElem($1, NULL, @1); } ; /* must be similar enough to def_arg to avoid reduce/reduce conflicts */ diff --git a/src/test/regress/expected/alter_operator.out b/src/test/regress/expected/alter_operator.out index 71bd484282..57cf4df155 100644 --- a/src/test/regress/expected/alter_operator.out +++ b/src/test/regress/expected/alter_operator.out @@ -25,7 +25,7 @@ ORDER BY 1; (3 rows) -- --- Reset and set params +-- test reset and set restrict and join -- ALTER OPERATOR === (boolean, boolean) SET (RESTRICT = NONE); ALTER OPERATOR === (boolean, boolean) SET (JOIN = NONE); @@ -106,34 +106,240 @@ ORDER BY 1; schema public | n (3 rows) --- --- Test invalid options. --- -ALTER OPERATOR === (boolean, boolean) SET (COMMUTATOR = ====); -ERROR: operator attribute "commutator" cannot be changed -ALTER OPERATOR === (boolean, boolean) SET (NEGATOR = ====); -ERROR: operator attribute "negator" cannot be changed +-- test cannot set non existant function ALTER OPERATOR === (boolean, boolean) SET (RESTRICT = non_existent_func); ERROR: function non_existent_func(internal, oid, internal, integer) does not exist ALTER OPERATOR === (boolean, boolean) SET (JOIN = non_existent_func); ERROR: function non_existent_func(internal, oid, internal, smallint, internal) does not exist -ALTER OPERATOR === (boolean, boolean) SET (COMMUTATOR = !==); -ERROR: operator attribute "commutator" cannot be changed -ALTER OPERATOR === (boolean, boolean) SET (NEGATOR = !==); -ERROR: operator attribute "negator" cannot be changed --- invalid: non-lowercase quoted identifiers +-- test non-lowercase quoted identifiers invalid ALTER OPERATOR & (bit, bit) SET ("Restrict" = _int_contsel, "Join" = _int_contjoinsel); ERROR: operator attribute "Restrict" not recognized -- --- Test permission check. Must be owner to ALTER OPERATOR. +-- test must be owner of operator to ALTER OPERATOR. -- CREATE USER regress_alter_op_user; SET SESSION AUTHORIZATION regress_alter_op_user; ALTER OPERATOR === (boolean, boolean) SET (RESTRICT = NONE); ERROR: must be owner of operator === --- Clean up RESET SESSION AUTHORIZATION; +-- +-- test set commutator, negator, hashes, and merges which can only be set if not +-- already set +-- +-- for these tests create operators with different left and right types so that +-- we can validate that function signatures are handled correctly +CREATE FUNCTION alter_op_test_fn_bool_real(boolean, real) +RETURNS boolean AS $$ SELECT NULL::BOOLEAN; $$ LANGUAGE sql IMMUTABLE; +CREATE FUNCTION alter_op_test_fn_real_bool(real, boolean) +RETURNS boolean AS $$ SELECT NULL::BOOLEAN; $$ LANGUAGE sql IMMUTABLE; +-- operator +CREATE OPERATOR === ( + LEFTARG = boolean, + RIGHTARG = real, + PROCEDURE = alter_op_test_fn_bool_real +); +-- commutator +CREATE OPERATOR ==== ( + LEFTARG = real, + RIGHTARG = boolean, + PROCEDURE = alter_op_test_fn_real_bool +); +-- negator +CREATE OPERATOR !==== ( + LEFTARG = boolean, + RIGHTARG = real, + PROCEDURE = alter_op_test_fn_bool_real +); +-- test cannot set hashes or merges to false +ALTER OPERATOR === (boolean, real) SET (HASHES = false); +ERROR: alter operator cannot disable "hashes" +ALTER OPERATOR === (boolean, real) SET (MERGES = false); +ERROR: alter operator cannot disable "merges" +-- test cannot set commutator or negator without owning them +SET SESSION AUTHORIZATION regress_alter_op_user; +-- we need need a new operator owned by regress_alter_op_user so that we are +-- allowed to alter it. ==== and !==== are owned by the test user so we expect +-- the alters below to fail. +CREATE OPERATOR ===@@@ ( + LEFTARG = boolean, + RIGHTARG = real, + PROCEDURE = alter_op_test_fn_bool_real +); +ALTER OPERATOR ===@@@ (boolean, real) SET (COMMUTATOR= ====); +ERROR: must be owner of operator ==== +ALTER OPERATOR ===@@@ (boolean, real) SET (NEGATOR = !====); +ERROR: must be owner of operator !==== +-- validate operator is unchanged and commutator and negator are unset +SELECT oprcom, oprnegate FROM pg_operator WHERE oprname = '===@@@' + AND oprleft = 'boolean'::regtype AND oprright = 'real'::regtype; + oprcom | oprnegate +--------+----------- + 0 | 0 +(1 row) + +DROP OPERATOR ===@@@ (boolean, real); +RESET SESSION AUTHORIZATION; +-- test cannot set self negator +ALTER OPERATOR === (boolean, real) SET (NEGATOR = ===); +ERROR: operator cannot be its own negator +-- validate no changes made +SELECT oprcanmerge, oprcanhash, oprcom, oprnegate +FROM pg_operator WHERE oprname = '===' + AND oprleft = 'boolean'::regtype AND oprright = 'real'::regtype; + oprcanmerge | oprcanhash | oprcom | oprnegate +-------------+------------+--------+----------- + f | f | 0 | 0 +(1 row) + +-- test set hashes +ALTER OPERATOR === (boolean, real) SET (HASHES); +SELECT oprcanhash FROM pg_operator WHERE oprname = '===' + AND oprleft = 'boolean'::regtype AND oprright = 'real'::regtype; + oprcanhash +------------ + t +(1 row) + +-- test set merges +ALTER OPERATOR === (boolean, real) SET (MERGES); +SELECT oprcanmerge FROM pg_operator WHERE oprname = '===' + AND oprleft = 'boolean'::regtype AND oprright = 'real'::regtype; + oprcanmerge +------------- + t +(1 row) + +-- test set commutator +ALTER OPERATOR === (boolean, real) SET (COMMUTATOR = ====); +-- validate that the commutator has been set on both the operator and commutator, +-- that they reference each other, and that the operator used is the existing +-- one we created and not a new shell operator +SELECT op.oprname AS operator_name, com.oprname AS commutator_name, + com.oprcode AS commutator_func + FROM pg_operator op + INNER JOIN pg_operator com ON (op.oid = com.oprcom AND op.oprcom = com.oid) + WHERE op.oprname = '===' + AND op.oprleft = 'boolean'::regtype AND op.oprright = 'real'::regtype; + operator_name | commutator_name | commutator_func +---------------+-----------------+---------------------------- + === | ==== | alter_op_test_fn_real_bool +(1 row) + +-- test set negator +ALTER OPERATOR === (boolean, real) SET (NEGATOR = !====); +-- validate that the negator has been set on both the operator and negator, that +-- they reference each other, and that the operator used is the existing one we +-- created and not a new shell operator +SELECT op.oprname AS operator_name, neg.oprname AS negator_name, + neg.oprcode AS negator_func + FROM pg_operator op + INNER JOIN pg_operator neg ON (op.oid = neg.oprnegate AND op.oprnegate = neg.oid) + WHERE op.oprname = '===' + AND op.oprleft = 'boolean'::regtype AND op.oprright = 'real'::regtype; + operator_name | negator_name | negator_func +---------------+--------------+---------------------------- + === | !==== | alter_op_test_fn_bool_real +(1 row) + +-- validate that the final state of the operator is as we expect +SELECT oprcanmerge, oprcanhash, + pg_describe_object('pg_operator'::regclass, oprcom, 0) AS commutator, + pg_describe_object('pg_operator'::regclass, oprnegate, 0) AS negator + FROM pg_operator WHERE oprname = '===' + AND oprleft = 'boolean'::regtype AND oprright = 'real'::regtype; + oprcanmerge | oprcanhash | commutator | negator +-------------+------------+-----------------------------+------------------------------ + t | t | operator ====(real,boolean) | operator !====(boolean,real) +(1 row) + +-- test cannot set commutator, negator, hashes, and merges when already set +ALTER OPERATOR === (boolean, real) SET (COMMUTATOR = =); +ERROR: operator attribute "commutator" cannot be changed if it has already been set +ALTER OPERATOR === (boolean, real) SET (NEGATOR = =); +ERROR: operator attribute "negator" cannot be changed if it has already been set +ALTER OPERATOR === (boolean, real) SET (COMMUTATOR = !=); +ERROR: operator attribute "commutator" cannot be changed if it has already been set +ALTER OPERATOR === (boolean, real) SET (NEGATOR = !=); +ERROR: operator attribute "negator" cannot be changed if it has already been set +ALTER OPERATOR === (boolean, real) SET (HASHES); +ERROR: operator attribute "hashes" cannot be changed if it has already been set +ALTER OPERATOR === (boolean, real) SET (MERGES); +ERROR: operator attribute "merges" cannot be changed if it has already been set +ALTER OPERATOR === (boolean, real) SET (HASHES = false); +ERROR: alter operator cannot disable "hashes" +ALTER OPERATOR === (boolean, real) SET (MERGES = false); +ERROR: alter operator cannot disable "merges" +-- validate no changes made +SELECT oprcanmerge, oprcanhash, + pg_describe_object('pg_operator'::regclass, oprcom, 0) AS commutator, + pg_describe_object('pg_operator'::regclass, oprnegate, 0) AS negator + FROM pg_operator WHERE oprname = '===' + AND oprleft = 'boolean'::regtype AND oprright = 'real'::regtype; + oprcanmerge | oprcanhash | commutator | negator +-------------+------------+-----------------------------+------------------------------ + t | t | operator ====(real,boolean) | operator !====(boolean,real) +(1 row) + +-- +-- test setting undefined operator creates shell operator (matches the +-- behaviour of CREATE OPERATOR) +-- +DROP OPERATOR === (boolean, real); +CREATE OPERATOR === ( + LEFTARG = boolean, + RIGHTARG = real, + PROCEDURE = alter_op_test_fn_bool_real +); +ALTER OPERATOR === (boolean, real) SET (COMMUTATOR = ===@@@); +ALTER OPERATOR === (boolean, real) SET (NEGATOR = !===@@@); +-- validate that shell operators are created for the commutator and negator with +-- the commutator having reversed args and the negator matching the operator. +-- The shell operators should have an empty function below. +SELECT pg_describe_object('pg_operator'::regclass, op.oid, 0) AS operator, + pg_describe_object('pg_operator'::regclass, com.oid, 0) AS commutator, + com.oprcode AS commutator_func, + pg_describe_object('pg_operator'::regclass, neg.oid, 0) AS negator, + neg.oprcode AS negator_func + FROM pg_operator op + INNER JOIN pg_operator com ON (op.oid = com.oprcom AND op.oprcom = com.oid) + INNER JOIN pg_operator neg ON (op.oid = neg.oprnegate AND op.oprnegate = neg.oid) + WHERE op.oprname = '===' AND com.oprname = '===@@@' AND neg.oprname = '!===@@@' + AND op.oprleft = 'boolean'::regtype AND op.oprright = 'real'::regtype; + operator | commutator | commutator_func | negator | negator_func +----------------------------+-------------------------------+-----------------+--------------------------------+-------------- + operator ===(boolean,real) | operator ===@@@(real,boolean) | - | operator !===@@@(boolean,real) | - +(1 row) + +-- +-- test setting self commutator +-- +DROP OPERATOR === (boolean, boolean); +CREATE OPERATOR === ( + LEFTARG = boolean, + RIGHTARG = boolean, + PROCEDURE = alter_op_test_fn +); +ALTER OPERATOR === (boolean, boolean) SET (COMMUTATOR = ===); +-- validate that the oprcom is the operator oid +SELECT oprname FROM pg_operator + WHERE oprname = '===' AND oid = oprcom + AND oprleft = 'boolean'::regtype AND oprright = 'boolean'::regtype; + oprname +--------- + === +(1 row) + +-- +-- Clean up +-- DROP USER regress_alter_op_user; DROP OPERATOR === (boolean, boolean); +DROP OPERATOR === (boolean, real); +DROP OPERATOR ==== (real, boolean); +DROP OPERATOR !==== (boolean, real); +DROP OPERATOR ===@@@ (real, boolean); +DROP OPERATOR !===@@@(boolean, real); DROP FUNCTION customcontsel(internal, oid, internal, integer); DROP FUNCTION alter_op_test_fn(boolean, boolean); +DROP FUNCTION alter_op_test_fn_bool_real(boolean, real); +DROP FUNCTION alter_op_test_fn_real_bool(real, boolean); diff --git a/src/test/regress/sql/alter_operator.sql b/src/test/regress/sql/alter_operator.sql index fd40370165..a7e0d4331f 100644 --- a/src/test/regress/sql/alter_operator.sql +++ b/src/test/regress/sql/alter_operator.sql @@ -22,7 +22,7 @@ WHERE classid = 'pg_operator'::regclass AND ORDER BY 1; -- --- Reset and set params +-- test reset and set restrict and join -- ALTER OPERATOR === (boolean, boolean) SET (RESTRICT = NONE); @@ -71,30 +71,215 @@ WHERE classid = 'pg_operator'::regclass AND objid = '===(bool,bool)'::regoperator ORDER BY 1; --- --- Test invalid options. --- -ALTER OPERATOR === (boolean, boolean) SET (COMMUTATOR = ====); -ALTER OPERATOR === (boolean, boolean) SET (NEGATOR = ====); +-- test cannot set non existant function ALTER OPERATOR === (boolean, boolean) SET (RESTRICT = non_existent_func); ALTER OPERATOR === (boolean, boolean) SET (JOIN = non_existent_func); -ALTER OPERATOR === (boolean, boolean) SET (COMMUTATOR = !==); -ALTER OPERATOR === (boolean, boolean) SET (NEGATOR = !==); --- invalid: non-lowercase quoted identifiers +-- test non-lowercase quoted identifiers invalid ALTER OPERATOR & (bit, bit) SET ("Restrict" = _int_contsel, "Join" = _int_contjoinsel); -- --- Test permission check. Must be owner to ALTER OPERATOR. +-- test must be owner of operator to ALTER OPERATOR. -- CREATE USER regress_alter_op_user; SET SESSION AUTHORIZATION regress_alter_op_user; ALTER OPERATOR === (boolean, boolean) SET (RESTRICT = NONE); --- Clean up RESET SESSION AUTHORIZATION; + +-- +-- test set commutator, negator, hashes, and merges which can only be set if not +-- already set +-- + +-- for these tests create operators with different left and right types so that +-- we can validate that function signatures are handled correctly + +CREATE FUNCTION alter_op_test_fn_bool_real(boolean, real) +RETURNS boolean AS $$ SELECT NULL::BOOLEAN; $$ LANGUAGE sql IMMUTABLE; + +CREATE FUNCTION alter_op_test_fn_real_bool(real, boolean) +RETURNS boolean AS $$ SELECT NULL::BOOLEAN; $$ LANGUAGE sql IMMUTABLE; + +-- operator +CREATE OPERATOR === ( + LEFTARG = boolean, + RIGHTARG = real, + PROCEDURE = alter_op_test_fn_bool_real +); + +-- commutator +CREATE OPERATOR ==== ( + LEFTARG = real, + RIGHTARG = boolean, + PROCEDURE = alter_op_test_fn_real_bool +); + +-- negator +CREATE OPERATOR !==== ( + LEFTARG = boolean, + RIGHTARG = real, + PROCEDURE = alter_op_test_fn_bool_real +); + +-- test cannot set hashes or merges to false +ALTER OPERATOR === (boolean, real) SET (HASHES = false); +ALTER OPERATOR === (boolean, real) SET (MERGES = false); + +-- test cannot set commutator or negator without owning them +SET SESSION AUTHORIZATION regress_alter_op_user; + +-- we need need a new operator owned by regress_alter_op_user so that we are +-- allowed to alter it. ==== and !==== are owned by the test user so we expect +-- the alters below to fail. +CREATE OPERATOR ===@@@ ( + LEFTARG = boolean, + RIGHTARG = real, + PROCEDURE = alter_op_test_fn_bool_real +); + +ALTER OPERATOR ===@@@ (boolean, real) SET (COMMUTATOR= ====); +ALTER OPERATOR ===@@@ (boolean, real) SET (NEGATOR = !====); + +-- validate operator is unchanged and commutator and negator are unset +SELECT oprcom, oprnegate FROM pg_operator WHERE oprname = '===@@@' + AND oprleft = 'boolean'::regtype AND oprright = 'real'::regtype; + +DROP OPERATOR ===@@@ (boolean, real); + +RESET SESSION AUTHORIZATION; + +-- test cannot set self negator +ALTER OPERATOR === (boolean, real) SET (NEGATOR = ===); + +-- validate no changes made +SELECT oprcanmerge, oprcanhash, oprcom, oprnegate +FROM pg_operator WHERE oprname = '===' + AND oprleft = 'boolean'::regtype AND oprright = 'real'::regtype; + +-- test set hashes +ALTER OPERATOR === (boolean, real) SET (HASHES); +SELECT oprcanhash FROM pg_operator WHERE oprname = '===' + AND oprleft = 'boolean'::regtype AND oprright = 'real'::regtype; + +-- test set merges +ALTER OPERATOR === (boolean, real) SET (MERGES); +SELECT oprcanmerge FROM pg_operator WHERE oprname = '===' + AND oprleft = 'boolean'::regtype AND oprright = 'real'::regtype; + +-- test set commutator +ALTER OPERATOR === (boolean, real) SET (COMMUTATOR = ====); + +-- validate that the commutator has been set on both the operator and commutator, +-- that they reference each other, and that the operator used is the existing +-- one we created and not a new shell operator +SELECT op.oprname AS operator_name, com.oprname AS commutator_name, + com.oprcode AS commutator_func + FROM pg_operator op + INNER JOIN pg_operator com ON (op.oid = com.oprcom AND op.oprcom = com.oid) + WHERE op.oprname = '===' + AND op.oprleft = 'boolean'::regtype AND op.oprright = 'real'::regtype; + +-- test set negator +ALTER OPERATOR === (boolean, real) SET (NEGATOR = !====); + +-- validate that the negator has been set on both the operator and negator, that +-- they reference each other, and that the operator used is the existing one we +-- created and not a new shell operator +SELECT op.oprname AS operator_name, neg.oprname AS negator_name, + neg.oprcode AS negator_func + FROM pg_operator op + INNER JOIN pg_operator neg ON (op.oid = neg.oprnegate AND op.oprnegate = neg.oid) + WHERE op.oprname = '===' + AND op.oprleft = 'boolean'::regtype AND op.oprright = 'real'::regtype; + +-- validate that the final state of the operator is as we expect +SELECT oprcanmerge, oprcanhash, + pg_describe_object('pg_operator'::regclass, oprcom, 0) AS commutator, + pg_describe_object('pg_operator'::regclass, oprnegate, 0) AS negator + FROM pg_operator WHERE oprname = '===' + AND oprleft = 'boolean'::regtype AND oprright = 'real'::regtype; + +-- test cannot set commutator, negator, hashes, and merges when already set +ALTER OPERATOR === (boolean, real) SET (COMMUTATOR = =); +ALTER OPERATOR === (boolean, real) SET (NEGATOR = =); +ALTER OPERATOR === (boolean, real) SET (COMMUTATOR = !=); +ALTER OPERATOR === (boolean, real) SET (NEGATOR = !=); +ALTER OPERATOR === (boolean, real) SET (HASHES); +ALTER OPERATOR === (boolean, real) SET (MERGES); +ALTER OPERATOR === (boolean, real) SET (HASHES = false); +ALTER OPERATOR === (boolean, real) SET (MERGES = false); + +-- validate no changes made +SELECT oprcanmerge, oprcanhash, + pg_describe_object('pg_operator'::regclass, oprcom, 0) AS commutator, + pg_describe_object('pg_operator'::regclass, oprnegate, 0) AS negator + FROM pg_operator WHERE oprname = '===' + AND oprleft = 'boolean'::regtype AND oprright = 'real'::regtype; + +-- +-- test setting undefined operator creates shell operator (matches the +-- behaviour of CREATE OPERATOR) +-- + +DROP OPERATOR === (boolean, real); +CREATE OPERATOR === ( + LEFTARG = boolean, + RIGHTARG = real, + PROCEDURE = alter_op_test_fn_bool_real +); + +ALTER OPERATOR === (boolean, real) SET (COMMUTATOR = ===@@@); +ALTER OPERATOR === (boolean, real) SET (NEGATOR = !===@@@); + +-- validate that shell operators are created for the commutator and negator with +-- the commutator having reversed args and the negator matching the operator. +-- The shell operators should have an empty function below. +SELECT pg_describe_object('pg_operator'::regclass, op.oid, 0) AS operator, + pg_describe_object('pg_operator'::regclass, com.oid, 0) AS commutator, + com.oprcode AS commutator_func, + pg_describe_object('pg_operator'::regclass, neg.oid, 0) AS negator, + neg.oprcode AS negator_func + FROM pg_operator op + INNER JOIN pg_operator com ON (op.oid = com.oprcom AND op.oprcom = com.oid) + INNER JOIN pg_operator neg ON (op.oid = neg.oprnegate AND op.oprnegate = neg.oid) + WHERE op.oprname = '===' AND com.oprname = '===@@@' AND neg.oprname = '!===@@@' + AND op.oprleft = 'boolean'::regtype AND op.oprright = 'real'::regtype; + + +-- +-- test setting self commutator +-- + +DROP OPERATOR === (boolean, boolean); +CREATE OPERATOR === ( + LEFTARG = boolean, + RIGHTARG = boolean, + PROCEDURE = alter_op_test_fn +); + +ALTER OPERATOR === (boolean, boolean) SET (COMMUTATOR = ===); + +-- validate that the oprcom is the operator oid +SELECT oprname FROM pg_operator + WHERE oprname = '===' AND oid = oprcom + AND oprleft = 'boolean'::regtype AND oprright = 'boolean'::regtype; + +-- +-- Clean up +-- + DROP USER regress_alter_op_user; + DROP OPERATOR === (boolean, boolean); +DROP OPERATOR === (boolean, real); +DROP OPERATOR ==== (real, boolean); +DROP OPERATOR !==== (boolean, real); +DROP OPERATOR ===@@@ (real, boolean); +DROP OPERATOR !===@@@(boolean, real); + DROP FUNCTION customcontsel(internal, oid, internal, integer); DROP FUNCTION alter_op_test_fn(boolean, boolean); +DROP FUNCTION alter_op_test_fn_bool_real(boolean, real); +DROP FUNCTION alter_op_test_fn_real_bool(real, boolean); [application/octet-stream] create_op_fixes_v1.patch (15.9K, ../../CAEhP-W-vGVzf4udhR5M8Bdv88UYnPrhoSkj3ieR3QNrsGQoqdg@mail.gmail.com/4-create_op_fixes_v1.patch) download | inline diff: diff --git a/src/backend/catalog/pg_operator.c b/src/backend/catalog/pg_operator.c index 95918a77a1..e2590cfce8 100644 --- a/src/backend/catalog/pg_operator.c +++ b/src/backend/catalog/pg_operator.c @@ -54,11 +54,10 @@ static Oid OperatorShellMake(const char *operatorName, Oid leftTypeId, Oid rightTypeId); -static Oid get_other_operator(List *otherOp, - Oid otherLeftTypeId, Oid otherRightTypeId, - const char *operatorName, Oid operatorNamespace, - Oid leftTypeId, Oid rightTypeId, - bool isCommutator); +static Oid get_or_create_other_operator(List *otherOp, + Oid otherLeftTypeId, Oid otherRightTypeId, + const char *operatorName, Oid operatorNamespace, + Oid leftTypeId, Oid rightTypeId); /* @@ -361,53 +360,17 @@ OperatorCreate(const char *operatorName, errmsg("\"%s\" is not a valid operator name", operatorName))); - if (!(OidIsValid(leftTypeId) && OidIsValid(rightTypeId))) - { - /* If it's not a binary op, these things mustn't be set: */ - if (commutatorName) - ereport(ERROR, - (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), - errmsg("only binary operators can have commutators"))); - if (OidIsValid(joinId)) - ereport(ERROR, - (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), - errmsg("only binary operators can have join selectivity"))); - if (canMerge) - ereport(ERROR, - (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), - errmsg("only binary operators can merge join"))); - if (canHash) - ereport(ERROR, - (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), - errmsg("only binary operators can hash"))); - } - operResultType = get_func_rettype(procedureId); - if (operResultType != BOOLOID) - { - /* If it's not a boolean op, these things mustn't be set: */ - if (negatorName) - ereport(ERROR, - (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), - errmsg("only boolean operators can have negators"))); - if (OidIsValid(restrictionId)) - ereport(ERROR, - (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), - errmsg("only boolean operators can have restriction selectivity"))); - if (OidIsValid(joinId)) - ereport(ERROR, - (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), - errmsg("only boolean operators can have join selectivity"))); - if (canMerge) - ereport(ERROR, - (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), - errmsg("only boolean operators can merge join"))); - if (canHash) - ereport(ERROR, - (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), - errmsg("only boolean operators can hash"))); - } + OperatorValidateParams(leftTypeId, + rightTypeId, + operResultType, + commutatorName, + negatorName, + OidIsValid(joinId), + OidIsValid(restrictionId), + canMerge, + canHash); operatorObjectId = OperatorGet(operatorName, operatorNamespace, @@ -438,18 +401,12 @@ OperatorCreate(const char *operatorName, if (commutatorName) { - /* commutator has reversed arg types */ - commutatorId = get_other_operator(commutatorName, - rightTypeId, leftTypeId, - operatorName, operatorNamespace, - leftTypeId, rightTypeId, - true); - - /* Permission check: must own other operator */ - if (OidIsValid(commutatorId) && - !object_ownercheck(OperatorRelationId, commutatorId, GetUserId())) - aclcheck_error(ACLCHECK_NOT_OWNER, OBJECT_OPERATOR, - NameListToString(commutatorName)); + commutatorId = OperatorGetOrCreateValidCommutator(commutatorName, + operatorObjectId, + operatorName, + operatorNamespace, + leftTypeId, + rightTypeId); /* * self-linkage to this operator; will fix below. Note that only @@ -462,20 +419,12 @@ OperatorCreate(const char *operatorName, commutatorId = InvalidOid; if (negatorName) - { - /* negator has same arg types */ - negatorId = get_other_operator(negatorName, - leftTypeId, rightTypeId, - operatorName, operatorNamespace, - leftTypeId, rightTypeId, - false); - - /* Permission check: must own other operator */ - if (OidIsValid(negatorId) && - !object_ownercheck(OperatorRelationId, negatorId, GetUserId())) - aclcheck_error(ACLCHECK_NOT_OWNER, OBJECT_OPERATOR, - NameListToString(negatorName)); - } + negatorId = OperatorGetOrCreateValidNegator(negatorName, + operatorObjectId, + operatorName, + operatorNamespace, + leftTypeId, + rightTypeId); else negatorId = InvalidOid; @@ -574,17 +523,16 @@ OperatorCreate(const char *operatorName, } /* - * Try to lookup another operator (commutator, etc) + * Try to lookup another operator (commutator, negator) * * If not found, check to see if it is exactly the operator we are trying - * to define; if so, return InvalidOid. (Note that this case is only - * sensible for a commutator, so we error out otherwise.) If it is not - * the same operator, create a shell operator. + * to define; if so, return InvalidOid. If it is not the same operator, create + * a shell operator. */ static Oid -get_other_operator(List *otherOp, Oid otherLeftTypeId, Oid otherRightTypeId, - const char *operatorName, Oid operatorNamespace, - Oid leftTypeId, Oid rightTypeId, bool isCommutator) +get_or_create_other_operator(List *otherOp, Oid otherLeftTypeId, Oid otherRightTypeId, + const char *operatorName, Oid operatorNamespace, + Oid leftTypeId, Oid rightTypeId) { Oid other_oid; bool otherDefined; @@ -612,13 +560,12 @@ get_other_operator(List *otherOp, Oid otherLeftTypeId, Oid otherRightTypeId, otherRightTypeId == rightTypeId) { /* - * self-linkage to this operator; caller will fix later. Note that - * only self-linkage for commutation makes sense. + * Do not create as the other operator is the same as the operator: + * self-linkage. This only occurs when the operator has not already + * been created and we expect that the caller is going to create it. + * We can't create it in this case so we return and let the caller + * handle this scenario. */ - if (!isCommutator) - ereport(ERROR, - (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), - errmsg("operator cannot be its own negator or sort operator"))); return InvalidOid; } @@ -867,3 +814,166 @@ makeOperatorDependencies(HeapTuple tuple, return myself; } + +/* + * OperatorGetOrCreateValidCommutator + * + * This function gets or creates a commutator for an existing or yet to be + * created operator. + * + * For an operator that does not exist, pass in InvalidOid for the operatorOid. + * + * It will return: + * - InvalidOid if the commutator is the same as the operator and the operator + * does not exist. + * - The Oid of the commutator (including the operator itself) if it exists + * - The Oid of a newly created commutator shell operator if the commutator does + * not exist and is not the same as the operator. + * + * The function will error if the commutator is not owned by the user. + */ + +Oid +OperatorGetOrCreateValidCommutator(List *commutatorName, + Oid operatorOid, + const char *operatorName, + Oid operatorNamespace, + Oid operatorLeftTypeId, + Oid operatorRightTypeId) +{ + Oid commutatorId; + + /* commutator has reversed arg types */ + commutatorId = get_or_create_other_operator(commutatorName, + operatorRightTypeId, operatorLeftTypeId, + operatorName, operatorNamespace, + operatorLeftTypeId, operatorRightTypeId); + + /* Permission check: must own other operator */ + if (OidIsValid(commutatorId) && + !object_ownercheck(OperatorRelationId, commutatorId, GetUserId())) + aclcheck_error(ACLCHECK_NOT_OWNER, OBJECT_OPERATOR, + NameListToString(commutatorName)); + + return commutatorId; +} + +/* + * OperatorGetOrCreateValidNegator + * + * This function gets or creates a negator for an existing or yet to be created + * operator. It will either return an existing operator or a newly created shell + * operator. + * + * For an operator that does not exist, pass in InvalidOid for the operatorOid. + * + * This function will either return a valid negator oid or error: + * - if the operator is the same as the negator as a negator cannot self link + * - if the negator is not owned by the user + */ + +Oid +OperatorGetOrCreateValidNegator(List *negatorName, + Oid operatorOid, + const char *operatorName, + Oid operatorNamespace, + Oid operatorLeftTypeId, + Oid operatorRightTypeId) +{ + Oid negatorId; + + /* negator has same arg types */ + negatorId = get_or_create_other_operator(negatorName, + operatorLeftTypeId, operatorRightTypeId, + operatorName, operatorNamespace, + operatorLeftTypeId, operatorRightTypeId); + + /* + * Check we're not self negating. An invalid negatorID means that the + * neither the operator or negator exist, but that their signatures are + * the same and therefore it's a self negator. + */ + if (!OidIsValid(negatorId) || negatorId == operatorOid) + ereport(ERROR, + (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), + errmsg("operator cannot be its own negator"))); + + /* Permission check: must own other operator */ + if (!object_ownercheck(OperatorRelationId, negatorId, GetUserId())) + aclcheck_error(ACLCHECK_NOT_OWNER, OBJECT_OPERATOR, + NameListToString(negatorName)); + + return negatorId; +} + +/* + * OperatorValidateParams + * + * This function validates that an operator with arguments of leftTypeId and + * rightTypeId returning operResultType can have the attributes that are set + * to true. If any attributes set are not compatible with the operator then this + * function will raise an error, otherwise it simply returns. + * + * This function doesn't check for missing requirements so any attribute that's + * false is considered valid by this function. As such, if you are modifying an + * operator you can set any attributes you are not modifying to false to validate + * the changes you are making. + */ + +void +OperatorValidateParams(Oid leftTypeId, + Oid rightTypeId, + Oid operResultType, + bool hasCommutator, + bool hasNegator, + bool hasJoinSelectivity, + bool hasRestrictionSelectivity, + bool canMerge, + bool canHash) +{ + if (!(OidIsValid(leftTypeId) && OidIsValid(rightTypeId))) + { + /* If it's not a binary op, these things mustn't be set: */ + if (hasCommutator) + ereport(ERROR, + (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), + errmsg("only binary operators can have commutators"))); + if (hasJoinSelectivity) + ereport(ERROR, + (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), + errmsg("only binary operators can have join selectivity"))); + if (canMerge) + ereport(ERROR, + (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), + errmsg("only binary operators can merge join"))); + if (canHash) + ereport(ERROR, + (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), + errmsg("only binary operators can hash"))); + } + + if (operResultType != BOOLOID) + { + /* If it's not a boolean op, these things mustn't be set: */ + if (hasNegator) + ereport(ERROR, + (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), + errmsg("only boolean operators can have negators"))); + if (hasRestrictionSelectivity) + ereport(ERROR, + (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), + errmsg("only boolean operators can have restriction selectivity"))); + if (hasJoinSelectivity) + ereport(ERROR, + (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), + errmsg("only boolean operators can have join selectivity"))); + if (canMerge) + ereport(ERROR, + (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), + errmsg("only boolean operators can merge join"))); + if (canHash) + ereport(ERROR, + (errcode(ERRCODE_INVALID_FUNCTION_DEFINITION), + errmsg("only boolean operators can hash"))); + } +} diff --git a/src/include/catalog/pg_operator.h b/src/include/catalog/pg_operator.h index e60cf782a6..7e1f8680a2 100644 --- a/src/include/catalog/pg_operator.h +++ b/src/include/catalog/pg_operator.h @@ -104,4 +104,28 @@ extern ObjectAddress makeOperatorDependencies(HeapTuple tuple, extern void OperatorUpd(Oid baseId, Oid commId, Oid negId, bool isDelete); +extern Oid OperatorGetOrCreateValidCommutator(List *commutatorName, + Oid operatorOid, + const char *operatorName, + Oid operatorNamespace, + Oid operatorLeftTypeId, + Oid operatorRightTypeId); + +extern Oid OperatorGetOrCreateValidNegator(List *negatorName, + Oid operatorOid, + const char *operatorName, + Oid operatorNamespace, + Oid operatorLeftTypeId, + Oid operatorRightTypeId); + +extern void OperatorValidateParams(Oid leftTypeId, + Oid rightTypeId, + Oid operResultType, + bool hasCommutator, + bool hasNegator, + bool hasJoinSelectivity, + bool hasRestrictionSelectivity, + bool canMerge, + bool canHash); + #endif /* PG_OPERATOR_H */ diff --git a/src/test/regress/expected/create_operator.out b/src/test/regress/expected/create_operator.out index f71b601f2d..49070ce90f 100644 --- a/src/test/regress/expected/create_operator.out +++ b/src/test/regress/expected/create_operator.out @@ -260,6 +260,42 @@ CREATE OPERATOR #*# ( ); ERROR: permission denied for type type_op6 ROLLBACK; +-- Should fail. An operator cannot have a self negator. +BEGIN TRANSACTION; +CREATE FUNCTION create_op_test_fn(boolean, boolean) +RETURNS boolean AS $$ + SELECT NULL::BOOLEAN; +$$ LANGUAGE sql IMMUTABLE; +CREATE OPERATOR === ( + leftarg = boolean, + rightarg = boolean, + procedure = create_op_test_fn, + negator = === +); +ERROR: operator cannot be its own negator +ROLLBACK; +-- Should fail. An operator cannot have a self negator. Here we test that when +-- 'creating' an existing shell operator, it checks the negator is not self. +BEGIN TRANSACTION; +CREATE FUNCTION create_op_test_fn(boolean, boolean) +RETURNS boolean AS $$ + SELECT NULL::BOOLEAN; +$$ LANGUAGE sql IMMUTABLE; +-- create a shell operator for ===!!! by referencing it as a commutator +CREATE OPERATOR === ( + leftarg = boolean, + rightarg = boolean, + procedure = create_op_test_fn, + commutator = ===!!! +); +CREATE OPERATOR ===!!! ( + leftarg = boolean, + rightarg = boolean, + procedure = create_op_test_fn, + negator = ===!!! +); +ERROR: operator cannot be its own negator +ROLLBACK; -- invalid: non-lowercase quoted identifiers CREATE OPERATOR === ( diff --git a/src/test/regress/sql/create_operator.sql b/src/test/regress/sql/create_operator.sql index f53e24db3c..168cad3814 100644 --- a/src/test/regress/sql/create_operator.sql +++ b/src/test/regress/sql/create_operator.sql @@ -210,6 +210,42 @@ CREATE OPERATOR #*# ( ); ROLLBACK; +-- Should fail. An operator cannot have a self negator. +BEGIN TRANSACTION; +CREATE FUNCTION create_op_test_fn(boolean, boolean) +RETURNS boolean AS $$ + SELECT NULL::BOOLEAN; +$$ LANGUAGE sql IMMUTABLE; +CREATE OPERATOR === ( + leftarg = boolean, + rightarg = boolean, + procedure = create_op_test_fn, + negator = === +); +ROLLBACK; + +-- Should fail. An operator cannot have a self negator. Here we test that when +-- 'creating' an existing shell operator, it checks the negator is not self. +BEGIN TRANSACTION; +CREATE FUNCTION create_op_test_fn(boolean, boolean) +RETURNS boolean AS $$ + SELECT NULL::BOOLEAN; +$$ LANGUAGE sql IMMUTABLE; +-- create a shell operator for ===!!! by referencing it as a commutator +CREATE OPERATOR === ( + leftarg = boolean, + rightarg = boolean, + procedure = create_op_test_fn, + commutator = ===!!! +); +CREATE OPERATOR ===!!! ( + leftarg = boolean, + rightarg = boolean, + procedure = create_op_test_fn, + negator = ===!!! +); +ROLLBACK; + -- invalid: non-lowercase quoted identifiers CREATE OPERATOR === ( ^ permalink raw reply [nested|flat] 67+ messages in thread
end of thread, other threads:[~2023-06-22 16:35 UTC | newest] Thread overview: 67+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2016-11-08 04:28 Re: Parallel tuplesort (for parallel B-Tree index creation) Peter Geoghegan <[email protected]> 2016-11-10 00:01 ` Robert Haas <[email protected]> 2016-11-10 00:54 ` Peter Geoghegan <[email protected]> 2016-11-10 01:03 ` Peter Geoghegan <[email protected]> 2016-11-10 02:57 ` Robert Haas <[email protected]> 2016-11-10 03:18 ` Peter Geoghegan <[email protected]> 2016-11-15 15:44 ` Robert Haas <[email protected]> 2016-12-04 01:29 ` Peter Geoghegan <[email protected]> 2016-12-04 01:45 ` Alvaro Herrera <[email protected]> 2016-12-04 02:37 ` Peter Geoghegan <[email protected]> 2016-12-04 03:23 ` Tomas Vondra <[email protected]> 2016-12-04 20:44 ` Peter Geoghegan <[email protected]> 2016-12-05 05:08 ` Haribabu Kommi <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-16 16:29 [PATCH 02/10] review Tomas Vondra <[email protected]> 2021-03-16 16:29 [PATCH 02/10] review Tomas Vondra <[email protected]> 2021-03-16 18:26 [PATCH 04/10] review Tomas Vondra <[email protected]> 2021-03-16 18:26 [PATCH 04/10] review Tomas Vondra <[email protected]> 2021-03-17 01:19 [PATCH 06/10] review Tomas Vondra <[email protected]> 2021-03-17 01:19 [PATCH 06/10] review Tomas Vondra <[email protected]> 2021-03-17 01:36 [PATCH 08/10] review Tomas Vondra <[email protected]> 2021-03-17 01:36 [PATCH 08/10] review Tomas Vondra <[email protected]> 2023-06-22 16:35 [PATCH] Extend ALTER OPERATOR to support adding commutator, negator, hashes, and merges Tommy Pavlicek <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox